Control cache, rate, data, and fallback

The gateway is in the path. Now let’s use its controls. Each one is a policy decision wearing a feature’s name.

Cache only what is safe to share

Caching saves cost and latency. Identical requests get the stored answer without touching the provider. But think about what “identical” means. If two tenants send the same question and one gets an answer generated from the other’s context, caching just became a data leak.

Build cache keys from every relevant input and tenant boundary. You can steer caching per request with headers:

cf-aig-cache-ttl: 3600
cf-aig-skip-cache: true

Skip the cache for anything user-specific. Cache “explain what a Worker is”. Don’t cache “summarize this customer’s order history”.

Limit before you pay

A gateway rate limit caps requests per time window at the edge. A runaway loop or a scripted abuser hits your limit instead of your invoice. Set the limit from your real traffic and leave some headroom. Not orders of magnitude, because then the limit protects nothing.

Prompts are data

Gateway logs are a debugging gift and a liability at the same time. Whatever users type ends up stored. Review retention and redaction before launch. Decide how long logs live, decide who can read them, and keep obvious secrets out of prompts in the first place. The cheapest data to protect is data you never logged.

Fallback is a model change

The gateway can try providers in order and fail over when the first one errors:

// universal endpoint: an array of requests, tried in order
const body = [
  { provider: 'workers-ai', endpoint: '@cf/meta/llama-3.2-3b-instruct', query: { messages } },
  { provider: 'openai', endpoint: 'chat/completions', query: { model: 'gpt-4o-mini', messages } },
]

Failover keeps the feature up. But a fallback model differs in quality, context size, tool support, and safety behavior, so the answers change character. A fallback you never tested is a surprise deployment that triggers during an outage. That’s the worst possible moment to find out it refuses your prompts.

Configure a small rate limit and one fallback, then force the primary to fail and run both outputs through the same acceptance checks. If the fallback’s answers fail your checks, better to learn that today than during the outage.