Control cache, rate, data, and fallback

The gateway is in the path. Now use its controls — each one is a policy decision wearing a feature’s name.

Cache only what is safe to share

Model caching can save cost and latency: identical requests get the stored answer without touching the provider. But personalized, time-sensitive, or secret-bearing prompts may be unsafe to share under one key. If two tenants send the same question and one gets an answer generated from the other’s context, caching just became a data leak.

Build cache keys from every relevant input and tenant boundary. You can steer caching per request with headers:

cf-aig-cache-ttl: 3600
cf-aig-skip-cache: true

Skip the cache for anything user-specific. Cache the request that says “explain what a Worker is” — not the one that says “summarize this customer’s order history.”

Limit before you pay

Apply rate limits and budgets before provider calls. A gateway rate limit caps requests per time window at the edge, so a runaway loop or a scripted abuser hits your limit instead of your invoice. Set the limit from your real traffic, then leave headroom, not orders of magnitude.

Prompts are data

Review log retention and redaction because prompts and outputs may contain private data. Gateway logs are a debugging gift and a liability at the same time: whatever users type ends up stored. Decide retention, decide who can read the logs, and keep obvious secrets out of prompts in the first place.

Fallback is a model change

A fallback model can differ in quality, context, tools, or safety behavior, so evaluate it explicitly. The gateway can try providers in order and fail over when the first errors:

// universal endpoint: an array of requests, tried in order
const body = [
  { provider: 'workers-ai', endpoint: '@cf/meta/llama-3.2-3b-instruct', query: { messages } },
  { provider: 'openai', endpoint: 'chat/completions', query: { model: 'gpt-4o-mini', messages } },
]

Failover keeps the feature up, but the answers change character. A fallback you never tested is a surprise deployment that triggers during an outage — the worst possible moment to discover it refuses your prompts.

Configure a small limit and one fallback, then force the primary failure and compare both outputs against the same acceptance checks. If the fallback’s answers fail your checks, better to learn that today.