Cloudflare AI Gateway: put a proxy in front of your LLM calls

By

Learn how Cloudflare AI Gateway sits between your application and AI models, then route and inspect a real request through a Worker.

~~~

Calling an AI model is easy.

You send a prompt to an API and receive a response.

The difficult part starts when real people use the application. You need to know which requests failed, how much they cost, and whether one user is sending too many of them.

Cloudflare AI Gateway adds that missing layer.

In this tutorial we’ll first understand what a gateway does. Then we’ll route a real Workers AI request through one and inspect the result.

What is an AI gateway?

An AI gateway is a service that sits between your application and an AI model.

Without a gateway, the request goes directly to the model provider:

your application -> model provider

With a gateway, the request takes one extra step:

your application -> AI Gateway -> model provider

The response travels back along the same path.

Your application still sends prompts and receives model output. The gateway observes and controls the request while it passes through.

This is similar to putting a reverse proxy in front of a web server. The proxy does not become the application. It gives you one place to add logging, caching, limits, and routing.

What AI Gateway is not

AI Gateway is not an AI model.

It does not make a weak model smarter. It does not fix a bad prompt. It does not store application data like D1 or KV.

Cloudflare also has Workers AI, which runs models on Cloudflare’s infrastructure. That is a different product.

You can use AI Gateway with Workers AI, OpenAI, Anthropic, Google, and other providers. The gateway controls the path. The provider runs the model.

Why put a gateway in the middle?

A direct model call is fine while experimenting. You can inspect the response in your terminal and retry it by hand.

Production is different.

A user might report that a request failed yesterday. A model may suddenly become slow. A small bug may send the same expensive prompt hundreds of times.

AI Gateway gives you one place to see and control those calls.

Logs and analytics

The gateway can record the provider, model, status, duration, token usage, and cost for each request.

This answers useful questions:

  • Which model is failing?
  • How long do requests take?
  • How many tokens are we using?
  • Did a request reach the provider at all?

Logs are also helpful when your application supports several providers. You don’t have to combine a different logging system for each one.

Caching

AI Gateway can cache a response and return it when the same request appears again.

This works well for prompts whose answers can safely be reused. A request that classifies the same public text is a good candidate.

Be careful with personalized prompts. Never let two users share a cached response that contains private or user-specific data.

Rate limiting and spend control

Model calls cost money. A public endpoint can become expensive when it is abused or when a client enters a retry loop.

The gateway can limit requests before they reach the provider. It can also enforce spend limits and route traffic according to metadata.

I still add application-level limits for users and features. The gateway is another boundary, not a replacement for understanding who is making the request.

Retries and fallbacks

A provider can return an error even when your code is correct.

AI Gateway can retry a request or send it to another model. This is useful when availability matters more than using one exact model.

A fallback model can behave differently, so test the output of every model in the route. A successful HTTP response is not automatically a useful answer.

The ways to call AI Gateway

Cloudflare currently offers three main paths.

From a Cloudflare Worker

Inside a Worker, the shortest path is the AI binding.

You call env.AI.run() and pass a gateway ID in the options. Cloudflare authenticates the binding inside your account, so you do not add a gateway token to the request.

This is the path we’ll use in the tutorial.

Through the unified REST API

The AI Gateway REST API gives you one Cloudflare endpoint for Workers AI and third-party models.

It supports several request formats, including OpenAI-compatible chat completions. Third-party model names include the provider:

openai/gpt-4.1
anthropic/claude-sonnet-4.5
google/gemini-3-flash

Cloudflare handles authentication and unified billing. You send a Cloudflare API token instead of a provider key.

This is the recommended path for a new integration outside Workers.

Through a provider-native endpoint

Provider-native endpoints keep the original provider’s API shape.

For OpenAI, for example, you replace this base URL:

https://api.openai.com/v1

with:

https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/openai

This path is useful when you already have provider-specific code or want to bring your own provider key.

We’ll return to its authentication headers later.

Create an AI Gateway

Let’s create a gateway for the example.

Open the Cloudflare dashboard and select your account. Then:

  1. Go to AI → AI Gateway
  2. Click Create Gateway
  3. Name it tutorial-gateway
  4. Leave logging enabled
  5. Create the gateway

The gateway ID becomes part of your application configuration. Use separate gateways for development and production if you want separate logs and limits.

Cloudflare can also create a gateway named default on the first authenticated request. I prefer an explicit name because it is easier to recognize in code and in the dashboard.

Build a small gateway example

We’ll route a Workers AI request through the gateway.

You need a Cloudflare Worker project. If this is your first Worker, start with my Cloudflare Workers tutorial.

Add the AI binding to wrangler.jsonc:

{
  "ai": {
    "binding": "AI"
  }
}

The binding will be available as env.AI.

If you use TypeScript, regenerate the binding types after changing the configuration:

npx wrangler types

Now replace the Worker code with this:

export default {
  async fetch(request, env) {
    const result = await env.AI.run(
      '@cf/meta/llama-3.2-3b-instruct',
      {
        messages: [
          {
            role: 'user',
            content: 'Explain what an API is in one short sentence',
          },
        ],
        max_tokens: 80,
      },
      {
        gateway: {
          id: 'tutorial-gateway',
          metadata: {
            feature: 'api-explanation',
          },
        },
      },
    )

    return Response.json({
      answer: result.response,
      logId: env.AI.aiGatewayLogId,
    })
  },
}

The first argument selects the model.

The second argument contains the model input. In this case we send one user message and limit the response length.

The third argument sends the request through tutorial-gateway. We also attach a small piece of metadata so the request is easier to find later.

The logId identifies the matching gateway log.

Run the Worker

Start the development server:

npx wrangler dev

Wrangler prints a local URL, normally http://localhost:8787.

Open it in the browser or call it with curl:

curl http://localhost:8787

You should receive a response like this:

{
  "answer": "An API is a defined way for software programs to communicate with each other.",
  "logId": "01K..."
}

The exact answer changes because the model generates it.

Although the Worker runs through the local development server, the model inference happens on Cloudflare and counts toward Workers AI usage.

Inspect the request

Return to AI → AI Gateway → tutorial-gateway in the Cloudflare dashboard.

Open the logs and find the ID returned by the Worker. You can inspect the model, duration, status, token counts, and the feature metadata.

This is the main value of the gateway. The application made a normal model call, but we now have a record outside the application itself.

Make a few more requests. You will see a separate log entry for each one.

Add caching to one request

Gateway options can control individual requests.

For example, we can cache the previous response for five minutes:

gateway: {
  id: 'tutorial-gateway',
  cacheTtl: 300,
  metadata: {
    feature: 'api-explanation',
  },
}

Repeated identical requests can now be served from the gateway cache instead of running the model again.

You can also bypass caching for one request:

gateway: {
  id: 'tutorial-gateway',
  skipCache: true,
}

My advice is to start without caching. Add it only after you know which outputs are safe to reuse.

Calling a third-party model from the binding

The same binding can call third-party models through Unified Billing.

The model name changes to the provider/model format:

const result = await env.AI.run(
  'openai/gpt-4.1-mini',
  {
    messages: [
      { role: 'user', content: 'Write a two-line poem about Rome' },
    ],
  },
  {
    gateway: {
      id: 'tutorial-gateway',
    },
  },
)

Cloudflare supplies the provider credentials and deducts the cost from your AI Gateway credits.

The binding does not support your own provider key for third-party models. Use a provider-native endpoint when you need BYOK.

Understanding the two authentication headers

Provider-native endpoints have two separate authentication layers.

If you call OpenAI through an authenticated gateway and send your OpenAI key with the request, you need:

  1. Authorization: Bearer <OPENAI_API_KEY> for OpenAI
  2. cf-aig-authorization: Bearer <CF_AIG_TOKEN> for AI Gateway

Example:

const response = await fetch(
  'https://gateway.ai.cloudflare.com/v1/ACCOUNT_ID/GATEWAY_ID/openai/chat/completions',
  {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${env.OPENAI_API_KEY}`,
      'cf-aig-authorization': `Bearer ${env.CF_AIG_TOKEN}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: 'gpt-4o-mini',
      messages: [
        { role: 'user', content: 'Explain DNS in one sentence' },
      ],
    }),
  },
)

Store both values as Worker secrets. Never put them in wrangler.jsonc or source code:

npx wrangler secret put OPENAI_API_KEY
npx wrangler secret put CF_AIG_TOKEN

If you omit the cf-aig-authorization header, the authenticated gateway rejects the request before it reaches OpenAI.

Notice that the unified REST API is different. It uses a Cloudflare token in the normal Authorization header and Cloudflare handles provider billing.

Caching is not always correct

Caching a weather answer for a few minutes may be useful. Caching a private support conversation under a shared key is dangerous.

Before enabling caching, ask:

  • Does the answer depend on a user or organization?
  • Does the prompt contain private data?
  • Does the answer change with time?
  • Do temperature or other model options change?
  • Would returning an old answer be harmful?

If any answer worries you, skip the cache.

Streaming responses also need special treatment. Gateway caching does not apply to a streaming response, because the response is sent while it is being generated.

Rate limits need application context

A gateway-wide limit can protect the provider from a traffic spike.

It does not automatically know that one person has used their daily allowance. Your application still needs an identity and its own policy.

I usually combine both layers:

  • application limits for each user or feature
  • a gateway limit for the entire service
  • a spend limit as the final boundary

The application gives you precision. The gateway gives you a central safety net.

Logging and private data

Gateway logs are useful because they can include prompts and responses.

That also means they may contain private data.

Decide what your application is allowed to record before sending production traffic. AI Gateway can disable logging for individual requests, apply data-loss prevention rules, or run with zero data retention.

Do not collect prompts just because a dashboard makes it easy. Collect the minimum you need to operate the product.

How I use AI Gateway

I use AI Gateway for StackPlan, where production LLM calls need a visible trail.

The application keeps model calls behind a small adapter. The rest of the code does not know whether a request goes directly to a provider or through a gateway.

That is the pattern I would use again:

feature code -> model adapter -> AI Gateway -> provider

The adapter owns the URL, headers, request shape, and error handling. AI Gateway owns cross-provider controls and logs.

This also makes local experiments easy. I can change the adapter configuration without rewriting every route that uses AI.

I explain that separation in more detail in my LLM adapter pattern tutorial.

When to use AI Gateway

AI Gateway is useful when:

  • an AI feature is used by real people
  • you need one view of cost, latency, and failures
  • several applications or providers need the same controls
  • caching can avoid repeated model calls
  • you need retries, fallbacks, limits, or routing

Skip it for a disposable script or an early local experiment. A direct provider request has fewer moving parts.

Once a model call becomes part of a real product, the extra layer starts paying for itself. You gain a place to see the request, control it, and understand what happened when it fails.

Tagged: Cloudflare · All topics

Want me to talk about your product? You can sponsor this site.

~~~

Related posts about cloudflare: