# Serving my site as Markdown to AI agents

> Why and how I made every post on this site available as clean Markdown. Append .md to any post URL and you get the Markdown source, served as a static file.

Author: [Flavio Copes](https://flaviocopes.com/about/) | Published: 2026-07-11 | Topics: [Cloudflare](https://flaviocopes.com/tags/cloudflare/) | Canonical: https://flaviocopes.com/serving-markdown-to-ai-agents/

AI agents read websites all day now. ChatGPT, Claude, Perplexity, and a hundred smaller tools fetch pages, parse them, and use what they find.

But they don't really want my HTML. An HTML page is full of things they don't care about: the header, the navigation, the footer, scripts, styling, the ebook box at the bottom of every post.

What they want is the content. And the cleanest way to hand someone content is **Markdown**.

So I made every post on this site available as Markdown. Append `.md` to the URL and you get it:

```bash
curl https://flaviocopes.com/npm.md
```

Here's why I did it, and how.

## Why Markdown

Two reasons.

First, **tokens**. LLMs read text in tokens, and tokens cost money. An HTML page is mostly noise: tags, classes, inline styles, scripts. The same content in Markdown is a fraction of the size. Cheaper to read, faster to process. I built a free [token cost calculator](https://flaviocopes.com/tools/token-cost/) that counts the tokens in a piece of text and shows what it costs to send to each model.

Second, **focus**. Markdown is just the content. No menu, no footer, no "subscribe to my newsletter" box. The agent gets the article and nothing else.

If I want agents to use my writing, I should give it to them in the shape they like.

## My first attempt: converting at the edge

My first version of this was fancier.

This site is hosted on Cloudflare Pages, and I wrote a middleware function that ran in front of every request. If the request carried an `Accept: text/markdown` header, the middleware took the HTML page and converted it back to Markdown on the fly, at the edge.

It worked. Same URL, two formats, proper content negotiation. I was proud of it.

Then I looked at my usage stats.

A root middleware on Cloudflare Pages runs in front of **everything**. Every HTML page, every image, every CSS file. Each one of those became a metered Workers invocation. My fully static site, the one that cost nothing to serve, was suddenly burning through the free Workers quota just to serve images.

And here's the part that made it really silly. My posts are *written* in Markdown. Astro converts them to HTML at build time. So I was converting Markdown to HTML, then paying to convert the HTML back to Markdown, on every request.

I was fighting the platform. I removed it.

## The better way: just publish the source

The fix was obvious once I saw it. The Markdown already exists. I just have to publish it.

Astro can generate more than HTML pages. An **endpoint** is a `.ts` file in `src/pages/` that returns whatever you want at build time. Name it `[slug].md.ts` and it generates one `.md` file per post, right next to the HTML version.

Here's the heart of it:

```ts
import { getCollection } from 'astro:content'

export async function getStaticPaths() {
  const posts = await getCollection('posts')
  return posts.map((post) => ({
    params: { slug: post.id },
    props: { post },
  }))
}

export async function GET({ props }) {
  const { post } = props
  return new Response(post.body, {
    headers: { 'Content-Type': 'text/markdown; charset=utf-8' },
  })
}
```

`post.body` is the raw Markdown of the post, before any HTML conversion. I return it, and Astro writes it to `dist/<slug>.md` at build time.

The real file does a bit more. It adds the title, description, and canonical URL at the top, and it rewrites relative links to absolute ones, so an agent reading the file can follow them. But that's the whole idea.

The result is a folder of plain static files. Cloudflare serves them from the edge like any other asset. No Workers invocations, no conversion, no CPU budget. Zero cost.

My advice: when you find yourself converting something back into the format it started in, stop. Publish the original.

## Helping agents find it

Files nobody knows about don't get read. So I added three pointers.

First, a `<link>` tag in the head of every post:

```html
<link rel="alternate" type="text/markdown" href="/npm.md" />
```

This is the standard way to say "this page also exists in another format". It's how browsers discover RSS feeds.

Second, an [llms.txt](https://flaviocopes.com/llms.txt) file. It's an emerging convention (see [llmstxt.org](https://llmstxt.org)): a curated Markdown index at the root of your site that tells LLMs what's here, section by section. Mine lists the main topics, the courses, the books, and the tools. A second file, [llms-full.txt](https://flaviocopes.com/llms-full.txt), lists every single post with its description, each one linking to its `.md` version.

Third, `robots.txt`. Mine now explicitly welcomes the AI crawlers: GPTBot, ClaudeBot, PerplexityBot, and friends. A plain `User-agent: *` already allows them, but being explicit costs nothing and removes any doubt.

One warning if you're on Cloudflare: check your dashboard. Cloudflare can block AI crawlers at the edge, and on newer zones it does so **by default**. If that switch is on, your robots.txt is irrelevant, because the bots never get through.

## Try it

It works on every post. Pick any URL and add `.md`:

```bash
curl https://flaviocopes.com/javascript.md
```

You'll get a clean Markdown document, title and all. The same file I wrote, more or less.

## There's a built-in way too

I'll be honest with you. Cloudflare has a feature called [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) that does this at the zone level, no code at all. On the **Pro plan** and above, you enable it in the dashboard under AI Crawl Control. When a request carries `Accept: text/markdown`, Cloudflare converts the HTML to Markdown at the edge and returns that instead.

That's the same content negotiation I tried to build with middleware, except Cloudflare runs it for you and only when an agent asks for Markdown — not on every image and CSS file. I'm on the free plan, so I didn't use it.

But I'm glad I didn't. For a static site built from Markdown sources, the static approach is better anyway: no conversion step means nothing to get wrong, and no per-request cost means nothing to meter.

The content was Markdown all along. Now it stays that way.