robots.txt in the age of AI crawlers

By

A robots.txt refresher plus the new part: GPTBot, ClaudeBot, PerplexityBot and friends, blocking AI training vs AI search, and the tradeoffs.

~~~

For twenty years, robots.txt mostly told Googlebot and friends which parts of your site to skip.

Now GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers read it too. Some collect training data for AI models. Others fetch your pages to answer a user’s question in a chat.

Before deciding whether to block them, let’s refresh how the file works. The syntax rules are easy to get wrong.

The syntax refresher

robots.txt lives at the root of your domain: https://flaviocopes.com/robots.txt. It’s a plain text file made of groups. Each group starts with one or more User-agent lines, followed by rules:

User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /api/public/

Sitemap: https://flaviocopes.com/sitemap.xml

Two wildcards work inside paths: * matches any sequence of characters, and $ anchors the end. Disallow: /*.pdf$ blocks every PDF.

The precedence rules

This is where most robots.txt files go wrong.

Rule 1: a crawler picks exactly one group. It picks the group with the most specific User-agent match. If there’s a User-agent: GPTBot group, GPTBot follows that group and completely ignores the User-agent: * group instead of combining both.

This means if you block GPTBot from /private/ in its own group, you also need to repeat any general rules you want it to follow because it won’t inherit them from *.

Rule 2: within a group, the longest matching rule wins. Not the first one. For /api/public/data:

Disallow: /api/
Allow: /api/public/

Both match, but Allow: /api/public/ is longer (12 characters vs 5), so the path is allowed. When an Allow and a Disallow match with the same length, Allow wins.

Rule 3: no matching rule means allowed. robots.txt is opt-out, not opt-in. An empty file and no file at all mean the same thing: crawl everything.

You can test these rules interactively with my robots.txt tool. Paste a file, type a path and a user agent, and it shows you which rule matched and why.

The AI crawlers

Here are the ones worth knowing, and what they actually do:

Training vs AI search: two different decisions

Notice that the list splits into two categories, and the distinction matters.

Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) collect data to bake into a model. Your content becomes part of the model’s knowledge, uncredited, forever.

AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index your page for answers that can cite the source. That’s closer to what a search engine does, and it can send you visitors.

You can block training while allowing search indexing. Blocking GPTBot while allowing OAI-SearchBot says: don’t train on my content, but keep it eligible for ChatGPT search. Do not rely on ChatGPT-User for this split because OpenAI says robots.txt rules may not apply to it.

A robots.txt that blocks AI training

Here’s a file that blocks the training crawlers and leaves everything else open:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /

Sitemap: https://flaviocopes.com/sitemap.xml

Remember rule 1: each AI crawler matches its own group, everyone else falls through to *. The tool above has this as a one-click preset, with the full crawler list.

Do they actually comply?

robots.txt is an unenforced convention that crawlers can choose to follow.

OpenAI, Anthropic, and Google publish robots.txt guidance for their named crawlers. User-initiated agents are a separate case. OpenAI explicitly says robots.txt rules may not apply to ChatGPT-User.

Other bots have been caught ignoring the file or using undeclared user agents. Bytespider has a particularly bad reputation here.

So treat robots.txt as a statement of intent that honest crawlers follow. If you need to actually keep bots out, that happens at the network layer: user-agent filtering, rate limiting, or bot management on your CDN. Cloudflare, for example, can block AI crawlers with a toggle, verified by IP ranges rather than the self-declared user agent string. If you want to see what a crawler sends, the user agent arrives as a plain HTTP request header you can log and filter on.

And keep in mind robots.txt is public. Anyone can read yours, and a Disallow: /secret-admin-panel/ line is a signpost, not a lock.

The tradeoff of blocking

Blocking AI crawlers feels like the obvious move. Your content, your rules. But think about where readers are going.

A growing share of “searches” now happen inside ChatGPT, Claude, and Perplexity. If your pages can’t be fetched, you don’t appear in those answers. You get no citation, link, or visitor. It’s the same tradeoff as blocking Googlebot in 2005: technically your right, but strategically questionable if you write content you want people to find.

My advice is to block the pure training crawlers if you don’t want your work becoming uncredited model weights. Use the GPTBot, ClaudeBot, CCBot, and Google-Extended groups for that.

I would think twice before blocking search crawlers such as OAI-SearchBot and Claude-SearchBot. For a personal blog, I see being cited in an AI answer as the new ranking on page one, and that citation can bring readers to the source.

Tagged: Web Platform · All topics

Want me to talk about your product? You can sponsor this site.

~~~

Related posts about platform: