Skip to content
FLAVIO COPES
flaviocopes.com

robots.txt in the age of AI crawlers

By

A robots.txt refresher plus the new part: GPTBot, ClaudeBot, PerplexityBot and friends, blocking AI training vs AI search, and the tradeoffs.

~~~

For twenty years, robots.txt had one job: tell Googlebot and friends which parts of your site to skip.

Now a new wave of crawlers reads it. GPTBot, ClaudeBot, PerplexityBot. They’re not building a search index. Some collect training data for AI models. Some fetch your pages to answer a user’s question in a chat.

Should you block them? Let’s first refresh how the file works, because the syntax rules matter more than people think.

The syntax refresher

robots.txt lives at the root of your domain: https://flaviocopes.com/robots.txt. It’s a plain text file made of groups. Each group starts with one or more User-agent lines, followed by rules:

User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /api/public/

Sitemap: https://flaviocopes.com/sitemap.xml

Two wildcards work inside paths: * matches any sequence of characters, and $ anchors the end. Disallow: /*.pdf$ blocks every PDF.

The precedence rules

This is where most robots.txt files go wrong.

Rule 1: a crawler picks exactly one group. It picks the group with the most specific User-agent match. If there’s a User-agent: GPTBot group, GPTBot follows that group and completely ignores the User-agent: * group. Groups don’t combine.

This means if you block GPTBot from /private/ in its own group, you also need to repeat any general rules you want it to follow. It won’t inherit them from *.

Rule 2: within a group, the longest matching rule wins. Not the first one. For /api/public/data:

Disallow: /api/
Allow: /api/public/

Both match, but Allow: /api/public/ is longer (12 characters vs 5), so the path is allowed. When an Allow and a Disallow match with the same length, Allow wins.

Rule 3: no matching rule means allowed. robots.txt is opt-out, not opt-in. An empty file and no file at all mean the same thing: crawl everything.

You can test these rules interactively with my robots.txt tool — paste a file, type a path and a user agent, and it shows you which rule matched and why.

The AI crawlers

Here are the ones worth knowing, and what they actually do:

Training vs AI search: two different decisions

Notice that the list splits into two categories, and the distinction matters.

Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) collect data to bake into a model. Your content becomes part of the model’s knowledge, uncredited, forever.

AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index your page for answers that can cite the source. That’s closer to what a search engine does, and it can send you visitors.

You can block training while allowing search indexing. Blocking GPTBot while allowing OAI-SearchBot says: don’t train on my content, but keep it eligible for ChatGPT search. Do not rely on ChatGPT-User for this split because OpenAI says robots.txt rules may not apply to it.

A robots.txt that blocks AI training

Here’s a file that blocks the training crawlers and leaves everything else open:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /

Sitemap: https://flaviocopes.com/sitemap.xml

Remember rule 1: each AI crawler matches its own group, everyone else falls through to *. The tool above has this as a one-click preset, with the full crawler list.

Do they actually comply?

robots.txt is a convention, not an access control. There’s no enforcement. The file politely asks; the crawler decides.

OpenAI, Anthropic, and Google publish robots.txt guidance for their named crawlers. User-initiated agents are a separate case. OpenAI explicitly says robots.txt rules may not apply to ChatGPT-User.

Other bots have been caught ignoring the file or using undeclared user agents. Bytespider has a particularly bad reputation here.

So treat robots.txt as a statement of intent that honest crawlers follow. If you need to actually keep bots out, that happens at the network layer: user-agent filtering, rate limiting, or bot management on your CDN. Cloudflare, for example, can block AI crawlers with a toggle, verified by IP ranges rather than the self-declared user agent string. If you want to see what a crawler sends, the user agent arrives as a plain HTTP request header you can log and filter on.

And keep in mind robots.txt is public. Anyone can read yours, and a Disallow: /secret-admin-panel/ line is a signpost, not a lock.

The tradeoff of blocking

Blocking AI crawlers feels like the obvious move. Your content, your rules. But think about where readers are going.

A growing share of “searches” now happen inside ChatGPT, Claude, and Perplexity. If your pages can’t be fetched, you don’t appear in those answers. No citation, no link, no visitor. It’s the same tradeoff as blocking Googlebot in 2005 — technically your right, strategically questionable if you write content you want people to find.

My advice: block the pure training crawlers if the idea of your work becoming uncredited model weights bothers you — that’s what GPTBot, ClaudeBot, CCBot and Google-Extended groups are for. But think twice before blocking search crawlers such as OAI-SearchBot and Claude-SearchBot. For a personal blog, being the cited source in an AI answer is the new ranking on page one.

Tagged: Web Platform · All topics
~~~

Related posts about platform: