What is llms.txt? A website index for AI agents
By Flavio
llms.txt is a Markdown guide to a website's useful content for AI agents. See a real file, how agents follow its links, and how it differs from robots.txt.
llms.txt is a proposed Markdown format for a website guide that agents can read. It gives them a short introduction and links to useful content.
For this site’s implementation, read Serving my site as Markdown to AI agents. That post explains how the index connects to the Markdown copy of each article.
How llms.txt works
An agent reads the guide, picks a relevant link, and fetches that page. You can see the process with this site’s live llms.txt, which points to tutorials, courses, tools, and apps.
The proposal requires a project name as an H1. It also describes a summary blockquote and lists of links under H2 headings. A file can live at the root or under a path such as /docs/llms.txt.
robots.txt tells crawlers which paths they may access. llms.txt helps an agent find useful information. For crawler controls, read robots.txt for AI crawlers.
llms.txt around the coding tools
Give an agent one of these documentation indexes as a starting point when it needs a guide or API reference.
- Claude Code’s index links to Markdown guides for its tools and configuration.
- The Codex documentation index links to guides and a combined documentation export.
- Cursor’s index lists documentation URLs, including Markdown pages.
- Grok’s documentation index covers the API and app documentation. It explains how to request Markdown pages.
Details checked against each vendor’s documentation in November 2026.
A real example
Fetch this site’s guide from a terminal:
curl -fsSL https://flaviocopes.com/llms.txtThe guide explains that blog posts have Markdown mirrors. Following its instructions, fetch the npm post:
curl -fsSL https://flaviocopes.com/npm.mdThat lets an agent read the article without the site’s navigation. The token cost calculator lets you inspect text before adding it to a prompt.
When to use llms.txt
Use it when a documentation site offers one and you need to locate a specific API or setup guide. It also gives agents an entry point to content you publish yourself.
The llms.txt generator helps you write your own index, with a site name, summary, and sections of links.
Keep the file focused. Put detailed explanations behind links, so an agent can fetch the parts its current task needs. That is part of context engineering.
Pitfalls
Test the URLs in your index after moving content. A readable guide with stale links still sends an agent to the wrong page. Fetch a linked document too, rather than checking only the index.
Treat what comes back as website content. A fetched page can contain prompt injection, including instructions unrelated to the task.
For installed commands, the equivalent discovery problem is covered by helping a coding agent find your CLI tools, using CLI Tools Cabinet.
Related glossary entries
- Context engineering: Context engineering is choosing what an agent sees before it edits. How Claude Code, Codex, Cursor and Grok Build load rules, files and skills.
- Context window: The context window is everything a coding agent can see at once. How to check it, free it and keep it small in Claude Code, Codex, Cursor and Grok Build.
- Tool calling: Tool calling is when a model names a function and its arguments, and the app runs it. How Claude Code, Codex, Cursor and Grok Build take that step.
Want me to talk about your product? You can sponsor this site.