llms.txt: a curated map of your site for AI

01What it does, and why you need it.

An AI agent that reads the file gets a map of where the important content lives, in a form that fits inside its working memory. It then follows the links it needs instead of crawling the whole site.

For most sites, you do not need it yet. Google says it does not use the file, and measured traffic to it is near zero. It matters for documentation sites read by coding agents, and it costs little to publish.

02Who should deal with this.

The web or content team, with 1 person accountable for keeping it accurate. It is a text file, so the cost is an hour to write and a routine to update it when key pages change. The decision that needs a senior view is not whether to publish it but what it is for: if you run developer documentation that coding agents consult, it has a demonstrated use; if you run a marketing site and someone is selling it to you as an AI visibility lever, the evidence below does not support that claim.

A note on the name

The file is llms.txt, plural, with an s. It is often written as llm.txt, which is a different filename and will not be found by any tool that looks for the correct one.

Level of importance

Low for most organisations. Moderate for software and documentation businesses. Cheap enough that publishing an accurate one is reasonable hygiene, provided nobody counts it as a result.

03Where it comes from.

llms.txt was proposed by Jeremy Howard of Answer.AI on 3 Sep. 2024. The stated problem was that AI systems increasingly read websites, but web pages are built for people: HTML wraps information in navigation, advertising and JavaScript, converting it back to clean text is imprecise, and context windows are too small for most websites in their entirety.

The proposal was revised to version 2 in Aug. 2026. By then, in Howard’s account, thousands of sites published the file, documentation platforms generated it automatically, and coding agents used it reliably. OpenAI, Anthropic and Google publish llms.txt files for their own developer documentation, and Chrome’s Lighthouse tool audits for the file as part of its agentic browsing checks.

It is a community proposal, hosted at llmstxt.org with a public GitHub repository. It is not a standard from the W3C, IETF or any other standards body, and no AI provider is bound to read it.

04How it relates to robots.txt and sitemap.xml.

The proposal is explicit that these do different jobs.

robots.txt

Tells automated tools what access is acceptable.

sitemap.xml

Lists all indexable pages for search engines.

llms.txt

A curated overview for AI systems, used on demand when an agent needs information about a topic while assisting a user.

Howard’s expectation was that it would be used mainly at inference time, not for training, and that is how it has been used.

2 things llms.txt does not do: it cannot block or restrict any crawler, and it has no effect on search rankings.

05The format.

The specification is deliberately small. A file named llms.txt sits at the root, /llms.txt, or at a subpath such as /docs/llms.txt, in which case it covers the pages under that path. Where more than 1 file applies, agents should use the most specific one. The contents, in order:

1 · H1

An H1 with the name of the project or site. This is the only required element.

2 · Blockquote

A blockquote with a short summary containing the key information needed to understand the rest of the file.

3 · Detail

Zero or more Markdown sections of any type except headings, with more detail about the project and how to interpret the linked files.

4 · Link sections

Zero or more sections under H2 headings, each containing a list of links. Each entry is a Markdown hyperlink, optionally followed by a colon and a note.

An “Optional” H2 section is a convention for secondary links an agent can skip when it needs a shorter context. In version 1 it carried a mechanical meaning for context-expansion tools; in version 2 it is a convention only.

The whole file is valid CommonMark, so it is readable by a person, by a language model, and by ordinary parsing code.

06Markdown versions of pages.

The second half of the proposal is often missed. It recommends that pages an agent might need also be available as clean Markdown at the same URL with .md appended (page.html.md) or with the extension replaced (page.md). URLs without a filename use index.html.md or index.md. The links in llms.txt should point to these Markdown versions, not to the HTML.

Version 2 adds a way for agents to find these files without guessing, using standard link relations. rel="alternate" with type="text/markdown" points to a page’s Markdown version, and rel="describedby" points to the llms.txt file that covers it. These can be HTML link elements or an HTTP Link header, which can be added at the server or CDN without touching pages:

HTTP Link header

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

One caution reported by practitioners: if Markdown copies of pages are left indexable by search engines, they create duplicate content at scale. Where they are published, they should be excluded from search indexing.

07llms-full.txt.

Many documentation sites also publish llms-full.txt, a single file containing the full text of the documentation rather than links to it. This is a convention adopted by documentation platforms and by the AI labs’ own docs sites. It is not defined in the llmstxt.org specification.

08What the evidence says.

This is the part to read before commissioning anything.

Ahrefs, May 2026

Ahrefs analysed the server logs of 137,210 domains for May 2026. 28% published an llms.txt file, a figure Ahrefs says is an upper bound because its customers skew technical. Of roughly 38,000 valid files, 97% received zero requests in the month. Of the 3% that were fetched, 96% of requests came from bots, and most of those were not AI systems: SEO audit tools were the largest single group at about 21%, followed by unidentified bots, general web crawlers and technology profiling tools. Named AI tools accounted for 19.5% of fetches, with GPTBot first and Claude Code second. No AI bot requested an llms.txt file on a domain where one did not exist; they do not go looking for it.

SE Ranking, Nov 2025

SE Ranking analysed nearly 300,000 domains, published in Nov. 2025. 10.13% had the file. Using correlation tests and an XGBoost model, they found no relationship between having llms.txt and how often a domain was cited in answers from major LLM-based assistants. Removing the variable from the model improved its accuracy. Adoption was similar across traffic tiers, with the highest-traffic sites slightly less likely to have it than mid-tier sites.

Google, on record

Google’s position is on record in its Search Central documentation, “Optimizing your website for generative AI features on Google Search”, published in May 2026 and updated in Jun. 2026. In a section headed mythbusting, Google says you do not need to create machine-readable files, AI text files, markup or Markdown to appear in Google Search, including its generative AI features, because Google Search itself does not use them. It adds that maintaining such files will not harm or help visibility, and that it is fine to maintain them for other services that do use them. Separately, John Mueller of Google wrote in Jun. 2025 that no AI system was using llms.txt, and compared it to the keywords meta tag: a self-declared description that is not verified against the site.

Chrome Lighthouse

Against that, Chrome’s Lighthouse includes an llms.txt check in its agentic browsing audits, which is where much of the confusion comes from. The 2 Google positions are not contradictory. Search does not use the file; the Chrome team treats it as one indicator of readiness for browser agents.

The use case that survives the evidence is developer documentation. Coding agents and tools such as Claude Code, Cursor and MCP-based integrations can be pointed at an llms.txt or llms-full.txt file and use it to find API references and tutorials without crawling. That is the context Howard designed it for and the one where adoption is real.

09How to create one.

1 · Write it by hand

For most sites this is the right approach, because the value is in the curation. A generator that dumps your sitemap into Markdown misses the point.

llms.txt, an illustrative file

# Example Ltd

> Example Ltd makes booking software for independent clinics in the UK. This file lists the pages most useful for understanding the product, its pricing and its API.

Pricing is in GBP. The API is REST with JSON responses and requires an API key.

## Product

- [How the booking system works](https://www.example.com/product/overview.md): Features, supported clinic types and integrations
- [Pricing](https://www.example.com/pricing.md): Plans, what each includes and billing terms

## Developer documentation

- [API quick start](https://www.example.com/docs/quickstart.md): Authentication, first request and error handling
- [API reference](https://www.example.com/docs/reference.md): All endpoints with request and response examples

## Company

- [About Example Ltd](https://www.example.com/about.md): Company background, leadership and contact details

## Optional

- [Blog](https://www.example.com/blog/llms.txt): Index of articles, updated monthly

Guidance from the specification: use concise, clear language; give each link a brief, informative description; avoid unexplained jargon; and test the file by offering an agent only your llms.txt and asking it questions about your content.

2 · Place and serve it

Save the file as llms.txt at the site root so it resolves at https://www.example.com/llms.txt. Serve it as plain text. If the site also publishes Markdown versions of pages, add the Link header at the server or CDN.

Check it

curl -s https://www.example.com/llms.txt

nginx, a docs directory

location /docs/ {
    add_header Link '</docs/llms.txt>; rel="describedby"';
}

3 · Or let the platform generate it

Several platforms produce the file automatically: Mintlify and GitBook for documentation sites; Yoast SEO and AIOSEO plugins for WordPress; Wix for its hosted sites; plugins for VitePress and Docusaurus; and a Drupal recipe. Command line generators such as llms-generator crawl a site and write a first draft. Treat generated output as a starting point to edit, not a finished file.

4 · Keep it accurate

A stale llms.txt that points to moved or deleted pages is worse than none, because an agent that trusts it will fail and may report that failure to the person it is helping. Put the file on the same review cycle as the site navigation.

10Checking whether anything reads it.

Filter server logs or CDN analytics for requests to /llms.txt and /llms-full.txt, then split by user agent. The user agents to look for include GPTBot, ClaudeBot, Claude-Code, PerplexityBot, OAI-SearchBot and Google-Extended. If the requests are all from SEO audit tools and checkers, that tells you who the file is currently serving.

11Sources.

01The llms.txt proposal, version 2, by Jeremy Howard. The authoritative description of the format, the Markdown page convention, link relations, directories and integrations. llmstxt.org/
02What changed between version 1 (Sep. 2024) and version 2 (Aug. 2026), and why. llmstxt.org/changes.html
03The public GitHub repository for the proposal, where issues and discussion are held. github.com/AnswerDotAI/llms-txt
04The FastHTML documentation llms.txt, the reference example used in the proposal, placed at /docs/ to cover only the documentation. www.fastht.ml/docs/llms.txt
05Anthropic’s llms.txt for its developer documentation, an example from an AI lab. docs.anthropic.com/llms.txt
06OpenAI’s llms.txt for its developer documentation. developers.openai.com/llms.txt
07Google’s llms.txt for the Gemini API documentation. ai.google.dev/gemini-api/docs/llms.txt
08Cloudflare’s llms.txt for its developer documentation. developers.cloudflare.com/fundamentals/llms.txt
09Chrome Lighthouse documentation for its llms.txt audit within the agentic browsing checks. developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt
10Google Search Central’s guide to optimising for generative AI features, including the mythbusting section stating that llms.txt is not used by Google Search. developers.google.com/search/docs/fundamentals/ai-optimization-guide
11Search Engine Journal’s report on the Google guide and its mythbusting section, May 2026. www.searchenginejournal.com/googles-new-ai-search-guide-calls-aeo-and-geo-still-seo/575026/
12Ahrefs’ study of 137,210 domains, May 2026 logs, finding 97% of llms.txt files received zero requests, with the breakdown of who fetched the rest. ahrefs.com/blog/llmstxt-study/
13Ahrefs’ companion explainer, including the 28% adoption figure and its caveat. ahrefs.com/blog/what-is-llms-txt/
14Search Engine Journal’s summary of the Ahrefs findings. www.searchenginejournal.com/97-of-llms-txt-files-got-no-requests-ahrefs-data-shows/579478/
15SE Ranking’s analysis of nearly 300,000 domains, Nov. 2025, finding 10.13% adoption and no correlation with AI citations. seranking.com/blog/llms-txt/
16Search Engine Journal’s report on the SE Ranking study. www.searchenginejournal.com/llms-txt-shows-no-clear-effect-on-ai-citations-based-on-300k-domains/561542/
17Search Engine Land’s early explainer of the proposal, quoting Howard’s stated rationale. searchengineland.com/llms-txt-proposed-standard-453676
18Mintlify’s documentation on automatic llms.txt and Markdown page generation for hosted docs. www.mintlify.com/docs/ai/llmstxt
19Yoast SEO’s llms.txt feature for WordPress. yoast.com/features/llms-txt/
20A command line tool that crawls a site and generates a first-draft llms.txt and llms-full.txt. pypi.org/project/llms-generator/
21Community-maintained directory of public llms.txt files, useful for seeing how other organisations have written theirs. directory.llmstxt.cloud/
22Practitioner explainer with a worked example and a summary of the evidence, including the Mueller comparison to the keywords meta tag. guptadeepak.com/llms-txt-explained-spec-and-working-example/

llms.txt: a curated map of your website for AI systems.