llms.txt: What It Is and Whether It Matters

Jeremy Howard, co-founder of Answer.AI, published the proposal at llmstxt.org on 2024-09-03. Version 2 of that document, current on 2026-09-21, records what changed across two years of use. The proposal carries one predicate: llms.txt proposes a curated document index. It defines no directive grammar, no access rules and no enforcement.

No standards body has ratified it. The W3C tracks it in a strategy-funnel issue opened 2025-04-27, still open and still at the investigation stage, which is the stage before any working group, charter or draft exists. The specification declined the Well-Known URIs standard, RFC 8615, because a /.well-known/ file exists only at an origin root while llms.txt sits at any path.

The specification fixes six elements in order: an optional byte-order mark, an H1 naming the site, a blockquote summary, unheaded Markdown sections, H2 file-list sections, and an Optional section holding links an agent drops under short context. The H1 alone is required.

# Example Company

> Example Company installs and services commercial HVAC systems across the Kansas City metropolitan area.

Service area, call-out rates and warranty terms answer most incoming questions.

## Core pages

- [Services](https://example.com/services.md): every service offered, with the counties covered
- [Pricing](https://example.com/pricing.md): call-out rates, minimums and what each visit includes

## Optional

- [Company history](https://example.com/about.md): founding date and ownership

Read the example as five visible parts. The H1 names the site. The blockquote states what the company does and where. The unheaded paragraph adds interpretation, not links. The ## Core pages H2 opens a file list, and each item pairs a Markdown hyperlink with a one-line description. The ## Optional H2 holds the single link an agent drops when context runs short. Every target ends in .md, because the proposal also asks sites to serve a clean Markdown copy of each page.

llms-full.txt is the second convention: one file carrying every page's title, URL, description and full Markdown body. The specification defines no such file — Mintlify, a documentation platform, originated it. Meaning inside the page itself belongs to Schema Markup for AI Search.

robots.txt controls access and is read by every major crawler. llms.txt suggests priority and controls nothing.

Read the table as seven statements. robots.txt states which paths an agent is permitted to fetch, and llms.txt lists the documents a site wants read first. robots.txt uses User-agent, Allow and Disallow lines, and llms.txt uses Markdown headings and hyperlinks. robots.txt exists at the origin root alone, and llms.txt sits at the root or any path beneath it. The major crawlers document honouring robots.txt, and no provider documents reading llms.txt. Publishing a robots.txt directive changes access for every agent it names, and publishing llms.txt changes access for nobody. Omitting robots.txt leaves a site open to every agent, and omitting llms.txt changes nothing either. robots.txt is RFC 9309 on the IETF Standards Track, and llms.txt is a proposal no body has ratified.

The consequence runs one sentence. A retrieval crawler refused in robots.txt stays refused, and the llms.txt file it never fetches reverses nothing.

Read the list as five entries, each naming a provider, its commitment and a date. Google committed to ignoring the file, in Search Central guidance last updated 2026-07-10: maintaining LLMS.txt files "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them". OpenAI committed to nothing, its bot documentation naming four agents and no file, checked 2026-09-21. Anthropic committed to nothing, its crawler article naming three agents and no file, checked 2026-09-21. Perplexity committed to nothing, its bot documentation serving an llms.txt for its own docs and claiming no reading behaviour, checked 2026-09-21. Microsoft committed to nothing, its Bing Webmaster crawler help holding no occurrence of the string, checked 2026-09-21.

Publication is not consumption. Four of the five serve an llms.txt for their own developer documentation, and a served file describes those documents rather than a crawler's behaviour.

Requests arrive, and they are rare. Ahrefs measured 137,210 domains across one month and published on 2026-06-15: 28% of them serve the file, 97% of the valid files took zero requests, and Ahrefs states that zero requests came from AI bots for files that do not exist.

The verdict splits four ways:

• Publishon a documentation site, where coding agents follow the file to reach API references — the heaviest use the specification names.
• Publishon a platform that generates the file, where the cost is a setting rather than a task.
• Publishon a site whose few key documents answer most questions, where the file doubles as an internal index.
• Skipwhen the same hour repairs a fault that blocks a machine reader outright, because a blocked reader reaches no file.

Google's own statement removes the downside: maintaining the file neither harms nor helps ranking there. The case rests on systems arriving later, not on systems reading now. We found no page in this market answering the adoption question with a provider name and a date. Repair that changes what a machine reader receives is technical remediation.