AI Summary
llms.txt is a Markdown file at /llms.txt that lists a site’s most important pages for LLM consumption. Proposed by Jeremy Howard in September 2024, it has reached 10.13% web adoption as of a 300,000-domain study. No major AI search engine has confirmed it affects citation outcomes. We run it on this site anyway, and we will explain exactly when that calculation makes sense.
Key Takeaway
llms.txt is cheap to implement and currently unproven for citation lift. Google explicitly ignores it. Perplexity has published no support for it. Do it after you have covered the GEO fundamentals: atomic sentences, freshness signals, entity consistency, Tier-1 media mentions. If your primary audience is developers consuming your documentation programmatically, the token efficiency case is real and the implementation is justified on its own merits.
What llms.txt Is (and What It Is Not)
Jeremy Howard published the llms.txt proposal on September 3, 2024 on llmstxt.org. The spec is a Markdown file located at the root of a website (yoursite.com/llms.txt) that provides a curated summary of the site’s most important pages with brief descriptions, formatted in a way that both humans and LLMs can read.
The companion format is /llms-full.txt, which bundles entire page content as a single Markdown file for use in agentic contexts that need the full content rather than a link list. The technical argument for both formats is token efficiency. When an AI agent reads a standard HTML page, it processes navigation, ads, JavaScript artefacts, and layout markup that add no informational value. A clean Markdown equivalent strips all of that out.
llms.txt is not a replacement for robots.txt. Robots.txt controls crawler access to your site. llms.txt is an optional content directory intended for inference-time use, not crawl-time control. The spec explicitly states it is designed for use “when a user explicitly requests information about a topic,” not for training or crawl-direction purposes.
Adoption Reality: What the Data Shows
SE Ranking analyzed nearly 300,000 domains and found that only 10.13% had an llms.txt file in place. That is a long way from the near-universal adoption of robots.txt or XML sitemaps. The pattern by site size is counterintuitive: high-traffic sites (100,001+ visits) adopt it at 8.27%, while mid-traffic sites (1,001-5,000 visits) adopt at 10.54%. Large, authoritative domains are slightly less likely to have the file than mid-tier ones.
SE Ranking also ran a machine learning model using XGBoost to test whether llms.txt presence correlates with AI citation frequency. The finding: removing the llms.txt variable from the model actually improved its prediction accuracy. The file added noise rather than signal to the citation frequency prediction. As of the study’s publication date, llms.txt does not correlate with being cited more often by AI systems.
This is consistent with the hype cycle that began in mid-2024: high initial excitement from developers and SEOs, followed by data that failed to match the narrative. The honest position in mid-2026 is that the proposal is technically coherent but practically unproven for the use case most marketers care about.
What the Engines Actually Do With llms.txt
Google’s position is the clearest in the field. Google Search’s official guidance states that AI Overviews and AI Mode rely on traditional SEO signals. This is confirmed in the pre-verified pool from the Google Search documentation and repeated by Google representatives publicly. Google added llms.txt files to its own developer and Search Central documentation in December 2025, but its Search team has stated this does not imply endorsement: representatives have publicly compared llms.txt to the old keywords meta tag and confirmed Google does not use it for AI Overviews or AI Mode. The Google Search team does not endorse llms.txt files for citation purposes.
Perplexity has published no support for llms.txt in its crawler documentation. PerplexityBot respects standard robots.txt directives. In our Perplexity citation strategy research, the verified position is that there is no separate AI-instruction file supported by PerplexityBot. Perplexity’s span-labeling pipeline evaluates individual text segments in source documents; it does not use an external manifest to decide what to include.
OpenAI’s crawler documentation for OAI-SearchBot mentions robots.txt compliance and published IP lists but makes no reference to llms.txt. ChatGPT’s crawler reads standard HTML. Anthropic’s engineering documentation references flat llms.txt files in the context of developer documentation workflows, consistent with the token-efficiency use case rather than a citation-ranking use case. Microsoft’s Copilot and Bing have not published support for the format.
The pattern is consistent across all major AI search engines: llms.txt is not part of any documented citation-ranking or retrieval pipeline. If the engines that drive AI citation do not use the file, it cannot improve citation rates. That is not skepticism. That is what the current documentation shows.
Where llms.txt Actually Works
Two narrow use cases have real evidence behind them.
The first is developer documentation with programmatic consumers. If your users are developers writing agents or scripts that programmatically read your API or SDK docs, an /llms-full.txt measurably improves token efficiency. Developers can point a context window at a single clean Markdown file instead of scraping HTML pages. This reduces noise, reduces cost, and improves the accuracy of LLM-generated code against your API. The FastHTML project, cited in the original proposal, is an example of a documentation-first implementation done for exactly this reason.
The second is internal AI infrastructure. If you are building MCP servers, RAG pipelines, or internal knowledge bases over your own content, a curated content manifest in llms.txt format is a practical input format that reduces the engineering effort needed to maintain a clean content index. This is an internal tooling use case, not a citation-ranking use case.
Both use cases share a property: the benefit is measurable in a controlled context. Token efficiency for developer documentation is testable. Internal RAG pipeline quality is testable. “This file will make ChatGPT cite us more” is not currently testable in a way that produces positive results, because the engines do not read the file for that purpose.
Our Position: We Run It, and Here Is Why
We run llms.txt and llms-full.txt on this site. The theme generates them automatically, so the implementation cost is effectively zero. At zero marginal cost, the option value of being compatible with a format that might become consequential in the future is worth having, even with no current citation benefit.
The decision framework for whether you should implement it is not “will this improve my AI citations?” (answer: not currently). The right question is “what is the marginal cost relative to the option value?”
If your CMS generates it automatically (WordPress with certain themes, Drupal with the llm_support recipe, various documentation platforms): implement it, verify the output is clean, move on. If implementation requires developer time: do it after you have covered the fundamentals that actually drive citation outcomes. Those fundamentals are documented in our GEO audit checklist and in the research from the arXiv GEO paper (KDD 2024), which found up to 40% visibility gains from combining cite-sources, quotation, and statistics methods. None of those methods involve a file at the root of your domain.
The Implementation Decision Framework
| Your situation | Implement llms.txt? | Reason |
|---|---|---|
| CMS auto-generates it | Yes | Zero marginal cost, option value |
| Developer docs with programmatic API consumers | Yes, prioritise /llms-full.txt | Real token efficiency gain for developer agents |
| Internal RAG pipeline over your content | Yes | Useful content manifest format |
| Marketing team hoping for citation lift | After fundamentals | No documented citation impact; spend budget on atomic sentences and freshness first |
| Requires dedicated developer sprint | After fundamentals | Opportunity cost is too high given unproven impact |
What Actually Moves Citations in 2026
Our May 2026 study of 153,425 citations found that 76.95% of cited URLs across platforms are not in the organic top-10. The citation signal is driven by passage quality, freshness, and entity authority, not by file presence at the domain root. Cited sentences in that dataset average 9.27 words, with a median of 10 words, and none exceeded 18 words. The 6-10 word range accounts for 45.2% of citations.
Those numbers point to where the work is: writing factual claims as short, independently parseable declarative sentences. Structuring content so H2 and H3 headings define self-contained span boundaries. Surfacing published and updated dates in HTML, visible bylines, and Article schema. Building consistent entity descriptions across LinkedIn, Crunchbase, and any Wikipedia entries you control. Earning coverage in Tier-1 media that the rerankers of major AI engines treat as domain authority signals.
None of these tactics involve llms.txt. All of them are documented in our May 2026 citation study and the March 2026 study that preceded it. For the full content structure playbook, see our atomic sentence SEO guide and the top-35% positional bias analysis.
llms.txt is worth understanding and worth implementing when the cost is low. It is not worth substituting for the structural content work that the data consistently shows drives AI citation. If you are unsure where your site stands on the fundamentals, our GEO audit is the right starting point.