What is llms.txt, and do AI engines read it?
llms.txt is a proposed Markdown file that lists a site's key pages for language models. What it holds, whether AI engines read it, and what matters more.
What is llms.txt?
At a glance
| Where | /llms.txt at the site's root |
|---|---|
| Format | Markdown: a title, a summary, then lists of links |
| Proposed | 2024, llmstxt.org (now version 2) |
| Manifold tool | aeo_get_site_readiness checks it, free |
What goes in the file
The proposal sets a simple format:
- An H1 with the site's name, the only required part.
- A short summary in a blockquote.
- Optional notes.
- Sections under H2 headings, each a list of links with a line on what each page holds.
It also proposes a Markdown version of each page at the same URL plus .md, so an agent gets clean text instead of HTML.
Do AI engines read it?
The proposal says the files are used most by coding agents looking up software documentation. No AI answer engine has documented reading llms.txt when it answers a question, and an Ahrefs study found most files were never requested. Treat it as cheap and harmless, not as a fix for visibility.
What matters more
The engines do document these, and each can keep a site out of answers:
| Check | Why it matters |
|---|---|
AI crawlers allowed in robots.txt | An engine does not cite a page its crawler may not fetch |
| No CDN or firewall block on AI crawlers | A block at the edge hides the site before robots.txt is read |
| Content in the HTML without JavaScript | Many crawlers do not run JavaScript |
No noindex or nosnippet | Google documents that they keep a page out of its AI answers |
Checking it with Manifold
aeo_get_site_readiness is free. It fetches your site the way the engines do and reports each check above with how much it matters, llms.txt included and marked as unproven.
