llms.txt: what it is and whether it matters
llms.txt is a short Markdown file at the root of a website that tells AI models what the site is and which pages to read. It’s cheap to make, widely recommended, and widely misunderstood. Here’s what the proposal actually says, who reads the file, and whether your business needs one.
Updated
What llms.txt is
Web pages are built for people. The useful text sits inside menus, cookie banners, scripts and footers, and an AI model reading the page has to dig it out. llms.txt is a proposal to give models a shortcut: one small file, at a known address, that says what the site is about and links to the pages worth reading.
Jeremy Howard published the proposal on llmstxt.org on September 3, 2024, and it’s developed in the open on Answer.AI’s GitHub. In August 2026 he released a second version, based on two years of use. It says thousands of sites now publish a file, documentation platforms generate one automatically, and coding agents use them reliably.
It’s important to be clear about what it isn’t. llms.txt is not a W3C or IETF standard, no AI company has promised to read it, and it doesn’t control anything. It can’t block AI training or grant permission to crawl. That’s the job of robots.txt, which we cover in robots.txt for AI crawlers. The proposal puts the difference well: robots.txt says what access is acceptable; llms.txt is read on demand, “when an agent needs information about a topic while assisting a user”.
The format
The file is called llms.txt and normally lives at the root of the site: https://www.example.com/llms.txt. Version 2 also allows files in subfolders, such as /docs/llms.txt, each covering the pages under its path; where more than one applies, agents should use the most specific.
The contents are Markdown, in this order:
- An H1 with the name of the site or project. This is the only required part.
- A blockquote with a short summary, holding the key information needed to understand the rest of the file.
- Any amount of detail as paragraphs or lists, but no headings.
- H2 sections of links, each a Markdown list. Every item is a link,
[name](url), optionally followed by a colon and a note about what’s there.
The proposal’s own skeleton looks like this:
# Title
> Optional description goes here
Optional details go here
## Section name
- [Link title](https://link_url): Optional link details
## Optional
- [Link title](https://link_url)
A section called Optional is, by convention, for secondary links that an agent can skip when it’s short on room. In version 1 it had a mechanical meaning for tools that expanded the file into one long prompt. Version 2 dropped those tools from the proposal, so “Optional” is now just a helpful label.
The proposal’s advice for writing one is short: use concise, clear language, add a brief description to each link, avoid unexplained jargon, and test the file by asking an agent questions about your business with only the llms.txt to go on.
Markdown pages and llms-full.txt
The proposal has a second half that gets less attention: clean Markdown copies of your pages. A page at /services/leak-repair can offer a Markdown version at the same address with .md added or swapped in, such as /services/leak-repair.md. The links in llms.txt should point at those copies where they exist, so the agent gets text instead of HTML.
Version 2 adds a way to find them without guessing. A page can point to its Markdown copy with rel="alternate" type="text/markdown" and to the llms.txt that covers it with rel="describedby", either as <link> tags in the page head or as an HTTP header your host or CDN adds:
Link: </services/leak-repair.md>; rel="alternate"; type="text/markdown", </llms.txt>; rel="describedby"
You’ll also see llms-full.txt. It isn’t part of the proposal. It’s a convention from documentation platforms: Mintlify, for example, generates one that combines the entire documentation site into a single file, every page’s full Markdown in a row. That suits a software manual a coding agent wants to load in one go. For a small business site, one well-written llms.txt is plenty.
Is the Markdown work worth it for a service business? Usually not yet. Google’s John Mueller has called building separate Markdown pages for bots a bad idea, as Search Engine Journal reported, and Google says it doesn’t treat such files specially. If your CMS or a plugin produces them for free, fine. Don’t hand-maintain a second copy of your site.
Who actually reads it
This is where most articles about llms.txt go wrong. Here’s what each company has said, as of November 2026, with the source.
| Company | What it has said or done |
|---|---|
| Google Search | Its guide to generative AI features says you don’t need machine-readable files, AI text files or Markdown to appear in Search, “as Google Search itself doesn’t use them”. Keeping an llms.txt “will neither harm nor help” your visibility or rankings. Google added that note on June 15, 2026. |
| Google Chrome | Lighthouse’s experimental Agentic Browsing checks include an llms.txt audit. It flags a server error when fetching the file; a missing file is marked “not applicable”, and the audit calls the file optional. |
| OpenAI | Publishes an llms.txt for its developer docs. Its crawler page describes GPTBot, OAI-SearchBot and ChatGPT-User and says nothing about reading llms.txt. |
| Anthropic | Publishes an llms.txt for its developer docs. Its crawler help page describes ClaudeBot, Claude-SearchBot and Claude-User and says nothing about llms.txt. |
| Perplexity | Publishes an llms.txt for its developer docs. Its crawler page describes PerplexityBot and Perplexity-User and says nothing about llms.txt. |
So Google is split. The Search team says the file does nothing for Search, and before that John Mueller compared it to the old keywords meta tag. The Chrome team audits for it as part of making sites work for browser agents. Both can be true: Search doesn’t need a map of your site, but an agent landing on it cold might use one.
The other three companies host llms.txt files, which shows they find the format useful for documentation, especially for coding tools. It doesn’t show that ChatGPT, Claude or Perplexity look for the file on your website when answering a question about your business. None of them has said so.
The best public evidence on actual fetching is server logs, and the numbers are small. EZY.ai watched 83 small-business sites with an llms.txt for twelve weeks, from April to July 2026, and counted requests from each crawler:
| Crawler | robots.txt fetches | llms.txt fetches |
|---|---|---|
| OpenAI crawlers | 3,990 | 7 |
| Anthropic (ClaudeBot) | 3,120 | 9 |
| PerplexityBot | 775 | 0 |
| Googlebot | 5,125 | 67 |
| Meta-ExternalAgent | 172 | 193 |
Treat this as one data point, not proof. It’s a small panel, it skews towards owners who were already interested in AI visibility, bots were identified mostly by user agent, and EZY.ai sells tools that generate llms.txt files. Still, the shape is clear: the major AI crawlers request robots.txt thousands of times for every handful of llms.txt requests, with Meta’s crawler the exception.
Evidence vs. folklore
A lot of what’s written about llms.txt comes from people selling llms.txt services. Some common claims, checked:
- “llms.txt gets you cited in AI Overviews.” Google says the opposite in writing: Search ignores the file.
- “ChatGPT and Claude read llms.txt first.” Neither OpenAI nor Anthropic has said so, and the log data above suggests their crawlers rarely fetch it. One widely repeated figure, that ClaudeBot checks llms.txt on 45% of sites, comes with no source we could find.
- “llms.txt is an official standard.” It’s a proposal on a personal site with a GitHub repository and a Discord channel. That’s a perfectly good way to start a convention, but it isn’t a standard.
- “llms.txt lets you opt out of AI training.” It doesn’t. Training opt-outs live in robots.txt (GPTBot, ClaudeBot, Google-Extended and others).
- “The AI labs use it, so it must work.” They publish files for their developer documentation, where coding agents read them. That’s a real use, but a different one from an assistant researching local plumbers.
What the evidence does support: agents that find an llms.txt can use it, the proposal’s author reports heavy use by coding agents, and a growing set of tools (Lighthouse, documentation platforms, the Yoast and AIOSEO WordPress plugins, Wix) either check for the file or generate it.
Costs and benefits
What it costs
- An hour or two to write, less if your CMS or SEO plugin generates a draft.
- Upkeep. This is the real cost. A file that lists last year’s prices or a service you’ve dropped is worse than no file, because an agent that reads it will repeat the mistakes with confidence.
- Opportunity cost. Time and money spent on llms.txt aren’t spent on the things that demonstrably matter: letting AI crawlers in, putting facts like prices, hours and service area on real pages, and giving assistants a way to contact or book you.
What you get
- A clean summary for any agent that looks. Coding agents do; Meta’s crawler apparently does; others may start. A plain list of your services, area and contact options can’t hurt there.
- A place to point at your MCP server, if you have one, so an agent that reads the file learns it can ask questions and take actions rather than just read.
- A useful exercise. If you can’t summarise your business in one blockquote and ten links, your website probably doesn’t do it clearly either.
- No downside in Google, by Google’s own account.
An example for a service business
Here’s an llms.txt for a made-up plumbing company. It follows the format, puts the facts an assistant gets asked about (area, hours, what the business doesn’t do, how pricing works) near the top, and links to the pages that hold the detail.
# Oakleaf Plumbing
> Oakleaf Plumbing is a family-run plumbing and drain company in Portland, Oregon. We fix leaks, clear drains and install water heaters for homes and small businesses within 25 miles of downtown. Open Monday to Saturday, 7am to 7pm, with 24-hour emergency call-outs.
- Licensed and insured in Oregon. Free written estimates for jobs over $500.
- We don't work on gas lines, septic systems or commercial new builds.
- Prices on the pricing page are starting prices. A technician confirms the final price on site, before any work starts.
## Services
- [Leak repair](https://www.oakleafplumbing.example/services/leak-repair.md): Burst pipes, dripping taps, slab leaks
- [Drain cleaning](https://www.oakleafplumbing.example/services/drains.md): Kitchen, bathroom and main sewer lines, camera inspections
- [Water heaters](https://www.oakleafplumbing.example/services/water-heaters.md): Repair and replacement, tank and tankless
## Prices, area and policies
- [Pricing](https://www.oakleafplumbing.example/pricing.md): Call-out fee, starting prices, payment options
- [Service area](https://www.oakleafplumbing.example/service-area.md): Every town and neighborhood we cover
- [Guarantee](https://www.oakleafplumbing.example/guarantee.md): One-year labor guarantee and how to claim it
## Contact and booking
- [Request a quote](https://www.oakleafplumbing.example/quote): Form; a person replies within one business day
- [Book a visit](https://www.oakleafplumbing.example/book): Online booking for non-emergency jobs
- [MCP server](https://www.oakleafplumbing.example/mcp): For AI assistants: ask about our services, request a quote or book a visit
- Emergencies: call (503) 555-0100, answered 24 hours a day
## Optional
- [Blog](https://www.oakleafplumbing.example/blog.md): Seasonal maintenance tips
- [Careers](https://www.oakleafplumbing.example/careers.md)
A few things to copy from it:
- Lead with the answer to “can they help me?” What you do, where, and when, in the blockquote.
- Say what you don’t do. It saves an assistant from recommending you for jobs you’ll turn down.
- Describe every link. “Call-out fee, starting prices, payment options” tells the agent whether the page is worth fetching.
- Include every way to reach you, including the phone number for emergencies and, if you have one, an MCP server an assistant can use directly.
- Link to Markdown copies only if they exist. If they don’t, link to the normal pages.
- Keep it short. The detail lives behind the links. The file itself should fit comfortably in an assistant’s context.
Save it as plain UTF-8 text at /llms.txt, check that the address returns the file (not your 404 page or a redirect to the home page), and put a reminder in your calendar to reread it whenever prices, hours or services change.
Our recommendation
- Fix the basics first. Make sure your robots.txt and your CDN let AI search crawlers and user-triggered fetchers in, and that your key facts are on real, crawlable pages. That’s where AI visibility is actually decided today.
- Then write a short llms.txt, like the example above, if it takes you an hour or your CMS does it for you. It costs little, Google says it can’t hurt, and some agents use it.
- Don’t pay a retainer for it, and be sceptical of anyone promising rankings or citations from it. No AI company has said it uses the file that way.
- Skip the Markdown mirror unless your platform generates it, and skip llms-full.txt unless you publish long documentation.
- Measure instead of guessing. Check your server logs, or an analytics tool that recognizes AI agents, to see which assistants visit and what they request. If nobody ever fetches your llms.txt, you’ll know how much to invest in it.
Remember that llms.txt is read-only. It can describe your business, but it can’t take an inquiry or book a job. For that, assistants need an MCP server or tools on the page. See our guide to WebMCP, and how assistants discover MCP servers.
How Salesman Hal handles llms.txt
Our free AI-readiness scan fetches your /llms.txt and tells you whether it exists, whether it starts with a # heading as the format requires, how many links it has, and whether it mentions your MCP server. Hal doesn’t write llms.txt for you: your SEO plugin or CMS can do that, and you know your business better than a generator does. What Hal adds is the part llms.txt can’t do: a hosted MCP server that lets assistants ask questions, send inquiries and quote requests, and book meetings, the discovery files and tags that point agents to it, and analytics that show which AI agents visit your site.
FAQ
Does llms.txt help my Google rankings?
No. Google’s guidance says Search ignores llms.txt and that having one will neither help nor hurt your visibility or rankings, including in AI Overviews and AI Mode.
Does ChatGPT read my llms.txt?
OpenAI hasn’t said that it does. Its crawler documentation doesn’t mention llms.txt, and published server-log studies show OpenAI’s crawlers fetching it only rarely. An agent might still read it if it lands on your site and looks for a map, so a short, accurate file is harmless.
Can llms.txt stop AI companies training on my content?
No. llms.txt describes your site; it doesn’t grant or refuse anything. Training opt-outs go in robots.txt, using tokens such as GPTBot, ClaudeBot, Google-Extended and Applebot-Extended.
What’s the difference between llms.txt and llms-full.txt?
llms.txt is a short index with links. llms-full.txt isn’t part of the proposal; it’s a convention from documentation platforms that puts the full text of every page into one file. Small business sites rarely need it.
Can an llms.txt hurt me?
Not in Google, by Google’s account. The real risk is staleness: if it lists old prices, hours or services, an agent that reads it may pass those on to a customer.
Should I put my MCP server in llms.txt?
Yes, if you have one. It’s a cheap way to tell any agent that reads the file it can ask questions or take actions, not just read pages. Our free scan checks for it.
Sources
- The /llms.txt file, v2, Jeremy Howard, llmstxt.org
- Changes (v2, August 2026), llmstxt.org
- Optimizing your website for generative AI features on Google Search, Google Search Central
- Latest updates to Search documentation, Google Search Central
- llms.txt audit, Lighthouse, Chrome for Developers
- Google’s llms.txt guidance depends on which product you ask, Search Engine Journal
- Overview of OpenAI crawlers, OpenAI
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Claude Help Center
- Perplexity crawlers, Perplexity
- llms.txt, Mintlify docs
- We put llms.txt on 83 websites. OpenAI read it 7 times., EZY.ai Research
Further reading
- AI features and your website, Google Search Central
- llms-txt on GitHub, AnswerDotAI
llms.txt tells assistants what to read. Hal lets them act.
Give AI assistants a way to ask questions, send inquiries and book meetings with your business. Set up Hal in about ten minutes. Free during early access.
Or run the free AI-readiness scan first.