What llms.txt is
llms.txt is a proposal by Jeremy Howard, co-founder of Answer.AI, published at llmstxt.org in September 2024. The idea is small and practical: put a Markdown file at the root of your domain, at /llms.txt, that tells a language model or an AI agent what the site is and which pages it should read first. robots.txt says what a crawler may fetch and a sitemap lists every URL you have; llms.txt is a curated reading list with context, written for a reader that has a limited context window and no patience for navigation menus.
The format is plain Markdown with a fixed shape, so software can parse it and a person can read it:
- An H1 with the name of the site or project. This is the only required line.
- A blockquote with a short summary: what the site is, who it is for, and the facts a reader needs to interpret the rest.
- Optional paragraphs or lists with more detail, such as contact details, hours or a service area. Headings are not allowed here; the spec reserves them for the link sections.
- One or more H2 sections, each a list of links written as
- [title](url): note, where the note is a short hint about what the page contains. - An optional section titled
## Optional. Links placed there are lower priority and can be skipped when a shorter context is needed.
The generator above produces exactly that structure. Many sites also publish llms-full.txt, a companion file with the full text of the key pages so an assistant can read everything in one request. That is a convention that grew up around the proposal rather than part of it, and it only earns its keep when your content is long-form documentation.
Why a site might want one
Be clear-eyed about what this does and does not do. No major LLM provider has formally adopted llms.txt: OpenAI, Anthropic and Google have not committed to fetching it, and Google has said it is not a ranking signal. It will not move you up in search results, and it will not make a model memorize your site.
What it does is make you easy to read. When an assistant with browsing, a research agent or a coding agent lands on your domain and looks for an overview, llms.txt hands it your canonical pages, your own description and your contact details in a format designed for a model, instead of leaving it to scrape menus and footers. Developer documentation sites were the first adopters because coding agents consume docs all day. For a business site it is cheap insurance: twenty minutes of work, no risk to your existing SEO, and a say in how you are summarized. Think of it as a discoverability aid that sits next to your structured data and your sitemap, not a replacement for either.
How to publish it
- 01Save the file as
llms.txt, encoded as UTF-8, at the root of your domain so it is reachable athttps://yourdomain.com/llms.txt. On a static host or a Next.js export that is thepublicfolder; on WordPress it sits in the web root next to robots.txt, or an SEO plugin can write it for you. - 02Serve it as plain text. A content type of
text/plain; charset=utf-8ortext/markdownis right. Most hosts do this automatically for a.txtfile. - 03Optionally add
llms-full.txtbeside it with the full text of the pages you listed. - 04Reference it in two places so people and agents can find it: a comment line in robots.txt, such as
# llms.txt: https://yourdomain.com/llms.txt, and a small link in your site footer next to the sitemap link. - 05Keep it short and current. It is a curated list, not a sitemap: a dozen links with good notes beat two hundred without. Regenerate it when you add an important page.
How to verify it is live
Run curl -I https://yourdomain.com/llms.txt from a terminal. You want a 200 status and a text content type. A redirect to your homepage, a 404, or text/html means the file is not where you think it is. Then fetch it without -I and read it: no HTML, accents and symbols intact, every link absolute and resolving. Finally paste the file into an assistant and ask it to describe your business in two sentences. If the answer is wrong, fix the file, not the assistant.
Where to list it
Two public directories index llms.txt files, and both link back to your domain, so beyond discoverability they are a small, legitimate backlink from a relevant page:
- The llms.txt directory: submit your URL at directory.llmstxt.cloud/submit.
- The llmstxt-site list on GitHub at github.com/generaltranslation/llmstxt-site: open a pull request adding your domain.
Listing is optional. The file works the same whether or not anyone indexes it.
A worked example
This studio publishes its own file at thebaileyperspective.com/llms.txt. It opens with the H1 # The Bailey Perspective, a blockquote that says what we build, where we are and who runs it, then two plain lines with contact details and service area. A ## Key pages section lists the service pages, case studies, about and contact pages with a note on each, and a ## Facts an assistant can state section spells out the facts we want repeated correctly: speed, typical clients, pricing model, example work. The Load example button in the generator fills the form with a shortened version so you can see how each field maps to the output.