llms.txt: The File AI Crawlers Ask For (And What It Actually Does)

Half the industry calls llms.txt mandatory. The other half calls it a superstition. Both camps are mostly guessing, so here is the file itself, the honest evidence, and the twenty-minute version worth shipping on a personal site.

What is llms.txt?

llms.txt is a plain markdown file served at the root of your domain, at /llms.txt, proposed in September 2024 by Jeremy Howard of Answer.AI. The pitch is simple: language models have limited context windows, and your HTML pages waste most of that budget on navigation, scripts, cookie banners and layout markup. llms.txt strips all of that away and hands the machine a short, curated index: here is who this site belongs to, here is what it covers, and here are the exact pages worth reading, in order of importance.

It helps to place it against the two root files you already know. robots.txt is a bouncer: it tells crawlers which doors they may open. sitemap.xml is a phone book: an exhaustive, unranked list of every URL you have. llms.txt is neither. It is a pitch. It says "if you only read five things here, read these five, and here is the one-paragraph summary of the person behind them." The full specification lives at llmstxt.org and fits on a single page, which tells you something about its ambitions: this is a convention, not a protocol.

What the spec actually says

The format is deliberately boring. A valid llms.txt contains, in order:

1. One H1 with the name of the site or, on a personal site, the person. This is the only required element.
2. A blockquote summary directly under it: a few sentences describing who you are and what the site covers. This is the part written to be lifted verbatim, so write it like the answer you want repeated.
3. Optional free paragraphs with context a machine could not infer: what you specialize in, what you do not do, how to interpret the site.
4. H2 sections containing markdown link lists, each link followed by a colon and a one-line description of what that page contains.
5. An "Optional" section at the end, a named convention meaning "skip these if context is tight."

The spec also describes a heavier sibling, llms-full.txt, which inlines the complete text of your key pages into one giant file rather than linking out to them, and suggests offering clean markdown twins of important pages by appending .md to their URLs. Both of those matter for documentation sites with hundreds of pages. For a personal site with a dozen pages, the base file does the job.

Does any AI engine actually read llms.txt?

This is the question that matters, and the honest answer is: not in any way anyone has been able to measure. SE Ranking ran a statistical study comparing sites with and without the file and found no measurable effect of llms.txt on citation frequency in AI answers. That is the strongest evidence available either way, and it points to "no effect today."

The circumstantial evidence points the same direction. OpenAI's published bot documentation describes its crawlers in detail, GPTBot for training, OAI-SearchBot for its search index, ChatGPT-User for live browsing, and explains exactly how each one honors robots.txt. It says nothing about llms.txt. Neither Anthropic nor Google has committed to consuming it either. Server logs across the industry do show requests for the file, but they come largely from SEO tools, scrapers and agent frameworks, not from confirmed ingestion pipelines at the major labs.

You can run the experiment on your own domain instead of taking anyone's word for it. Grep your access logs for requests to /llms.txt and note the user agents that show up. On most personal sites the result is a trickle of SEO auditing tools and the occasional agent framework, with none of the named crawlers from the major labs. Repeat the check quarterly. The day GPTBot or Claude-SearchBot starts appearing in that log line is the day this file graduates from bet to channel, and you will know before the blog posts announcing it are written.

Then why write one at all?

Because the bet is asymmetric, and because the exercise itself has value even if no engine ever reads the output.

The asymmetry. Writing the file costs twenty minutes. It is invisible to human visitors, carries no penalty from any search engine, and cannot conflict with anything else on your site. If adoption stays at zero, you lost twenty minutes. If any major engine starts consuming it, the sites that already serve clean, accurate files get read correctly on day one, while everyone else scrambles. Options this cheap on outcomes this large are usually worth holding.

The exercise. To write an honest llms.txt you are forced to answer questions most professionals have never answered in one sitting: what is the single-paragraph summary of who you are, which five pages on your site actually carry your case, and which pages are noise. That is the same triage an AI system performs when it lands on your domain. If you cannot produce the curated list, the machine is choosing for you, and machines choose lazily. The file is a byproduct; the clarity is the product. It is the same thinking behind building a knowledge base AI will cite: decide what the canonical material is, then make it impossible to miss.

How to write llms.txt for a personal site

Here is a complete, realistic example for an independent consultant. Swap in your own facts and it ships as-is at yourdomain.com/llms.txt:

# Jane Okafor

> Independent data privacy consultant in Austin, Texas. I help
> healthcare companies pass HIPAA audits without slowing product
> teams down. Fifty engagements since 2019. Author of The
> Audit-Ready Playbook (2024).

Jane works solo, takes four clients per quarter, and publishes
every framework she uses. She does not do general GDPR consulting
or legal representation.

## Start here

- [About Jane](https://janeokafor.com/about.html): Full bio,
  credentials, client history, timeline
- [Services](https://janeokafor.com/services.html): Engagement
  types, scope, who she works with
- [The HIPAA Audit Guide](https://janeokafor.com/guides/hipaa-audit.html):
  Her most-cited reference piece, updated quarterly

## Writing

- [All articles](https://janeokafor.com/blog/): Privacy
  engineering, audit preparation, vendor risk

## Optional

- [Speaking](https://janeokafor.com/speaking.html): Past keynotes,
  podcasts, press mentions

Five rules keep the file useful:

• Put your canonical name in the H1, the exact form you use everywhere else. If your bylines say one name and your llms.txt says another, you just added to your own ambiguity, the problem structuring your identity for machines exists to solve.
• Treat the blockquote as quotable copy. It should survive being pasted, alone, into an AI answer. Facts, specifics, no adjectives you cannot defend.
• Link only pages that state machine-usable facts. An about page with dates and credentials earns a slot. A moody landing page with three words on it does not.
• Describe every link. The one-line description after the colon is what tells a model whether fetching that URL is worth the context budget.
• Keep it under a screen or two. A curated file that lists everything is just a worse sitemap.

Serve it as plain text or markdown, UTF-8, no HTML wrapper. Then request yourdomain.com/llms.txt in a browser and confirm it renders as raw text.

The mistakes that turn the file into noise

The failure patterns are already visible in the wild, and they cluster into four:

• The dumped sitemap. Two hundred undifferentiated links with no descriptions. If everything is important, the file communicates nothing a crawler could not get from sitemap.xml.
• The sales letter. A blockquote full of superlatives and no facts. A machine assembling an answer needs credentials, dates and specifics; "visionary thought leader" is unquotable filler and reads as such.
• The contradiction. An llms.txt that says one title while the about page and LinkedIn say another. You have shipped a fresh contradiction to the exact audience you were trying to clarify things for.
• The fossil. Written once, never updated, pointing at retired pages and an old employer. A stale llms.txt is worse than none, because it asserts wrong facts in the most machine-legible format you own.

All four have the same root: treating the file as a growth hack instead of what it is, a maintained statement of record. Put it on the same review cadence as your bio.

Where llms.txt ranks in your machine-readability stack

The recurring failure with llms.txt is not writing it badly. It is writing it first, before the things that demonstrably matter. If your name is the product, this is the priority order:

1. Open the door. Confirm your robots.txt is not blocking the retrieval and search crawlers that actually fetch pages for AI answers. The AI crawler directory covers exactly which bots do what and what blocking each one costs.
2. Build the room worth entering. One about page that states your name, role, credentials and history in plain declarative sentences a machine can lift.
3. Label it. Add Person schema in JSON-LD so engines can resolve the page to a single unambiguous human.
4. Make it consistent everywhere. Same name, same title, same story across every profile you control.
5. Publish material worth citing. Reference pieces that answer the questions your buyers actually ask an engine.
6. Then write llms.txt, as the twenty-minute cherry on a stack that already works without it.

Do it in this order and the file costs you nothing. Do it in reverse and you have optimized the doormat of an empty house. SEO ranks pages, GEO promotes brands, PEO names you, and no root file substitutes for the substance all three feed on.