llms.txt File: What It Does and Whether You Need One

llms.txt is a proposed plain-text markdown file that a website can publish at its root, yoursite.com/llms.txt, to give large language models a short, curated map of the site's most important content, written in a format that is cheap for a model to parse. It was proposed in September 2024 by Jeremy Howard of Answer.AI, and it has spread mostly through developer tooling and documentation platforms rather than through any formal standards body.

If you have seen it mentioned next to robots.txt and sitemap.xml, that is an accident of file placement, not of function: it does not block crawlers and it does not list every URL on the site. This article covers what the file actually does, how it differs from the two files it is constantly compared to, which systems are actually confirmed to read it, and what a sensible minimal version looks like if you decide to publish one.

What llms.txt Actually Is

llms.txt is a markdown document, not XML and not a rules file. The proposal specifies a fixed shape: an H1 with the site or project name, a blockquote with a one or two sentence summary, optional plain paragraphs with context a model would need before reading further, and then one or more H2 sections, each containing a markdown list of links. Each link entry is written as a title, a colon, and a short description, so a model can decide which pages are worth fetching without downloading the whole site. A closing Optional section is allowed for secondary links that can be skipped if the model is working with a small context budget.

The stated problem it solves is real. A modern website's HTML is bloated with navigation, scripts and layout markup that a model has to wade through before reaching the actual content, and a full crawl of every page can blow past what fits usefully in a context window. llms.txt is meant to sit in front of that problem as a hand-picked index, the equivalent of handing a new hire a one-page start-here document instead of the company wiki's full directory. It answers a curation question, not a permissions question, and that distinction is the single most important thing to understand about the file before deciding whether to build one.

How It Differs From robots.txt

robots.txt is an actual protocol. It was followed informally by web crawlers for roughly three decades and then formalized as the Robots Exclusion Protocol in RFC 9309 in 2022, which means compliant crawlers have a defined, machine-checkable syntax to follow: user-agent blocks, Allow and Disallow directives, and a pointer to the sitemap. A crawler that respects robots.txt uses it to decide what it is permitted to request in the first place. It is an access-control file, and ignoring it has a defined meaning.

llms.txt makes no access claims at all. It has no Allow, no Disallow, no user-agent targeting, and nothing in the proposal binds any crawler to honor it, because there is no enforcement mechanism, since it is not a permissions layer to begin with. A model could ignore it entirely and remain fully compliant with everything robots.txt actually governs. The two files can coexist without any conflict for exactly this reason: one says what is allowed, the other says what is worth reading first if you are only going to read a little. Calling llms.txt "robots.txt for AI" misdescribes it. The more honest comparison is a curated README sitting next to a rulebook, not a second rulebook.

How It Differs From sitemap.xml

sitemap.xml exists to make a site's full URL inventory discoverable and to hint at freshness through a lastmod field and, historically, at relative priority. It is exhaustive by design. A large ecommerce sitemap can list hundreds of thousands of URLs split across an XML sitemap index, and search engines have supported the format for over two decades as a standard part of crawl discovery.

llms.txt inverts that goal entirely. It is meant to be short, dozens of links at most rather than thousands, and hand-curated rather than mechanically generated from every URL that happens to exist. A page can appear in your sitemap and never appear in your llms.txt, and that is the intended behavior, not an oversight: sitemap.xml says here is everything, llms.txt says here is what matters if you are only reading a little. Trying to make llms.txt exhaustive defeats its stated purpose and produces exactly the bloated document it was designed to avoid in the first place.

Which AI Systems Actually Read It Today

This is the part worth being precise about, because the file's popularity online has run well ahead of any confirmed adoption. Publishing an llms.txt does not have a documented, confirmed effect on how any major assistant crawls, indexes, or cites your content as things currently stand. Google has addressed the file directly and publicly: its own representatives have stated that Google Search and its AI features do not use llms.txt as a ranking or retrieval input. No other major AI crawler operator, including the crawlers behind the leading conversational assistants, has published documentation committing to reading or prioritizing it either.

What has actually happened is adoption on the publishing side, not the consuming side. Documentation platforms, static-site generators and developer-tool vendors added llms.txt generation as a feature, and some agentic tools and retrieval assistants that let a person manually paste or fetch a URL will happily consume an llms.txt if pointed at it directly, because it is just a clean markdown link list, and any tool that can read markdown can use one if a human or an agent chooses to fetch it. That is a different claim from a crawler autonomously discovering and prioritizing the file during routine, unprompted crawling, which is the part that adoption discussions tend to blur together.

• Confirmed: some agentic and retrieval tools will use an llms.txt when a person or agent explicitly points them at the URL.
• Not confirmed: any major search or AI crawler fetching /llms.txt automatically as part of its routine, unprompted crawling.
• Explicitly denied: Google has stated its Search systems do not use it as an input today.

Anatomy of a Minimal, Well-Formed File

A minimal llms.txt does not need to cover the whole site. It needs to answer the same question a good README answers: what is this, and where should someone start if they only have a moment. In practice, the proposal's structure looks like this:

• An H1 with the site or project name, one line, no extra styling
• A blockquote directly under the H1 with a single-sentence summary of what the site or product is
• Zero or more plain paragraphs giving any context a reader would need before the links make sense, such as audience, scope, or anything unusual about the domain
• One or more H2 sections, each grouping a handful of links by theme such as Documentation, Guides, or Reference, formatted as a markdown link followed by a colon and a one-line description
• An optional final H2 named Optional, for secondary links that can be skipped entirely under a tight context budget

Two implementation details cause most of the practical problems. First, the description after each link should describe what is actually on that page, not restate the title in different words; a model deciding whether to fetch a link has nothing else to go on but that one sentence. Second, the file should point only at content that is actually readable without a login wall, a paywall, or heavy client-side rendering, because a link to a page that needs JavaScript to render its body defeats the purpose of curating a lightweight entry point in the first place.

Deciding Whether Your Site Needs One

The honest answer for most sites today is that publishing one costs little and skipping it carries no confirmed downside, so the decision mostly comes down to whether the exercise of writing one is useful on its own terms. Sites where it is easiest to justify tend to share a trait: a body of reference content that is genuinely hard to navigate from the outside, such as API documentation, a large knowledge base, or a product with many interlinked guides, where a short curated index is a real editorial improvement independent of whether any AI system ever consumes it. If writing the file forces your team to identify the ten pages that actually matter, that exercise has value even if every crawler alive today ignores the result.

Sites where it is harder to justify include a small marketing site with a handful of pages already one click from the homepage, or any site where the important pages change on a weekly basis, because a stale curated list is arguably worse than no curated list at all. It actively points a reader, human or otherwise, toward outdated material under a header that confidently claims it is current. If your content changes faster than someone on your team can realistically keep a short markdown file in sync with it, that is a signal the file will decay rather than help, and a decayed index is worse than a missing one.

Common Mistakes

A handful of avoidable errors compromise most of the llms.txt files that exist today:

• Treating it as an access-control file and adding Allow or Disallow style syntax that the format has no concept of and no crawler will interpret
• Auto-generating it from the full sitemap, which produces a multi-thousand-line file that defeats the entire point of curation
• Linking to pages that sit behind a login wall or that render only client-side, so a plain fetch of the URL will not actually surface any usable content
• Writing link descriptions that just repeat the page title instead of describing what the page actually contains
• Publishing the file once and never revisiting it, so it quietly drifts out of sync with the site it claims to describe
• Assuming that publishing it changes how the site is crawled, indexed, or cited anywhere, which is not something currently confirmed for any major system

Maintaining It Over Time

If you do publish one, treat it the way you would treat a hand-edited top-level navigation menu rather than an automated data feed. Review it on the same cadence you would review your site's primary navigation: when a major section launches or is retired, when the ten most important pages on the site change, or on a routine quarterly basis for a site that ships new content often. Because nothing enforces freshness the way a broken-link monitor enforces working links, a stale llms.txt fails silently. It keeps returning a normal response and keeps looking correctly formatted while quietly describing a site that no longer exists in that shape. Put a recurring reminder on reviewing it, or fold it into whatever process already reviews your sitemap, rather than treating it as a one-time setup task you complete and forget.