llms.txt basics
llms.txt is a community-proposed convention for a plain-text file that helps large language models find and read your most relevant content. It complements robots.txt rather than replacing it, and like robots.txt it is a hint that cooperating tools may follow — not an enforced control.
What llms.txt is
llms.txt is a convention published at llmstxt.org proposing a Markdown-formatted file at your site root that points language models at your most useful content in a clean, low-noise form. The idea is that LLM-based tools, which work best with concise context, can read a curated map rather than parsing full HTML pages.
It is a proposal and an emerging convention, not an official standard like RFC 9309. Adoption by AI tools is voluntary and still developing, so treat it as an optional supplement.
How it relates to robots.txt
llms.txt and robots.txt serve different goals. robots.txt expresses crawl permissions; llms.txt offers a content map for LLMs. They are complementary — you can publish both. Neither is an enforcement mechanism: just as a non-compliant crawler can ignore robots.txt, no tool is obliged to read or honour llms.txt.
Because adoption and behaviour are still evolving, avoid relying on llms.txt for any control you actually need to guarantee.
- Proposed convention from llmstxt.org, not an official standard
- Complements robots.txt — different purpose
- Voluntary; not an enforcement mechanism
How it appears in analytics and logs
llms.txt is a discovery aid for cooperating LLM tools. Its presence does not change crawl permissions and does not guarantee any tool will read it.
Diagnostic use case
Offer LLM-based tools a curated, plain-text map of your key pages, while understanding that adoption is voluntary and evolving.
What WebmasterID can help detect
WebmasterID shows which AI crawlers and assistants reach your pages, so you can observe activity around your content regardless of whether tools consume an llms.txt file.
Common mistakes
- Treating llms.txt as a standard or an enforced control — it is neither.
- Listing sensitive content in a public llms.txt file.
- Assuming every AI tool reads llms.txt — adoption is voluntary.
Privacy and accuracy notes
An llms.txt file is public, like robots.txt. Only list content you are happy to expose; it is not a privacy or access-control mechanism.
Related pages
- Writing an AI crawler policy for robots.txt
An AI crawler policy is a deliberate decision about which AI-related tokens you allow and which you disallow in robots.txt. This page offers a structured way to make and document those choices, while staying realistic: robots.txt is a request to compliant crawlers, not a legal or technical guarantee.
- robots.txt basics: what it does and what it cannot do
robots.txt is a plain-text file at your site root that tells compliant crawlers which paths they may request. This page covers the directives, how user-agent groups are matched, and the limits that trip people up: robots.txt is advisory, it does not hide pages from search, and it is not a security boundary.
- AI visibility analytics
Observe AI-crawler activity on your content regardless of llms.txt.
Sources and verification notes
- llmstxt.org — the llms.txt proposalCommunity proposal; adoption by AI tools is voluntary and evolving.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.