llms.txt, explained
llms.txt is a proposal, published in 2024 by Jeremy Howard of Answer.AI, for a simple Markdown file placed at the root of a website (for example, example.com/llms.txt). It contains a short summary of the site and a curated list of links to its most important pages, each with a one-line description. The idea is that a language model or AI agent can read one small, clean file instead of parsing many pages built for human browsers.
It is often compared to robots.txt, but the purpose is different. robots.txt tells crawlers what they may access. llms.txt offers a guide to what is worth reading. Some sites also publish an llms-full.txt containing the full text of key pages.
Be clear about the evidence. llms.txt is a proposal, not a standard, and major AI search providers have not confirmed that they use it when answering questions. Google has said no special AI text files are needed to appear in its AI features. A well-maintained llms.txt is cheap, harmless and occasionally useful for AI agents and developer tools; it should be treated as hygiene and a hedge, not sold as a ranking factor. The fundamentals of crawlable pages, clear content and a consistent entity matter far more.