llms.txt, and how much to believe about it
The idea, proposed by Jeremy Howard in 2024, is straightforward: a markdown file at /llms.txt that tells a language model which pages on your site are worth reading. Robots.txt says what a crawler may not touch; llms.txt says what it should prioritise.
The honest position on adoption
No major AI vendor has committed to reading it. Anyone telling you llms.txt is necessary for AI visibility is selling something. What is true: adoption is spreading among documentation sites and developer tools, the file is static and costs nothing to serve, and if the convention holds you are already there.
It is a cheap bet on a plausible convention. Worth ten minutes, not worth a project.
What actually moves AI visibility
In order: letting retrieval crawlers reach the page at all, then having content structured so a clean answer can be lifted out of it. A file listing your URLs does nothing if OAI-SearchBot is disallowed, or if every section takes three paragraphs to reach the point.
Check the first with the AI crawler checker and the second with the AI visibility checker as well.
Curate it
We generate from your sitemap because that is the only complete list available, but a sitemap aims for completeness and llms.txt aims for selectivity. Twenty pages that genuinely answer something beat four hundred that include your tag archive. Delete generously, and write a real one-line description at the top.