llms.txt generator

Build an llms.txt from your sitemap so language models can find what is worth reading

Enter a site to build its llms.txt

We read your homepage and sitemap.xml, then draft a markdown map of the pages worth reading.

llms.txt, and how much to believe about it

The idea, proposed by Jeremy Howard in 2024, is straightforward: a markdown file at /llms.txt that tells a language model which pages on your site are worth reading. Robots.txt says what a crawler may not touch; llms.txt says what it should prioritise.

The honest position on adoption

No major AI vendor has committed to reading it. Anyone telling you llms.txt is necessary for AI visibility is selling something. What is true: adoption is spreading among documentation sites and developer tools, the file is static and costs nothing to serve, and if the convention holds you are already there.

It is a cheap bet on a plausible convention. Worth ten minutes, not worth a project.

What actually moves AI visibility

In order: letting retrieval crawlers reach the page at all, then having content structured so a clean answer can be lifted out of it. A file listing your URLs does nothing if OAI-SearchBot is disallowed, or if every section takes three paragraphs to reach the point.

Check the first with the AI crawler checker and the second with the AI visibility checker as well.

Curate it

We generate from your sitemap because that is the only complete list available, but a sitemap aims for completeness and llms.txt aims for selectivity. Twenty pages that genuinely answer something beat four hundred that include your tag archive. Delete generously, and write a real one-line description at the top.

Common questions

What is llms.txt?
A proposed convention, published by Jeremy Howard in 2024: a markdown file at /llms.txt listing the pages on your site most worth reading, so a language model fetching your site can find the substance without crawling navigation, cookie banners and footers.
Do AI companies actually read it?
Not reliably, and it is worth being straight about that. No major AI vendor has committed to honouring llms.txt. Adoption is growing among documentation sites and developer tools, and the cost is one static file. Treat it as a cheap bet on a convention, not as something with measurable returns today.
Where does the file go?
At the root of your domain: https://example.com/llms.txt. Some sites also publish llms-full.txt containing the full text of key pages rather than links to them.
Why only the first 100 URLs?
Because a useful llms.txt is curated, not exhaustive. A sitemap dump of 4,000 URLs defeats the point - the file exists to say which pages matter. Treat what we generate as a starting draft and cut it down.
Is this the same as a sitemap?
No. A sitemap is XML for search crawlers and aims for completeness. llms.txt is markdown for language models and aims for selectivity, with room for a description explaining what the site is and why a given page matters.