Should we use LLMS.txt files?
They’re being floated as a way to help AI systems understand your website, sort of like a sitemap, but in Markdown, with room to add summaries and context.
There are a few concerns, though…
1) There is no clear adoption.
- Google has already said they won’t support it.
- OpenAI and Perplexity haven’t signaled anything.
- The only real sign of interest so far is from Anthropic.
2) There is no evidence of bots using this file.
There are some studies out showing bots are crawling the llms.txt file. However, no studies prove that llms.txt led to a meaningful change in performance.
3) There are no clear standards of what the file should look like. There are a lot of open questions:
- How long should a summary be for each URL?
- Which URLs are included?
- How should URLs be organized within the file? Does organization even matter?
- Should the text repeat or enhance what is on the website?
- How long should the file be?
That doesn’t mean you shouldn’t experiment. I’m experimenting with this (view my llms.txt file). I’d like it to work as a tool to help with AI search optimization. However, my advice to be smart about it.
To test llms.txt successfully:
- Add an X-Robots noindex so the file itself doesn’t end up in Google’s search results.
- Test it with a small set of pages, not your entire site.
- Track whether those test pages actually get more AI bot hits or citations.
- Avoid manipulative summaries and spam. Keep it accurate and fair.
However, the bulk of your time should be on the things we know will work: clear HTML, proper rendering, strong internal linking, well-structured content, faster load times, following usability/accessibility best practices, etc.
Bottom line: It could make sense to test out these files. However, at present, don’t expect big results and keep the bulk of your time on the more proven tactics.