What is an llms.txt file — and do I need one?
My journey optimising my website for GEO.
I had a chat with a GEO agency recently, and they casually dropped:
“You should consider adding an llms.txt file to your site.”
Cue the familiar spiral that happens right before I fall into a research rabbit hole.
"What the hell is an llms.txt file, and do I need one?"
A few hours, several tabs, and an unhealthy amount of Googling/ChatGPT-ing later, I landed on a great explainer by Derick Ruiz — and here’s the gist:
🔍 llms.txt is basically a sitemap — but for AI systems, not search engines.
It’s a proposed web standard that helps large language models understand and navigate developer documentation more efficiently.
So while robots.txt tells search engines where not to go, and sitemap.xml tells them what to index, llms.txt gives LLMs a tidy little map of how to reason about your content — particularly APIs, docs, and structured resources.
💡 But is it useful for every website???
Unless your site has detailed developer documentation (like API references or SDK guides), you probably don’t need one. I don’t. My site isn’t built for devs — so this file would just be empty calories for an AI. But if you're shipping developer tools, running a docs platform, or just generally want to stay ahead of the curve — this is worth knowing.
Wait, where did this even come from?
On Nov 14th, Mintlify — a docs platform — added automatic llms.txt support to thousands of their hosted docs (think Cursor, Anthropic, etc).
That move kicked off a mini wave. Devs started posting about it. Community sites popped up. Tools were spun up to help generate your own (like this one by dotenvx).
It went from random GitHub proposal to “everyone’s talking about it” overnight.
You can even browse a full directory of sites with llms.txt support here.
What does a LLMs.txt file actually look like?
Here's the basic structure:
Check out Anthropics: https://docs.anthropic.com/llms.txt TL;DR
LLMs.txt = sitemap for AI systems
Seems to only makes sense if you have developer-focused documentation
Doesn't replace SEO files like sitemap.xml or robots.txt — it complements them
Not auto-crawled (yet), so you’ll need to copy/paste the content to your LLM if you want it to reason over it
Important to know for LLMcoders! Instead of asking your agent to set up OAuth for example, feed it googles llms.txt file then ask it to do it. The file will help the Agent navigate the developer docs.
✨ My takeaway?
Another small-but-smart tool emerging in the post-ChatGPT web. Not relevant for me right now, but I’m glad I now know what it is — and maybe you are too.
If you’ve got dev docs or a product that works with APIs, you might want to get ahead of this one.
If you’re just a curious builder like me — now you can save yourself the rabbit hole.