If you work in SEO, GEO, or AI search optimization, you have probably seen the same recommendation repeatedly:
“Add an llms.txt file to your website so AI can understand it.”
It sounds logical.
We already have robots.txt for crawlers and sitemap.xml for search engines. So a dedicated file for large language models seems like an obvious next step.
But there is an important question:
Are ChatGPT, Google, Perplexity, Claude, and other AI systems actually using llms.txt?
In 2026, we finally have enough documentation and real-world crawler data to give a more useful answer.
The short version is:
llms.txt does not currently provide a proven SEO or AI-search ranking advantage. Google explicitly says it doesn’t use the file for Search. AI retrieval crawlers rarely request it. But llms.txt may still have value for AI agents, coding tools, and documentation-heavy websites.
That distinction matters.
Let’s look at what llms.txt actually does, what the latest evidence shows, and whether you should add one to your website.
What Is llms.txt?
llms.txt is a proposed standard for giving large language models a simplified, machine-readable overview of a website.
The proposal was introduced by Jeremy Howard in September 2024.
Instead of asking an AI system to navigate menus, JavaScript, advertisements and hundreds of HTML pages, a website can provide a Markdown file at:
https://example.com/llms.txt
The file can describe the website and point an AI system toward its most important resources.
A simplified example might look like this:
# Example Company
> Example Company provides inventory management software
> for ecommerce businesses.
## Products
- [Inventory Management](https://example.com/inventory)
- [Order Management](https://example.com/orders)
## Documentation
- [API Documentation](https://example.com/docs/api)
- [Developer Guide](https://example.com/docs)
The important point is that llms.txt is a guide, not an access-control file.
It does not tell crawlers what they are allowed to crawl.
That is still the job of robots.txt.
llms.txt vs robots.txt vs sitemap.xml
These files are frequently confused, but they solve different problems.
| File | Main purpose |
|---|---|
robots.txt | Controls crawler access |
sitemap.xml | Helps search engines discover URLs |
llms.txt | Provides a curated overview intended for LLMs and agents |
The original llms.txt specification itself describes the file as something that can coexist with sitemaps and robots.txt rather than replace them.
That means adding:
/llms.txt
does not automatically make your content available to ChatGPT.
And it does not override:
/robots.txt
If an AI crawler cannot access your pages because of your crawler configuration, having a perfect llms.txt file will not solve the problem.
Why Did llms.txt Become So Popular?
The idea fits perfectly with the rise of Generative Engine Optimization (GEO).
Businesses increasingly want their content to appear in:
- ChatGPT
- Google AI Overviews
- Google AI Mode
- Perplexity
- Claude
- AI-powered browser agents
- other generative search experiences
Naturally, website owners started looking for an equivalent of a “sitemap for AI.”
llms.txt looked like the answer.
Interest has grown rapidly. Ahrefs reported in 2026 that searches for “llms.txt” increased 154% year over year, while searches for “what is llms.txt” increased even faster.
But popularity does not necessarily mean adoption by AI platforms.
And this is where the data becomes interesting.
The 2026 Data: 97% of llms.txt Files Received Zero Requests
One of the largest studies so far came from Ahrefs in June 2026.
Ahrefs analyzed 137,210 domains using its Web Analytics and Bot Analytics data.
Around 28% of those domains had a valid llms.txt file.
But among the sites that had one:
97% of llms.txt files received zero requests during May 2026.
No AI crawler.
No regular crawler.
No human visitor.
Nothing.
Only around 3% of the llms.txt files in the study received any requests at all.
There is an important limitation here: these websites were Ahrefs Web Analytics users, so the sample is likely more technical and SEO-aware than the average website.
Still, the scale gives us a useful picture of actual crawler behavior.
And the most interesting finding came from looking at who was requesting the files.
AI Search Crawlers Barely Used Them
Among llms.txt files that received traffic, Ahrefs found that named AI tools accounted for only part of the requests.
AI retrieval crawlers—the systems most relevant to appearing in live AI search results—were particularly uncommon.
Ahrefs classified just 1.1% of requests as coming from AI retrieval bots such as OAI-SearchBot and PerplexityBot.
Meanwhile, AI agents and agentic infrastructure accounted for considerably more activity.
That leads to an important distinction:
llms.txt may be more interesting for AI agents than for AI search engines.
And recent Google documentation supports that distinction.
Does Google Use llms.txt?
For Google Search, the answer is now unusually clear.
No.
Google’s current documentation explicitly says you do not need llms.txt, special AI text files, Markdown versions, or additional machine-readable files to appear in Google Search or its generative AI experiences.
Google goes further and states that maintaining an llms.txt file will neither help nor hurt your visibility or rankings in Google Search.
So if someone tells you:
“Add llms.txt to improve Google AI Overview rankings.”
There is currently no basis for that claim.
For Google Search, your effort is better spent improving the fundamentals:
- crawlability
- indexing
- internal linking
- technical SEO
- useful original content
- clear site architecture
- structured data where appropriate
- entity consistency
Those signals already work across traditional and generative Google Search experiences.
But Chrome Checks for llms.txt — Isn’t That a Contradiction?
This is where things become more interesting.
While Google Search says it doesn’t use llms.txt, Google’s Chrome team added an llms.txt check to its experimental Agentic Browsing Lighthouse audits.
Chrome describes llms.txt as an emerging convention that can give AI agents a machine-readable summary of a website.
According to the documentation, without such a file an agent may need to spend more time crawling a website to understand its structure.
But Chrome also clearly marks the file as optional. If /llms.txt returns a normal 404, the Lighthouse audit is marked “Not Applicable,” rather than failed.
There isn’t really a contradiction once we separate the two use cases.
Google Search
Needs to discover, index and rank content.
llms.txt: not used.
AI agents
May need to quickly understand how a website or documentation system is organized.
llms.txt: potentially useful.
That distinction is likely to become increasingly important as the web moves from search engines that simply retrieve pages toward agents that actually interact with websites.
What About ChatGPT?
OpenAI’s current publisher guidance gives us another useful signal.
If you want website content to be discoverable and surfaced in ChatGPT search, OpenAI recommends ensuring that:
OAI-SearchBot is not blocked.
OpenAI specifically tells publishers to review their robots.txt configuration and allow OAI-SearchBot access when appropriate.
Notice what isn’t part of the requirement:
llms.txt.
OpenAI’s documented route for ChatGPT search discoverability currently revolves around crawler accessibility rather than publishing an llms.txt file.
That doesn’t prove OpenAI systems will never use llms.txt.
In fact, Ahrefs observed some OpenAI-related crawlers requesting existing llms.txt files.
But there is currently no evidence that simply publishing the file improves your probability of appearing in ChatGPT answers.
Does llms.txt Help GEO?
This is where I think the terminology needs to be precise.
If by GEO you mean:
“Will adding llms.txt increase my rankings or citations in ChatGPT, Google AI Overviews or Perplexity?”
The evidence today says:
Probably not.
There is no established ranking benefit.
Google explicitly ignores it for Search.
And live AI retrieval crawlers appear to request the file very rarely.
But if your definition of GEO is broader:
“Can I structure my website so AI systems and future agents can understand it more efficiently?”
Then llms.txt becomes more interesting.
It may provide an additional machine-readable layer for systems that choose to consume it.
The key word is:
additional.
Not foundational.
When llms.txt Actually Makes Sense
I would consider adding an llms.txt file in several situations.
1. You Have Developer Documentation
This is probably the strongest current use case.
If your website contains:
- API documentation
- SDK documentation
- software libraries
- technical references
- integration guides
an llms.txt file can give AI coding tools a curated path through the information.
This is also close to the use case envisioned by the original specification.
2. Your Website Is Large or Structurally Complex
Suppose you have thousands of pages but only 30 are essential for understanding your business.
A curated llms.txt can point agents toward:
- main products
- documentation
- pricing
- policies
- support content
- important resources
Instead of functioning like a sitemap of everything, it acts more like a machine-readable table of contents.
3. You Are Building for AI Agents
This may become the most important use case.
AI agents don’t just search.
They may:
- navigate websites
- compare products
- complete forms
- retrieve documentation
- interact with applications
- perform tasks on behalf of users
Chrome’s experimental agentic browsing work shows that the web is already moving in this direction.
For websites preparing for that environment, providing an llms.txt file is inexpensive experimentation.
When I Would Not Prioritize llms.txt
I would not make llms.txt a priority if your website still has basic technical problems.
For example, fixing these matters much more:
1. AI crawlers are accidentally blocked
Check your robots.txt.
If OAI-SearchBot cannot crawl your site, creating another text file does not fix the underlying issue.
2. Important pages are not indexable
Check canonical tags, noindex, rendering and HTTP responses.
3. Your site architecture is unclear
Important pages should be reachable through logical internal links.
4. Your content doesn’t clearly answer real questions
A machine-readable summary cannot compensate for weak source content.
5. Your brand/entity signals are inconsistent
Your company, products, people and expertise should be described consistently across your website and authoritative external sources.
6. Your structured data is incomplete
Structured data is not a special GEO hack, but it remains useful for helping search systems interpret eligible entities and page types.
Google specifically recommends continuing normal SEO best practices rather than creating separate content purely for generative AI.
My GEO Priority Order in 2026
If I were auditing a website for AI search discoverability today, I would prioritize the work roughly like this:
1. Crawl accessibility
Can Googlebot, OAI-SearchBot and other relevant crawlers actually reach the content?
↓
2. Indexability and rendering
Is the important information available in crawlable HTML?
↓
3. Clear information architecture
Can machines and humans easily understand how pages relate?
↓
4. High-quality source content
Does the website contain information worth citing?
↓
5. Entity clarity
Is it obvious who the company/person/product is and what it does?
↓
6. Structured data
Use appropriate schema to reinforce meaning.
↓
7. Internal linking
Connect important concepts, services and supporting content.
↓
8. AI-agent accessibility
Semantic HTML, accessibility structure and predictable interactions become increasingly important here.
↓
9. llms.txt
Add it as an optional machine-readable layer.
That is very different from treating llms.txt as the foundation of GEO.
Should You Create an llms.txt File?
My answer in 2026 is:
Yes—if it is easy to maintain. But don’t expect an SEO boost from it.
For a small website, manually maintaining another file may not be worth the effort.
For a CMS, documentation platform or application where the file can be generated automatically, the cost is almost zero.
In that case, publishing one can make sense as a form of future-proofing.
But I would not spend hours building an elaborate llms.txt strategy while ignoring technical SEO, crawlability, content quality or entity signals.
Think of it as:
Nice to have, not must have.
A Simple llms.txt Example
You don’t need hundreds of links.
A useful file should be concise and intentional.
# Acme
> Acme builds inventory management software for ecommerce businesses.
## Products
- [Inventory Management](https://example.com/inventory):
Real-time stock management across stores and warehouses.
- [Order Management](https://example.com/orders):
Centralized ecommerce order processing.
## Resources
- [Documentation](https://example.com/docs):
Product and API documentation.
- [About](https://example.com/about):
Company information and expertise.
## Optional
- [Blog](https://example.com/blog):
Articles about ecommerce operations and inventory.
The official proposal recommends concise language, informative link descriptions and avoiding unnecessary jargon.
That’s a better approach than dumping your entire sitemap into another text file.
Common llms.txt Mistakes
Treating it like robots.txt
llms.txt does not grant or deny crawler permissions.
Use robots.txt for crawler access.
Assuming Google requires it
Google explicitly says it doesn’t.
Automatically adding every URL
The purpose is curation, not duplication of your sitemap.
Creating special “AI content”
Your primary website content should remain the source of truth.
Ignoring the actual crawlers
If AI visibility matters, inspect your server logs and understand which crawlers are actually accessing your website.
Real crawler behavior is more useful than assumptions about what bots might do.
Frequently Asked Questions
What is llms.txt?
llms.txt is a proposed Markdown-based standard that lets a website provide LLMs and AI agents with a concise description of the site and links to important resources.
Does llms.txt improve SEO?
There is currently no evidence that llms.txt improves traditional search rankings. Google explicitly says the file does not positively or negatively affect Google Search visibility.
Does Google use llms.txt?
Google Search says it does not use llms.txt. Chrome’s experimental agentic browsing tools, however, recognize the file as an optional mechanism that may help AI agents understand websites.
Does ChatGPT use llms.txt?
OpenAI has not made llms.txt a requirement for ChatGPT search discoverability. Its current publisher guidance instead recommends allowing OAI-SearchBot to crawl relevant website content.
Is llms.txt the same as robots.txt?
No.
robots.txt manages crawler permissions.
llms.txt provides optional contextual information and links for systems that choose to read it.
Should every website have llms.txt?
Not necessarily.
It currently makes the most sense for documentation-heavy websites, developer platforms, complex websites and businesses experimenting with AI-agent discoverability.
Final Verdict: llms.txt Is Not a GEO Shortcut
llms.txt is an interesting idea.
But in 2026, the hype is ahead of the evidence.
Google Search does not use it.
OpenAI does not require it for ChatGPT search.
And one large-scale analysis found that 97% of existing llms.txt files received no requests at all during the month studied.
At the same time, dismissing the idea completely would also be premature.
Chrome’s agentic browsing work and the higher proportion of requests from AI-agent infrastructure suggest that llms.txt may eventually find its place—not primarily as an SEO file, but as an agent-discovery and context file.
So my recommendation is simple:
Fix the fundamentals first. Make your site crawlable, understandable, authoritative and technically accessible. Then add llms.txt if it is inexpensive to maintain.
That’s a much safer GEO strategy than chasing a file simply because everyone else is adding one.
The future of AI discovery probably won’t be decided by a single text file.
It will be decided by how easily machines can access, understand, trust and use the information already on your website.
Comments