Should Websites Implement llms.txt in 2026?

2024 marked the initial buzz around llms.txt, which was proposed as a way to help LLMs better understand the content within a website’s pages. 

Several industry-recognized SEOs and developers joined the hype, endorsing it as a “possible missing link between websites and generative AI systems.” 

Today, we’re in 2026, and the shiny veil once adorned by llms.txt is no longer there. Despite this, we see sites implementing this file and investing development resources in it. 

That’s what we’re here to answer. Discover what llms.txt actually is, how it helps AI systems, and why websites use it. 

So without further ado, let’s get right into it!

What is llms.txt?

llms.txt is a proposed website file designed to help large language models (LLMs) explore and understand the content on a website.

The concept was proposed by Jeremy Howard in September 2024. The objective of his proposal was to compile all the information — particularly that which you want LLMs to discover — into a single, accessible file. This is how llms.txt works for AI bots: it provides them with a reference file that points them to the information that matters.

When AI crawlers land on a site, they spend a significant amount of time exploring pages and indexing them. However, with no clear guidelines on what’s important and what isn’t, they can pick information from outdated blog posts or old pages. Using llms.txt can solve this problem. 

This standard presents a way to provide AI crawlers with a curated file listing your most important content. This way, they won’t have to scan through complex HTML pages to render useful information. 

On several occasions, llms.txt was compared to robots.txt. We concur that the confusion was primarily due to the file type. Robots.txt and XML sitemaps are web standards that contribute to efficient crawling and indexation of a website. The robots.txt file lists all URLs that are not to be indexed by search engines. On the contrary, llms.txt is a proposal file supplied to AI systems for information extraction. 

Structure of llms.txt File

The official proposal emphasizes that websites add a /llms.txt markdown to make their content LLM-friendly. It also indicates that an llms.txt file is created using markdown formatting. 

The file exists in a website’s root and is essentially a document listing all important content on a website. The root file also links out to page markdown files, so AI crawlers get all the information about a website in one place. 

The root file contains a website’s background information and links to other markdown files across the site. The proposal also encourages creating separate markdown files for pages containing information you specifically want LLMs to discover. 

In order to apply this, your site needs to use plain text formatting syntax consistent with the markup language. Part of knowing how to do llms.txt implementation is understanding which markdown elements to use: 

Here’s an example of how it’s executed:

These elements represent the most basic components of an llms.txt file. However, for larger websites with a more complex content framework, there may be other, more structural elements required to reflect sections. These could be subheadings (H4, H5), tables, or code snippets. 

What is the purpose of llms.txt?

• Make website content readable not just for humans but for LLMs as well.
• Define which AI bots or systems are permitted to access your site.
• Identify the pages, data sources, or APIs that AI tools are authorized to access.
• Include relevant metadata or content referencing conditions for AI tools, such as licensing details, attribution requirements, or access limits.

Do Websites Actually Use llms.txt?

We can’t say for sure whether LLMs use this file to find information, but there have been some studies and data points that can help determine the scope of adoption.

SE Ranking carried out a survey analyzing 300k domains. In it, they found that llms.txt had a 10.13% adoption rate. That’s quite low considering how quickly site owners are leaping to tap into AI search engines, like ChatGPT and Claude. Since the focus hasn’t yet shifted to llms.txt, its effectiveness is still questionable. 

Meanwhile, looking at the stats from the same study, we came to the conclusion that medium- and low-traffic sites have a higher adoption rate for llms.txt. Whereas authoritative sites did not deem it worth their resources, despite having larger teams and budgets.

Now shifting over to the SEO community forums, it seems people aren’t so thrilled with llms.txt proposal. Some even went as far as saying that. 

That said, the proposal isn’t widely adopted yet. While some resort to testing, others don’t find it worthy enough for all the time and effort it demands. Adoption is relevant to willingness. Because llms.txt has yet to become a web standard like robots.txt.

Do LLMs Use llms.txt?

The adoption of any new standard relies on its tangible benefits. In this case, an increase in AI search visibility or traffic. 

If LLMs are using this file to extract information, it would justify implementing this standard. But what percentage of LLMs are using it? Is there a solid reason to believe that llms.txt is necessary for SEO?

According to Search Engine Land, 8 out of 9 sites saw no measurable change in traffic after LLM.txt implementation. 

John Mueller also reinforced this point, saying that none of the AI crawlers have claimed they extract information via llms.txt yet. Here’s how he responded to a Redditor complaining he hasn’t seen any impact on crawl logs after llms.txt submission. 

“AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it).” 

Moreover, Mark Wliliams Cook came to address the same idea in the following words. 

To expand on this perspective, let’s glance at what the major AI search engines or service providers have to say about this. 

• Google continues to claim that its organic citations within AI Overviews and AI Mode are contingent on traditional SEO standards and signals. In contraction to that, in December 2024, they added llms.txt files across their developer and documentation sites. However, the file was retrieved within 24 hours. 
• In the OpenAI disclaimer on web crawlers and user agents, there is no mention of llms.txt, but they do emphasize allowing OAI-SearchBot access to their robots.txt and a list of their published IPs.

There’s also the flip side of the coin, where an alternate perspective is brewing, one where traffic pipelines are flourishing after applying llms.txt. 

• Ray Martinez published the llms.txt file and tracked a GPTBot crawling that file the very next day. 
• In Nov, 2024, Claude listed llms.txt and llms-full.txt in their official documentation, reflecting a clear endorsement of these standards from a major player in the AI industry. 
• Mintlify noted 436 visits to their website from AI crawlers after applying llms.txt in their domain. Most of this traffic, as they say, came from ChatGPT. 

So it’s safe to say that llms.txt is a polarized topic one can hardly pass a clear judgment on—yet. While the proposal is targeted towards LLM or agent retrieval, whether it causes any real shift in traffic is still debatable. To be clear, the tested data sets aren’t large enough to prove a real impact on citation frequency, traffic, and clicks. 

The Final Verdict

So it’s not definitive yet if websites should actively implement llms.txt. They either side with those who consider llms.txt a revolutionary move or the countless lot who view it as a hypothesis with no clear evidence to support its standardization.

Keep in mind that server logs barely show AI crawlers requesting this file, but plugins like Yoast have already started introducing one-click auto-generated llms.txt infrastructure. Webflow even provides a system to upload the file to your root. These things keep raising the same question, and only time will tell where the tides turn. 

In the end, it all comes down to whether you have the resources, development budget, and operational justification to get it done. 

It’s also important to gauge if this should be your priority at the moment, or whether you can invest those resources in something more tangible that provides definitive results. We’ve already seen a trend where website size doesn’t influence llms.txt implementation. Many sites keep it on the burner, waiting for it to turn into a web standard until they proceed with implementation.