Llms.txt is a proposed web standard that works to make websites more accessible for LLMs. If you want to give LLMs a way to discover and understand your website’s content better, it may be time to implement llms.txt file.
Step-by-Step Process to Implement llms.txt
With this process, you’re essentially converting lengthy HTML pages into LLM-friendly content that’s free of complex JavaScript and navigation. The idea is to offer LLMs all important content as one file they can render and index.
Implementing llms.txt might require you to have a developer by your side, but here’s a step-by-step guide to get you started.
Determine File Content
What do you want AI crawlers to see on your site? The answer to this question will determine which sections of your website are displayed in the llms.txt file.
For individual pages, this could vary. For example, if you want one specific section of a blog to get highlighted, you can include that in your page markup. For website schema, it should include all the key pages within a website, particularly those containing important business information like the Contact page, About Us page, and Service pages.
Structure The File
The official proposal of llms.txt by Jeremy Howard specifies which sections are to be included in the markdown. This is a generic guideline for the structure of a file available on a website’s root path. It should include the following sections as markdown and in the same order as listed.
- A mandatory H1 listing the website name or brand name.
- A summary of the headline containing contextual information to interpret the remaining file content. This needs to be in a blockquote.
- Markdown sections to highlight page elements like paragraphs, bullet points, etc. This is where the document goes into further details to enable interpretation of the file content.
- A list of files (containing page markdown) separated by H2 headers. This is where the dissection happens, when the root file redirects crawlers to other pages. Each of the file lists carries a markdown hyperlink [name](url). You can also add additional notes using the colon (:).
Get A Developer To Write the Code
With a set structure for your llms.txt, it’s time to write down your code.
Use the notepad editor or Visual Studio Code to record the markdown structure. Save using the file name “llms.txt.”
You’d probably need the assistance of a developer in this step. They’ll help create a file that’s ready to upload to the website.
For a file containing website information of the entire site, it would be accessible as “https://website.com/llms.txt”. Review this llms.txt file from FastHTML to see how it’s implemented practically.
On the other hand, if you’re creating a markdown page, it would have .md appended with a structure similar to what is seen in this file.
Upload File to Website
The file will either be uploaded to the root directory or a subdirectory, depending on what it’s about. The distinction corresponds to what the domain looks like. For example:
- https://website.com/llms.txt – uploaded in root directory
- https://docs.website.com/llms.txt – uploaded in the corresponding subdirectory
The first one contains website information, whereas the second file is particularly about a specific document within the website. Both of them are uploaded to your server via the control panel of your hosting provider. Follow these steps to upload.
- Log in to your hosting provider dashboard.
- Find File Manager in the cPanel. Click to open a window listing your website files.
- If the file is designated for the entire site, navigate to the public_html/ folder. In case it’s for a subdomain, find the subdirectory folder dedicated to the specific subdomain.
- Use the Upload button at the top of the file manager to upload your llms.txt file and save.
- Visit the URL in a new tab to verify that the llms.txt file is uploaded.
Bonus Points:
Here are a few bonus points to help you successfully implement llms.txt and ensure your objective is fully achieved.
- Structure sentences such that they are easy to understand. Steer clear of jargon terms.
- Always add a brief description when linking to resources in your llms.txt file.
- Keep the file updated by removing outdated pages and adding new relevant links.