Search engine optimization (SEO) is changing before our eyes. While for the past 20 years we optimized primarily for Googlebot, today entirely new visitors access our websites too – users and bots of language models (LLMs) such as ChatGPT, Claude or Perplexity.
And they “read” the web differently than a classic crawler.
Recently a new standard has been much discussed in the community: the llms.txt file. In this article we’ll look at what exactly it is, why you should care and how to handle it technically without spending days on it.
What is llms.txt?
llms.txt is a text file in Markdown format that you place in the root directory of your website (similar to robots.txt). Its goal is to give AI models a concise, clean and clear overview of who you are, what you do and where to find your most important content.
Think of it as a structured résumé of your website intended directly for artificial intelligence.
Why classic tools aren’t enough
You might be saying that you already have robots.txt and sitemap.xml. Why another file?
- robots.txt tells robots where they must not go. It is a prohibition sign.
- sitemap.xml is a list of URLs for indexing, but says nothing about the context and importance of the content.
- llms.txt tells AI models: “This is what matters. Here are my key products, here is the price list and here is the documentation. Ignore the clutter around it.”
Language models have limited resources (the so-called context window). If a crawler arrives at your website full of JavaScript, pop-ups and complex coding, it may use up its capacity before it finds what matters.
llms.txt serves the content to them on a silver platter in the language they understand best – Markdown.
Do I have to add llms.txt to my website?
Not yet, llms.txt is currently a so-called “proposed standard”. It is not something required as a condition for your website to appear in search engines and AI answers.
Representatives of Google, OpenAI, Anthropic etc. have so far been rather reserved about it and have not officially stated that they use this file.
So why do I recommend that clients deploy it?
Because it costs nothing and the potential benefit is large.
- All AI models are desperately looking for structured data that would make gathering information about websites less costly.
- Anthropic (Claude) already uses this file on its own website, which suggests that their crawler may take it into account. Some SEO specialists and website owners have even already recorded various models accessing these files.
- You’re preparing for the future. When it becomes a standard, you’ll already have it.
It’s a bet with minimal risk and almost zero cost that can give you a head start in optimizing your website for visibility in AI models.
What the file should look like
The structure is simple. No complicated code, just headings and links.
Example of the file’s content:
Markdown
# Your Company Name
> We are a leading Czech agency focused on data analytics and business intelligence.
## Key services
- [Web analytics audit](https://yourdomain.com/services/audit): A complete check of GA4 tracking.
- [Server-side GTM implementation](https://yourdomain.com/services/gtm): More accurate measurement for e-shops.
## Why work with us
- [Our case studies](https://yourdomain.com/references): Real examples of our work.
- [About us](https://yourdomain.com/about-us): Our team and certifications.
Guide: How to add the llms.txt file to your website
You have three main options depending on what your website runs on and how technically skilled you are.
1. Your website is on WordPress
This is the easiest route. If you use one of the big SEO plugins (e.g. RankMath or Yoast), you can set up generating the llms.txt file in them.
If you don’t have an SEO plugin, look for dedicated free plugins for llms.txt.
2. Using a generator (static solution)
If you don’t have a website whose content changes every day, you can generate the file and upload it statically via FTP to the root folder (where the wp-content and wp-admin folders are). There are online generators that will help you create the structure.
However, this solution is “one-off”. If you add a new important service or article to your website, you have to edit the llms.txt file manually. If you forget, AI models will receive outdated information.
3. Custom solution (for advanced needs)
If you have a custom e-shop, Shoptet (where you don’t have direct access to the root) or a more complex web application, you need a dynamic solution. The ideal state is when llms.txt is generated automatically from your database – similar to sitemap.xml.
This often requires a developer’s intervention. If you want the file implemented correctly, to strategically choose what to show AI, and you don’t know how to go about it – get in touch. I’ll gladly help you with the setup and the overall marketing strategy for your website.
How do I measure whether deploying the file helped me?
In marketing the rule is: what I don’t measure, I don’t manage. With llms.txt measurement is a bit more specific, because it does not (yet) show up in the regular Google Search Console reports.
So you need to analyze your access logs (server access logs).
In your server log you can filter accesses to the /llms.txt file. Look for the specific user agents of AI bots. If you see accesses from the following robots in the log, you know they are aware of your file:
GPTBot/OAI-SearchBot(OpenAI)ClaudeBot(Anthropic)Google-Extended(Google Gemini)PerplexityBot
llms.txt file every 15 minutes.The fact that a robot visited the file doesn’t give you a 100% guarantee that it will use its content to generate answers. But it is a much better starting position than when the robot knows nothing at all about the structure of your website.
In online marketing, those who take that one extra step that the competition considers unnecessary often win. Deploying llms.txt is exactly such a step.
