Ansar Bhatti
If you follow AI search news, you have probably seen advice to add an llms.txt file to your website. Some people call it the new robots.txt. Others call it a distraction. This guide explains what llms.txt is, what it is supposed to do, what evidence exists that it works, and whether your site actually needs one. The short version: it is cheap and harmless to add, but it should never replace the fundamentals that really move visibility.
What is llms.txt?
llms.txt is a proposed standard for a plain text file, written in Markdown, that sits at the root of your site at /llms.txt. It was proposed in 2024 as a way to give language models a clean, short guide to your most useful content. Instead of making a model dig through navigation, scripts and ads, the file lists your key pages with a brief description of each.
A typical file includes:
- A title with your site or brand name.
- A short summary that explains who you are and what you do.
- Sections with links to important pages, such as services, pricing, guides and documentation.
- Optional links to extra material that can be skipped when space is limited.
A related idea is llms-full.txt, a longer file that contains the actual text of your key content in one place so a model can read it without crawling many pages.
How is it different from robots.txt and sitemap.xml?
| File | Main job | Format | Status |
|---|---|---|---|
| robots.txt | Tells crawlers what they may or may not fetch | Plain text rules | Long-established standard |
| sitemap.xml | Lists URLs so search engines can discover them | XML | Long-established standard |
| llms.txt | Points models to your best content with short descriptions | Markdown | Community proposal, not an official standard |
Notice that llms.txt does not block or allow anything. It is a recommendation list, not an access control. If you want to control which AI crawlers can visit, you still use robots.txt and server rules.
Do AI systems actually use llms.txt?
This is the question that matters, and the honest answer is: there is no strong public evidence that major AI search products rely on it today. Google representatives have said Google does not use llms.txt for Search, and the large AI providers have not published commitments to read it for ranking or citing sources. Some developer tools and AI coding assistants do make use of it, especially for documentation, because it gives them a neat index of a project.
So treat llms.txt as a low-cost bet on the future, not as a proven ranking factor. Anyone who tells you that adding it will make ChatGPT or Google cite you is overselling.
Does your site need an llms.txt file?
It depends on your site type and how much effort you want to spend.
- Documentation and developer sites. Most likely to benefit, because tools that read docs can use the index directly.
- Service businesses and agencies. Low benefit, but cheap to add, and a useful exercise because it forces you to decide which pages matter most.
- Large content or ecommerce sites. Possible benefit if you can keep it accurate. Do not list thousands of URLs.
- Tiny sites. Probably not worth much time, since there are only a few pages to find anyway.
The main risk is neglect. A file that links to deleted pages or outdated prices is worse than no file.
How to create an llms.txt file
1. Pick your most important pages
Choose 10 to 30 pages that best describe your business and help users: key services, pricing, about, contact, top guides and policies. Skip tags, archives, thank-you pages and duplicates.
2. Write the file in Markdown
Use a heading for the site name, a one or two sentence summary, and sections with links and short notes. Here is the structure:
# Your Brand Name
> One or two sentences about what your business does and who it serves.
## Services
- [Service name](https://example.com/service/): What this page explains
## Guides
- [Guide title](https://example.com/guide/): What the reader learns
## Optional
- [Privacy policy](https://example.com/privacy/)3. Publish it at the root
The file must be reachable at https://yourdomain.com/llms.txt and return plain text. On WordPress, Rank Math includes an llms.txt module you can switch on, or you can upload the file to the site root over SFTP. Check that it loads in a browser and returns a 200 status.
4. Keep it up to date
Review it whenever you publish a major page, change prices or remove content. Add a calendar reminder every quarter.
Best practices
- Use clear, factual descriptions. No hype.
- Link only to pages that return a 200 status and are indexable.
- Keep descriptions consistent with your page titles and schema.
- Do not stuff keywords or try to “instruct” AI systems. It is a guide, not a place for prompts.
- Do not put private or sensitive URLs in it. The file is public.
What matters more than llms.txt
If your goal is to appear in AI answers, spend your time on the things that have real support behind them:
- Pages that can be crawled, indexed and loaded quickly. See our technical SEO checklist.
- Clear, answer-first content, as described in our AEO guide.
- Consistent brand facts and credible third-party mentions, which are the heart of GEO.
- Accurate structured data, covered in our schema markup guide.
Our guide to getting mentioned in ChatGPT shows how these pieces fit together, including which AI crawlers to allow in robots.txt.
Frequently asked questions
Is llms.txt an official standard?
No. It is a community proposal. It has gained adoption in some developer and documentation communities, but it is not governed by a standards body.
Will llms.txt improve my Google rankings?
There is no evidence that it does, and Google has said it does not use it for Search. Do not expect ranking changes from adding it.
Can llms.txt block AI crawlers?
No. It does not control access. Use robots.txt or server rules for that.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a concise index with links. llms-full.txt is a longer file containing the actual text of your key content. Most sites only need the first one, if any.
How long should the file be?
Short. A few dozen well-chosen links with clear notes is better than a long dump of every URL.
Also read: AI crawlers in robots.txt.
If you want the file written, published and tested for you, see our llms.txt setup service.
Need a plan for AI search visibility?
llms.txt is a small piece of a bigger picture. If you want a clear plan covering technical health, answer-ready content and brand signals, contact us or read about our GEO services.
