Should you block AI crawlers? For most businesses that is a mistake
The short version. More myth than practice has grown around blocking AI crawlers. llms.txt is not an official standard, adoption sits near ten percent of domains and measurements put it at roughly 0.1 % of AI bot requests, meaning almost nothing reads it. For small and local businesses the answer is usually clear: let them crawl, because being cited in AI answers is worth more than protecting text from being read.
What llms.txt is and is not
It is a community proposal for a file summarising a site for language models. Reasonable in theory, but without official backing from the major AI companies and barely used in practice.
The most common misconception is that adding llms.txt gives you control over whether your content is used for training. It does not. If someone sells it to you as a solution, they are selling peace of mind, not a feature.
The second misconception is that a robots.txt block is a clean opt-out. It is not. It handles well behaved crawlers and does not cover every path by which content reaches a model.
What actually works
Google Search Console. Since 17 June 2026 Google honours a toggle that removes a site from appearing in and grounding generative AI features. That is the most concrete control available.
Separating training from search. Crawlers can be told apart. You can disallow training crawlers such as GPTBot, Google-Extended or ClaudeBot while leaving search-serving agents alone.
How to decide
This is a commercial question, not an ideological one. Ask yourself: is your content the product, or is it marketing?
| Situation | Recommendation |
|---|---|
| Local business, services, trades, e-shop | allow, you want to be in the answers |
| B2B company selling solutions | allow, AI answers are a new top of funnel |
| Publisher living off traffic | consider blocking, here content really is the product |
| Databases, courses, paid content | protect behind a login, not robots.txt |
For most companies the answer is allow. When a customer asks an assistant for a recommendation, the shortlist comes from what the models have read. Blocking removes you from a growing channel and does not give you the traffic back.
What to do today
- Read your robots.txt. Surprisingly often there is a block nobody knows about, left by a previous supplier.
- Decide deliberately using the table above and write down why.
- If you want to limit training, disallow specific crawlers and keep the search ones.
- Check the load. Some crawlers are aggressive, and then a rate limit is the answer, not a ban.
- Watch who visits. Server logs give you the real picture instead of guesses.
This is exactly the kind of thing that belongs in ongoing website care: set once, review regularly.
FAQ
Should I add llms.txt? It does no harm, but expect no effect. It does not replace clear, readable content.
How do I stop my text being used for training? You cannot guarantee it. Disallowing specific crawlers plus the Search Console toggle is today’s maximum. Anything that must stay private belongs behind a login.
Will crawlers slow my site? They can. The fix is rate limiting or caching, not a blanket ban.
Will I lose traffic by allowing them? Some informational queries get answered either way. The difference is whether your name or a competitor’s is mentioned in that answer.
This summarises publicly available sources as of 28 August 2026. Tools and rules change, review your setup periodically.
