Ansar Bhatti
Short answer: to allow AI crawlers on WordPress, make sure your robots.txt does not disallow them and that no security plugin, firewall or CDN rule blocks them. To block them, add a separate User-agent group with Disallow: / for each crawler. The key is to decide per crawler, because some are used for training and others for live AI search, and blocking the wrong one can remove you from AI answers. This guide lists the main AI crawlers, gives copy-ready robots.txt examples and shows how to test your setup.
The main AI crawlers and what they do
| User agent | Company | Main purpose |
|---|---|---|
| GPTBot | OpenAI | Collecting content that may be used to train models |
| OAI-SearchBot | OpenAI | Finding pages for ChatGPT search results |
| ChatGPT-User | OpenAI | Fetching a page when a user asks ChatGPT to open it |
| ClaudeBot | Anthropic | Collecting content that may be used for model training |
| Claude-SearchBot | Anthropic | Improving search result quality for Claude users |
| Claude-User | Anthropic | Fetching pages on behalf of a user |
| PerplexityBot | Perplexity | Indexing pages so they can appear in Perplexity answers |
| Perplexity-User | Perplexity | Fetching pages when a user asks a question |
| Google-Extended | A control token for whether content is used for Gemini training and grounding. It is not a separate crawler | |
| Applebot-Extended | Apple | A control token for Apple AI training |
| CCBot | Common Crawl | Builds a public web archive used in many AI datasets |
Company documentation changes over time, so check each provider’s current crawler page before making a final decision. Two important notes: Google AI Overviews and AI Mode use the normal Googlebot index, so blocking Google-Extended does not remove you from them, and you cannot opt out of AI Overviews without also affecting normal Search snippets.
Should you allow or block AI crawlers?
- Allow search and user crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and similar) if you want to be cited and recommended in AI answers.
- Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) are a business decision. Blocking them keeps your content out of future training data but does not directly remove you from live AI search.
- Block everything only if you have a strong reason, such as paid content or licensing concerns. Understand that you may disappear from AI answers.
For most small businesses that want customers to find them, allowing the search and user crawlers is the sensible default. See our guide to getting mentioned in ChatGPT for the bigger picture.
Where robots.txt lives in WordPress
WordPress generates a virtual robots.txt at yourdomain.com/robots.txt unless a physical file exists in your site root. You can edit it in three ways:
- SEO plugin editor. Rank Math has a robots.txt editor under General Settings, and Yoast has a file editor under Tools. This is the easiest option.
- A physical file. Upload a robots.txt file to the site root by SFTP. It overrides the virtual one.
- A filter in code. Developers can use the
robots_txtfilter. Keep this in a small plugin, not in the theme.
Robots.txt examples you can copy
Allow all AI search and user crawlers (default)
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xmlIf you have no rule that mentions the AI crawlers, they follow the rules for User-agent: *, which allows crawling of public pages.
Block training crawlers but allow AI search
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Block one crawler only
User-agent: GPTBot
Disallow: /Block only a private section
User-agent: GPTBot
Disallow: /members/
Disallow: /private-docs/Remember that robots.txt is a request, not a lock. Well-behaved crawlers follow it. It does not protect private content, so use passwords or login protection for anything sensitive.
Check for hidden blocks
Many sites block AI crawlers by accident through layers other than robots.txt:
- Security plugins. Some bot-blocking features treat unknown user agents as threats.
- Firewalls and CDNs. Cloudflare and similar services offer toggles to block or challenge AI bots, and some are on by default for new accounts.
- Hosting-level rules. Some hosts rate-limit or block aggressive bots.
- Rules in .htaccess or Nginx. Old snippets copied from tutorials may deny certain user agents.
If your robots.txt allows a crawler but your firewall blocks it, the crawler still cannot reach your pages. Our guide on how to check if AI bots are crawling your website shows how to test this.
How to test your robots.txt
- Open
yourdomain.com/robots.txtin a browser and read it line by line. - Check there is no blanket
Disallow: /underUser-agent: *. That is a common staging leftover. - Use a robots.txt tester or the robots report in Search Console for Google’s view.
- Request a key page with a crawler’s user agent string using a command-line tool and confirm you get a normal 200 response, not a 403 or a challenge page.
- Re-test after plugin, security or hosting changes.
Common mistakes
- Blocking
User-agent: *withDisallow: /and forgetting to remove it after launch. - Blocking all “bots” in a security plugin and accidentally including AI search crawlers.
- Assuming that blocking GPTBot removes you from ChatGPT search. They are separate crawlers.
- Expecting robots.txt to remove pages that are already indexed. Use
noindexfor that. - Adding crawler rules that conflict, with the more specific user-agent group ignored by mistake.
Does llms.txt replace robots.txt?
No. llms.txt is a proposed content guide for language models and does not control access. See what llms.txt is and whether you need one.
Frequently asked questions
Is GPTBot blocked by default in WordPress?
No. WordPress does not block it by default. If it is blocked, the cause is a rule in robots.txt, a plugin, a firewall or your host.
How do I allow GPTBot in robots.txt?
Do not disallow it. If you have no GPTBot rule and no blanket disallow, it is allowed. You can add User-agent: GPTBot followed by Allow: / to be explicit.
Will blocking AI crawlers hurt my Google rankings?
Blocking GPTBot, ClaudeBot or Google-Extended does not change your normal Google rankings. Blocking Googlebot would.
Can I block AI Overviews with robots.txt?
Not separately. AI Overviews use the Googlebot index, and controls that limit snippets also affect normal results.
How often should I review my robots.txt?
Check it after any redesign, migration, plugin change or hosting move, and at least a couple of times a year.
Need it done for you? See our llms.txt setup service.
Need your crawl access checked?
A single wrong line can hide your whole site from search and AI. Contact us and we will review your robots.txt, firewall rules and indexing, as part of our WordPress SEO services.
