Ansar Bhatti
Short answer: you can check whether AI bots are crawling your website by looking for their user agents in your server access logs, by checking your CDN or firewall bot reports, and by testing that your site returns a normal response to those user agents. If no AI crawler ever appears, or they only get blocked responses, your content cannot be retrieved for AI answers. This guide shows how to check each method on a WordPress site, what the results mean and how to fix common problems.
Why this check matters
Your robots.txt can say “allowed” while a firewall quietly blocks the same crawler. AI search tools can only cite pages they can fetch, so access problems are a hidden cause of missing AI visibility. A five-minute check can save months of wondering why nothing changes.
The user agents to look for
| User agent contains | Operator | What it means |
|---|---|---|
| OAI-SearchBot | OpenAI | Indexing for ChatGPT search |
| ChatGPT-User | OpenAI | A user asked ChatGPT to open your page |
| GPTBot | OpenAI | Content collection that may be used for training |
| PerplexityBot | Perplexity | Indexing for Perplexity answers |
| Perplexity-User | Perplexity | A user question triggered a fetch |
| ClaudeBot, Claude-SearchBot, Claude-User | Anthropic | Training, search and user-initiated fetches |
| Googlebot | Feeds Google Search, including AI Overviews | |
| bingbot | Microsoft | Feeds Bing and Copilot |
See our guide to allowing or blocking AI crawlers for what each one does.
Method 1: Check your server access logs
Access logs are the most reliable evidence, because they record every request your server received.
- In your hosting panel (cPanel, Plesk or a managed host dashboard) find Raw Access Logs or Logs and download the latest file.
- Search the file for the user agent names above. On a computer with a command line you can run
grep -i "GPTBot|OAI-SearchBot|PerplexityBot|ClaudeBot" access.log. - Look at the status code on each line. A
200means the page was served. A403,429or503means it was blocked or rate limited. - Count visits per bot per week to see whether they come regularly.
If you use a managed host that hides raw logs, ask support whether bot traffic can be exported.
Method 2: Use your CDN or firewall dashboard
Services such as Cloudflare, Sucuri or Wordfence show bot traffic and blocked requests.
- Open the firewall or security events report and filter by user agent.
- Check the AI bot settings. Some providers have a switch to block or challenge AI crawlers, and it may be on by default.
- Look for managed rules that challenge unknown bots, since they can block legitimate crawlers.
Method 3: Test with a simulated request
You can test whether your server responds normally to a crawler’s user agent string. From a command line, request your homepage with the user agent set to a crawler name, for example:
curl -I -A "OAI-SearchBot" https://example.com/You want a 200 response (or a redirect to a page that returns 200). A 403 or a challenge page suggests a firewall rule is blocking that user agent. Repeat for the other names in the table.
A caution: this only tests rules based on the user agent text. Real crawlers also come from known IP ranges, and some firewalls block by IP or behavior, so a passing test is a good sign and not a complete guarantee. Because user agents can be faked, serious verification uses the IP lists that some operators publish. Check each company’s documentation.
Method 4: Use a WordPress security plugin’s live traffic
Plugins like Wordfence can show recent visits with their user agents. This is easier than reading logs, but live traffic views often cover only a short window, so check regularly.
Method 5: Check your analytics for referrals
Crawler visits do not show up in browser-based analytics, but real people who click a citation in an AI answer do. Check traffic sources for referrals from chat and AI platforms. Names change over time, so look through your referral list now and then. This shows impact, not access. For full tracking, see how to track brand visibility in AI search.
How to read the results
| What you see | Likely meaning | What to do |
|---|---|---|
| Regular 200 responses to AI bots | Access is working | Focus on content and trust signals |
| No AI bot visits at all | Not yet discovered, blocked at the network edge, or logs are incomplete | Check robots.txt, CDN settings and sitemap; build links and mentions |
| 403 or 429 responses | A firewall, rate limit or security rule is blocking them | Review security settings and allow the crawlers you want |
| Only GPTBot visits, no search bot | Training crawler arrived but search crawler is blocked or not yet visited | Check OAI-SearchBot rules specifically |
| Visits only to a few pages | Poor internal linking or a thin sitemap | Improve internal links and submit a clean sitemap |
A lack of visits does not always mean a problem. Smaller or newer sites may simply be crawled less often.
Fixing the most common problems
- CDN bot toggle. Turn off the setting that blocks AI crawlers if you want AI visibility.
- Security plugin rules. Whitelist the user agents you want and review country or rate-limit rules.
- Robots.txt. Remove accidental disallow rules and add your sitemap. Our technical SEO checklist covers this.
- Slow server responses. Time-outs reduce how much gets crawled. Improve speed using our Core Web Vitals guide.
- JavaScript-only content. Make sure important text exists in the HTML response.
Frequently asked questions
How do I know if ChatGPT is crawling my site?
Look for OAI-SearchBot, ChatGPT-User or GPTBot in your access logs or firewall events, and check that they receive 200 responses.
Does Google Analytics show AI bot visits?
No. Bots usually do not run the analytics script. Use server logs or your CDN reports for crawler visits.
Why does no AI bot visit my site?
Common reasons are blocks at the firewall or CDN, a new or low-authority site, weak internal linking, or logs that do not cover the period.
Can I tell real crawlers from fake ones?
Yes, with care. User agents can be copied, so verify by IP address ranges when the operator publishes them.
Should I allow all AI bots?
That is a business decision. Allowing search and user crawlers supports AI visibility. Training crawlers are optional to allow.
Also read: AI visibility audit.
Once crawler access is sorted, an llms.txt setup is a small extra step.
Also read: how to get cited in Google Gemini.
Want a professional access check?
We review robots.txt, firewall, CDN and indexing together and fix what blocks your visibility. Contact us with your website address, or read about our WordPress SEO services.
