GEO | 2026-07-29 | 10 min read
AI Crawlers and Bots: What to Allow, Block, and Track
Choose which AI crawlers to allow, diagnose Cloudflare or firewall blocks, and distinguish readable pages from actual AI citations.
Direct answer: Decide separately whether you want AI search, training, and user-requested access. Check both robots.txt and the response a verified crawler receives. Discovery, successful access, and a citation are different events: none proves the next. Fix a demonstrated access problem without turning off site-wide security.
Written by: Esmail Hanif, AI Visibility Strategist & Founder, Martecks
Short answer
Do not treat every AI crawler the same. Some bots are for search visibility, some are for training, and some are user-triggered fetchers.
The safe approach is to decide what you want: visibility in AI search, protection from training use, reduced server load, or stricter security. Then configure robots.txt, WAF rules, and logs around that decision.
The crawler decision map
Start with intent, not fear. Different bots serve different jobs.
| Goal | What to allow or check | Why |
|---|---|---|
| Show up in Google AI features | Allow Googlebot and keep pages eligible for Search snippets. | Google says AI Overviews and AI Mode use normal Search controls and eligibility. |
| Show up in ChatGPT search | Allow OAI-SearchBot and requests from published OpenAI IP ranges. | OpenAI says OAI-SearchBot is used for ChatGPT search features. |
| Avoid OpenAI training use | Disallow GPTBot if that is your policy choice. | OpenAI says GPTBot is for crawling content that may be used in foundation-model training. |
| Show up in Perplexity search | Allow PerplexityBot and verify access through logs or WAF rules. | Perplexity says PerplexityBot is for surfacing and linking websites in search results. |
| Support user-triggered fetches | Understand that user agents like ChatGPT-User or Perplexity-User may behave differently from automatic crawlers. | These visits can be initiated by a user action and may not be controlled the same way as crawling. |
What Google says
Google says there are no extra technical requirements for AI Overviews or AI Mode beyond being eligible for Search and following normal technical and content best practices.
For controlling content in Google AI features, Google points to Googlebot, Search preview controls like nosnippet and max-snippet, and noindex when you want content removed from Search. Google also says you do not need new AI text files or special schema just to appear.
Sources: Google Search Central: AI features and your website, Google Search Central: robots.txt interpretation
What OpenAI says
OpenAI documents separate user agents for different use cases. OAI-SearchBot is for search in ChatGPT features. GPTBot is for crawling content that may be used in training generative AI foundation models. ChatGPT-User is tied to certain user actions and is not used for automatic web crawling.
That means a business can allow search visibility while choosing a different policy for training use. The important part is not copying a robots.txt snippet blindly. The important part is knowing which use case each bot represents.
| OpenAI user agent | Use case | Plain-English implication |
|---|---|---|
| OAI-SearchBot | Search visibility in ChatGPT search features. | Usually allow if you want ChatGPT search visibility. |
| GPTBot | Crawling content that may be used for training foundation models. | Allow or block based on your training-use policy. |
| ChatGPT-User | Certain user-triggered actions from ChatGPT or Custom GPTs. | Do not treat it as the same thing as automatic crawling. |
Sources: OpenAI: Overview of OpenAI Crawlers
Does Cloudflare help ChatGPT discover your pages?
Cloudflare and OpenAI reported early discovery-pilot results on October 15, 2026. They say network signals helped OpenAI find and index previously unknown pages. OpenAI independently evaluates the pages; being found does not guarantee selection as a source. This guidance was reviewed on October 26, 2026.
The pilot covered a participating cohort of about 1% of Cloudflare's network. Hosting on Cloudflare does not, by itself, establish that your site participated. The announcement does not report citation or referral gains for your business.
Cloudflare also says a discovery signal does not grant access. Keep your crawler permissions intentional. For your own site, inspect verified crawler requests, responses, and relevant firewall events before deciding whether access is working.
Separate discovery, access, and citation
Each stage answers a different question. A successful request proves much less than an actual citation, and an ordinary page request cannot establish participation in a private pilot.
Start with the AI crawler readiness checker for public page basics. To confirm actual crawler behaviour, you need the corresponding verified log evidence. Use the Bing AI Performance guide for a separate view of reported citations within Microsoft's supported experiences.
| Stage | Evidence to inspect | Limit |
|---|---|---|
| Discovery | An available indexing or discovery record | Cloudflare hosting alone proves no pilot participation |
| Access | Verified crawler request and actual response content | A 200 status can still return a challenge or empty page |
| Readable evidence | The facts available in the retrieved page | Readable text does not guarantee retrieval for a question |
| Citation | A saved answer or supported citation report | It does not establish a visit or recommendation |
| Commercial result | An identifiable visit or qualified enquiry | Attribution can be incomplete |
What Perplexity says
Perplexity documents PerplexityBot for surfacing and linking websites in Perplexity search results, and Perplexity-User for user actions. It also recommends allowing legitimate bot traffic through WAF rules by combining user-agent checks with official IP ranges.
This matters because a site can have a correct robots.txt file but still block useful AI crawler traffic at the firewall, CDN, or security layer.
Sources: Perplexity: Perplexity Crawlers
The security-screen problem
A firewall or CDN can serve a challenge instead of the article, even when robots.txt permits the crawler. A browser visit from your own computer does not establish what a different requester receives.
Keep protections for accounts, forms, and private data. If an intended search crawler is blocked on a public page, identify the rule responsible before changing it. Limit any exception to the verified crawler and necessary public paths. Do not bypass protections simply because a request claims a familiar bot name.
Cloudflare's crawler management documentation says unsuccessful requests can come from other rules or response errors, not just AI Crawl Control. Its free-plan identification uses user-agent strings; stronger detection is plan-dependent. Treat those labels as evidence with limits, not automatic proof of identity.
What to track
Crawler access is not a set-and-forget setting. Track it like technical visibility.
| Signal | Where to check | What it tells you |
|---|---|---|
| Indexing and clicks | Google Search Console | Whether Google can discover and show your pages. |
| Bot requests | Server logs, CDN logs, Cloudflare or Vercel logs | Which bots are visiting and whether they receive 200, 301, 403, 404, or challenge responses. |
| Robots rules | robots.txt and crawler documentation | Whether you are intentionally allowing or blocking each use case. |
| WAF behavior | Firewall events and bot management logs | Whether security rules are blocking legitimate public access. |
| AI visibility | Prompt tests and cited-source tracking | Whether access changes correlate with mentions or citations. |
A public page works, but the crawler gets a challenge
Consider a hypothetical repair business. Its emergency-service page opens in the owner's browser, and robots.txt allows its chosen search crawler. A request labelled with that crawler's name appears in the logs, but the saved response contains a security check rather than the service description.
First verify the requester using the operator's current published identity information. Then match the request time, URL, and security event to the rule that fired. If the business wants that verified crawler to access this public page, have the administrator review a narrowly scoped change. Test again and retain the before-and-after evidence.
If the repeat request gets the actual service content, record an access fix. Do not record a citation gain. Check saved answers or the relevant reporting product separately; a later citation still would not prove that this one fix caused it.
What should you save from the logs?
Give your developer a small evidence record for one important URL. Keep raw IP addresses and security logs private; a public case study can describe the finding without exposing them.
| Record | Why it matters |
|---|---|
| Exact URL and UTC time | Connect the page request with the matching security event |
| Claimed crawler and identity check | Separate a name in a header from verified operator traffic |
| Robots rule on the requested host | Check the policy that actually applies to this address |
| Redirect destination, status, and response content | Distinguish the article from an error, login, or challenge |
| Rule identifier and decision | Explain which control allowed or blocked the request |
| Change made and repeat observation | Show whether the specific access failure was repaired |
Can I block training without blocking search?
For OpenAI, the documented controls distinguish GPTBot from OAI-SearchBot. Choose the policy for each purpose. Do not assume a general “block AI” switch makes the same distinction for every operator or hosting provider.
Read the current documentation for both the crawler and your security service before applying a rule. A robots policy is not authentication or a way to protect confidential pages. Keep those pages behind real access controls.
Does a readiness checker prove a bot visited?
No. A public checker can inspect what its own request receives and flag technical issues. It cannot authenticate a past crawler visit or establish inclusion in an AI answer. A test request with a copied bot user-agent string is still a test request.
Use the AI crawler readiness checker for an initial page review. If access is already working, move to the AI search strategy guide: improve the evidence that answers the customer's question instead of repeatedly changing bot settings.
Reference links
These sources support the crawler access, robots.txt, AI search, and WAF guidance in this guide.
Sources: Google Search Central: AI features and your website, OpenAI: Overview of OpenAI Crawlers, Perplexity: Perplexity Crawlers, Google Search Central: robots.txt interpretation