GEO | 2026-07-29 | 11 min read
AI Crawlers and Bots: What to Allow, Block, and Track
A practical guide to robots.txt, AI crawler access, security screens, logs, and what website owners should track for AI visibility.
Direct answer: Allow crawlers that help your business appear in search and AI answer surfaces, block only the crawlers or use cases you intentionally do not want, and track access through robots.txt, server logs, Search Console, and WAF rules. Do not accidentally block public pages with security screens or blanket bot rules.
Written by: Esmail Hanif, AI Visibility Strategist & Founder, Martecks
Short answer
Do not treat every AI crawler the same. Some bots are for search visibility, some are for training, and some are user-triggered fetchers.
The safe approach is to decide what you want: visibility in AI search, protection from training use, reduced server load, or stricter security. Then configure robots.txt, WAF rules, and logs around that decision.
The crawler decision map
Start with intent, not fear. Different bots serve different jobs.
| Goal | What to allow or check | Why |
|---|---|---|
| Show up in Google AI features | Allow Googlebot and keep pages eligible for Search snippets. | Google says AI Overviews and AI Mode use normal Search controls and eligibility. |
| Show up in ChatGPT search | Allow OAI-SearchBot and requests from published OpenAI IP ranges. | OpenAI says OAI-SearchBot is used for ChatGPT search features. |
| Avoid OpenAI training use | Disallow GPTBot if that is your policy choice. | OpenAI says GPTBot is for crawling content that may be used in foundation-model training. |
| Show up in Perplexity search | Allow PerplexityBot and verify access through logs or WAF rules. | Perplexity says PerplexityBot is for surfacing and linking websites in search results. |
| Support user-triggered fetches | Understand that user agents like ChatGPT-User or Perplexity-User may behave differently from automatic crawlers. | These visits can be initiated by a user action and may not be controlled the same way as crawling. |
What Google says
Google says there are no extra technical requirements for AI Overviews or AI Mode beyond being eligible for Search and following normal technical and content best practices.
For controlling content in Google AI features, Google points to Googlebot, Search preview controls like nosnippet and max-snippet, and noindex when you want content removed from Search. Google also says you do not need new AI text files or special schema just to appear.
Sources: Google Search Central: AI features and your website, Google Search Central: robots.txt interpretation
What OpenAI says
OpenAI documents separate user agents for different use cases. OAI-SearchBot is for search in ChatGPT features. GPTBot is for crawling content that may be used in training generative AI foundation models. ChatGPT-User is tied to certain user actions and is not used for automatic web crawling.
That means a business can allow search visibility while choosing a different policy for training use. The important part is not copying a robots.txt snippet blindly. The important part is knowing which use case each bot represents.
| OpenAI user agent | Use case | Plain-English implication |
|---|---|---|
| OAI-SearchBot | Search visibility in ChatGPT search features. | Usually allow if you want ChatGPT search visibility. |
| GPTBot | Crawling content that may be used for training foundation models. | Allow or block based on your training-use policy. |
| ChatGPT-User | Certain user-triggered actions from ChatGPT or Custom GPTs. | Do not treat it as the same thing as automatic crawling. |
Sources: OpenAI: Overview of OpenAI Crawlers
What Perplexity says
Perplexity documents PerplexityBot for surfacing and linking websites in Perplexity search results, and Perplexity-User for user actions. It also recommends allowing legitimate bot traffic through WAF rules by combining user-agent checks with official IP ranges.
This matters because a site can have a correct robots.txt file but still block useful AI crawler traffic at the firewall, CDN, or security layer.
Sources: Perplexity: Perplexity Crawlers
The security-screen problem
Security screens can protect a site, but they can also block search engines, AI crawlers, and real buyers if applied too broadly.
Public pages that need to rank, get cited, or convert should not challenge normal GET and HEAD requests by default. If the site needs WAF rules, use specific allow rules, verified IP ranges, bot management settings, and monitoring rather than a blanket challenge on every public page.
What to track
Crawler access is not a set-and-forget setting. Track it like technical visibility.
| Signal | Where to check | What it tells you |
|---|---|---|
| Indexing and clicks | Google Search Console | Whether Google can discover and show your pages. |
| Bot requests | Server logs, CDN logs, Cloudflare or Vercel logs | Which bots are visiting and whether they receive 200, 301, 403, 404, or challenge responses. |
| Robots rules | robots.txt and crawler documentation | Whether you are intentionally allowing or blocking each use case. |
| WAF behavior | Firewall events and bot management logs | Whether security rules are blocking legitimate public access. |
| AI visibility | Prompt tests and cited-source tracking | Whether access changes correlate with mentions or citations. |
Run a technical trust check
The free visibility checker is a starting point. For crawler access, the deeper work is checking robots.txt, logs, firewall events, sitemap, important pages, and AI prompt visibility together.
Reference links
These sources support the crawler access, robots.txt, AI search, and WAF guidance in this guide.
Sources: Google Search Central: AI features and your website, OpenAI: Overview of OpenAI Crawlers, Perplexity: Perplexity Crawlers, Google Search Central: robots.txt interpretation
Final answer
AI crawler strategy is not “allow everything” or “block everything.” It is choosing which use cases support your business and configuring access intentionally.
Allow the bots and public pages that support discovery, track the logs, avoid accidental security challenges, and block only the use cases you have deliberately decided not to support.