AI Crawler Access in 2026: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended & PerplexityBot

There are two technical details worth making central to the article:

OpenAI separates search from training. OAI-SearchBot is used for ChatGPT Search discovery, while GPTBot is associated with content that may be used to improve/train foundation models. A site can therefore allow search discovery while blocking potential training. OpenAI Developers

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Anthropic now has three separate controls, so writing only about ClaudeBot would be incomplete:

ClaudeBot         → model development/training
Claude-SearchBot  → search indexing
Claude-User       → user-triggered retrieval

Anthropic states that all three respect its documented controls, although they serve different purposes. Claude Help Center

There is also an important Google nuance: Google-Extended is not a separate HTTP crawler user agent. It is a robots.txt control token. Crawling itself is performed using existing Google crawler user agents. Blocking Google-Extended does not remove a site from normal Google Search. Google for Developers

For Perplexity, PerplexityBot builds/searches its index and is not used to train foundation models, while Perplexity-User handles user-triggered retrieval. Perplexity says Perplexity-User generally ignores robots.txt because the request is user initiated. Perplexity

Article structure

H1 — AI Crawler Access in 2026: A Technical Guide to GPTBot, ClaudeBot, Google-Extended & PerplexityBot

1. AI Crawlers Are Not All the Same

• training crawlers
• search/index crawlers
• user-triggered fetchers
• why treating them all as User-agent: * can be a mistake

2. OpenAI Crawlers

• OAI-SearchBot
• GPTBot
• ChatGPT-User
• published IP ranges
• search vs training
• recommended robots configuration

OpenAI publishes IP ranges for its automated crawlers and specifically recommends checking CDN/firewall access in addition to robots.txt. OpenAI Developers

3. Anthropic Crawlers

• ClaudeBot
• Claude-SearchBot
• Claude-User
• Crawl-delay
• robots configuration

4. Google-Extended

• why it isn’t Googlebot
• why it isn’t a normal crawler UA
• Gemini/model controls
• relationship with Google Search
• what blocking it actually changes

5. Perplexity

• PerplexityBot
• Perplexity-User
• official IP JSON
• WAF configuration
• search indexing vs live retrieval

6. Build a Correct robots.txt for AI Crawlers

Example scenario:

Allow AI search visibility but block model-training crawlers.

# OpenAI Search
User-agent: OAI-SearchBot
Allow: /

# OpenAI model training
User-agent: GPTBot
Disallow: /

# Claude Search
User-agent: Claude-SearchBot
Allow: /

# Claude user-triggered retrieval
User-agent: Claude-User
Allow: /

# Claude model development
User-agent: ClaudeBot
Disallow: /

# Google Gemini controls
User-agent: Google-Extended
Disallow: /

# Perplexity Search
User-agent: PerplexityBot
Allow: /

We should clearly label this as an example policy, not a universal recommendation.

7. robots.txt Is Only Layer One

This is where the article becomes much more technical:

AI crawler
    ↓
DNS
    ↓
CDN
    ↓
WAF / Bot Protection
    ↓
Rate Limiter
    ↓
Web Server
    ↓
Application
    ↓
Content

A crawler can be:

robots.txt → ALLOWED
Cloudflare → BLOCKED
Result     → crawler still cannot access page

OpenAI specifically warns about WAFs, bot protection, JavaScript challenges, CAPTCHAs and 403/429 responses causing crawler failures. OpenAI Help Center Perplexity similarly recommends validating both user-agent and official IP ranges when configuring WAF rules. Perplexity

8. Detect AI Crawlers in Server Logs

Nginx:

grep -Ei \
'OAI-SearchBot|GPTBot|ClaudeBot|Claude-SearchBot|PerplexityBot' \
/var/log/nginx/access.log

Apache:

grep -Ei \
'OAI-SearchBot|GPTBot|ClaudeBot|Claude-SearchBot|PerplexityBot' \
/var/log/apache2/access.log

Then analyse:

crawler
URL
status
timestamp
IP
response size
response time

Especially investigate:

200 → accessible
301/302 → redirect
403 → WAF/security block
404 → crawler reached nonexistent URL
429 → rate limited
5xx → server-side failure

9. Never Trust User-Agent Alone

A fake bot can send:

User-Agent: GPTBot

So the technical section should show why production allowlisting should use the provider’s published infrastructure/IP information where available rather than blindly trusting a header. OpenAI and Perplexity currently publish crawler IP sources. OpenAI Developers

10. Cloudflare/WAF Problems

• Bot Fight Mode
• managed challenges
• JavaScript challenges
• rate limiting
• IP reputation
• geo restrictions
• custom firewall rules
• 403
• 429

11. Test Whether an AI Crawler Can Actually Reach Your Site

Include:

curl -I https://example.com/robots.txt

and:

curl -A "OAI-SearchBot" https://example.com/

with an important warning that changing the User-Agent with curl tests your own server rules; it does not prove you’re a genuine OpenAI crawler.

12. Recommended Architecture

robots.txt
     │
     ├── Training bots → allow/block by policy
     │
     ├── Search bots → allow
     │
     └── User fetchers → decide separately
                         ↓
                    CDN / WAF
                         ↓
                   IP validation
                         ↓
                     Web server
                         ↓
                  Access logging

13. Technical Audit Checklist

Finish with perhaps 15 practical checks, including:

• robots.txt returns 200
• correct bot groups
• OAI-SearchBot accessible
• training policy intentionally configured
• Claude search/training rules separated
• Google-Extended understood correctly
• CDN doesn’t challenge allowed crawlers
• WAF doesn’t return 403
• rate limiter doesn’t return excessive 429
• crawler IP verification
• server logs retained
• redirects tested
• HTML returned without login
• sitemap reachable
• important assets/content not accidentally blocked