Key takeaways
- Training and search are separate bots now. GPTBot is not OAI-SearchBot, and ClaudeBot is not Claude-SearchBot. Each has its own robots.txt token.
- The defensible default for most sites: block training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended), allow search and retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot).
- User-triggered fetchers (ChatGPT-User, Perplexity-User) may not follow robots.txt fully, and some bots (Bytespider) have a documented non-compliance history. For those, you need WAF or IP rules.
- A firewall or CDN rule is evaluated before robots.txt. A managed "block AI bots" rule can silently cancel your Allow lines.
- Meta is the newest wildcard. Meta-WebIndexer went from about 2% to 37.8% of tracked AI crawler requests in under a month in 2026, so decide on it deliberately.
Why one blanket rule is the wrong move
For a couple of years, "block the AI bots" was a reasonable one-line policy. It isn't anymore. The big vendors split their crawlers by purpose, so a rule that blocks one does not block the others. That cuts both ways. You can opt out of model training and still be eligible for citations. You can also accidentally delete yourself from ChatGPT search while thinking you only blocked training.
The cost of getting this wrong is real. One industry analysis claims 71% of news publishers that block a search bot do so by accident, and loses citations as a result. I would treat that figure with some caution since it comes from a single source, but it matches what I see in audits: most broken robots.txt files are broken because nobody made a decision, not because the decision was wrong.
This guide sorts every major AI crawler into one of three jobs, gives you a default for each, and shows you how to verify the result.
The three jobs AI crawlers do
| Job | What it does | Blocking it costs you |
|---|---|---|
| Training | Bulk-crawls pages to feed future models | Nothing in AI search. Future models know less about you. |
| Search indexing | Builds the index AI answer engines cite | Citations and referral traffic from that engine |
| User fetch | Reads a single URL because a person asked | A broken experience for someone actively trying to read your page |
Everything else in this guide follows from that table.
The decision matrix: bot by bot
| Crawler | Operator | Job | Suggested default |
|---|---|---|---|
| GPTBot | OpenAI | Training | Block (if you want to opt out of training) |
| OAI-SearchBot | OpenAI | ChatGPT search index | Allow |
| ChatGPT-User | OpenAI | User fetch | Allow |
| ClaudeBot | Anthropic | Training | Block (if opting out) |
| Claude-SearchBot | Anthropic | Claude web search | Allow |
| Claude-User | Anthropic | User fetch | Allow |
| PerplexityBot | Perplexity | Search index | Allow |
| Perplexity-User | Perplexity | User fetch | Allow |
| Google-Extended | Gemini training token | Block if opting out. No effect on Search ranking. | |
| Applebot-Extended | Apple | Apple Intelligence training | Block if opting out. No effect on Apple Search. |
| CCBot | Common Crawl | Open corpus | Block |
| Bytespider | ByteDance | Training | Block at WAF/IP level |
| Amazonbot / Amzn-SearchBot | Amazon | Training / search | Block Amazonbot, allow Amzn-SearchBot |
| Meta-WebIndexer | Meta | Meta AI search index | Your call, see below |
OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
OpenAI's own documentation says disallowing GPTBot signals that your content should not be used to train generative AI foundation models. It has no effect on whether ChatGPT search can cite you. That job belongs to OAI-SearchBot. Block it and you disappear from ChatGPT search answers, even though the same company runs both bots.
ChatGPT-User fires when someone asks ChatGPT to open a page. OpenAI says it may not be governed by robots.txt the way automated crawlers are, so don't count on a Disallow there. Blocking it is almost never worth it anyway, because you're breaking a request a real person made.
Why care about ChatGPT specifically? Because it cites a lot. Promptwatch Data found almost half of ChatGPT responses cite 10 or more unique domains (Sep 28 to Oct 4, 2026). There are plenty of slots to win, but only if the search bot can read you.
Anthropic: ClaudeBot, Claude-SearchBot, Claude-User
Anthropic runs three bots and says all three honor robots.txt and won't bypass CAPTCHAs. The part people miss: each token needs its own directive. Blocking ClaudeBot does not block Claude-SearchBot or Claude-User, and allowing ClaudeBot does nothing for the other two.
The old anthropic-ai and Claude-Web agents are deprecated. Leaving rules for them in place is harmless.
There is also a cost argument for blocking ClaudeBot. Cloudflare data cited by Digital Applied put Anthropic at roughly 70,900 pages crawled per referred visitor at its June 2025 peak, against about 5:1 for Googlebot. I haven't verified that figure independently and it's a snapshot, but the direction is clear: training crawls return little traffic.
Perplexity: PerplexityBot and Perplexity-User
Perplexity documents two agents. PerplexityBot builds the search index. Perplexity-User fetches pages live. If you want to appear in Perplexity answers, allow both.
Here's the caveat. Perplexity has said Perplexity-User is "an agent, not a bot," so it argues robots.txt doesn't bind it. And on August 4, 2025, Cloudflare published a report accusing Perplexity of using undeclared crawlers that rotate user-agents, IPs and ASNs to evade no-crawl rules. For brands who want visibility this is mostly academic. If you ever need to keep Perplexity out entirely, robots.txt alone won't do it.
Meta: the newest wildcard
Meta's crawlers deserve a separate decision because the numbers moved so fast. Promptwatch's Meta-WebIndexer report shows Meta-WebIndexer at about 2.2% of tracked AI crawler requests in mid-July 2026, rising to a peak of 37.8% on August 9. That is roughly a 17x jump in under a month, and it made Meta-WebIndexer the heaviest AI crawler in Promptwatch's logs.
The report says Meta-WebIndexer exists to improve Meta AI search results and cite sources. The timing matches unconfirmed reports that Meta is building its own web search index. Treat that part as rumor.
The Meta user-agents worth knowing:
- Meta-WebIndexer (search index for Meta AI)
- Meta-ExternalAgent
- Meta-ExternalFetcher (user-triggered fetching)
- FacebookBot
My take: if Meta AI matters to your audience, allow Meta-WebIndexer and budget for the crawl load, especially if your hosting bills per request. If it doesn't, block it. Just make it a choice. I could not verify from Meta's own documentation which of its fetchers honor robots.txt, and several vendors (Meta among them, per one guide) say their user-triggered fetchers may not. Check Meta's docs before relying on a Disallow.
Google, Apple, Common Crawl, ByteDance, Amazon
Google-Extended is a robots.txt control token, not a separate HTTP user-agent. Blocking it opts you out of Gemini training and, per Google, doesn't touch Search inclusion or ranking. Applebot-Extended works the same way for Apple Intelligence and leaves Applebot (Siri and Spotlight) alone.
CCBot feeds Common Crawl, an archive that has trained a large share of the major LLMs. robots.txt is the primary opt-out. Remember that a Disallow only stops future crawling. It can't pull your pages out of snapshots that already exist.
Bytespider is the problem child. It's independently reported as inconsistently compliant and ByteDance publishes no IP list or robots.txt policy. Block it at the WAF.
Amazon splits too. Amazonbot may train models; Amzn-SearchBot supports Alexa and Rufus search and says it doesn't crawl for generative AI training.
Which strategy fits you
There are two legitimate positions.
- Block training, keep search. Right for most business sites, SaaS companies, ecommerce stores and blogs. You opt out of model training and stay citable. This is the default I'd start with.
- Allow everything. Right if your revenue depends on being cited and you don't care about training use. Keep a server-level rule ready for misbehaving bots.
- Block everything. Right for publishers and brands with proprietary content they won't license. You pay for it with lost citations in ChatGPT, Claude and Perplexity.
A simple way to decide: ask whether being quoted by an AI assistant makes you money. If yes, never block a search bot. Then ask separately whether you mind your content training future models. That second question is about principle and IP, not traffic.
Copy-ready robots.txt: block training, allow search
# Training crawlers: blocked
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Bytespider
Disallow: /
# Search and retrieval crawlers: allowed
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Amzn-SearchBot
Allow: /
# Meta: decide deliberately (allowing shown here)
User-agent: Meta-WebIndexer
Allow: /
If you want maximum visibility instead, switch the training bots to Allow: / or simply delete their blocks. If you want a full lockout, flip every search bot to Disallow: / and accept the citation loss.
A few gotchas:
- Each user-agent needs its own group. Don't assume one rule covers a vendor's whole family.
- A tool like the Robots.txt Generator for AI on promptwatch.com can draft the file for you, but read the output before shipping.
- Test after you deploy. A typo in a robots.txt group is silent.
The enforcement trap: WAF rules beat robots.txt
This is the mistake I'd check first. Firewalls and CDNs are evaluated before robots.txt is even read. If a Cloudflare rule blocks "all AI crawlers," your Allow for OAI-SearchBot is irrelevant. You are still excluded from ChatGPT search.
Cloudflare's one-click "Block AI bots" rule has been available on all plans since July 2024. Per Digital Applied, it and Managed Robots.txt are now deprecated in favor of granular Search, Training and Agent controls plus Bot Preference Sync. If your zone still runs the old rule, replace it first and confirm it isn't catching the search bots you want to keep.
A rough ordering of control strength:
- WAF or CDN rules, enforced regardless of what the bot wants
- IP allow and block lists, using published ranges (OpenAI publishes gptbot.json and searchbot.json; Anthropic publishes bots.json)
- robots.txt, voluntary and only as good as the bot's compliance
One subtlety: Anthropic notes that blocking its bots by IP may not guarantee an opt-out, because the bot can no longer read your robots.txt. Use robots.txt for policy and IP rules for security.
Also, a 403 or redirect is a poor way to say no to a polite crawler. It burns server budget and can be dodged by user-agent spoofing. Use robots.txt for bots that respect it and reserve hard blocks for those that don't.
How to check what is actually hitting you
robots.txt gives you no feedback loop. You have to look at logs. Three places to check:
- Your robots.txt file itself, served at the root with a 200
- Server responses, by requesting a page with a bot user-agent string
- Server or CDN logs, to see who really visits
A quick curl test, using a search bot's user-agent, shows whether your firewall lets it through:
curl -L -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" https://example.com/
A 200 is good. A 403 or challenge page means something upstream is blocking the bot, whatever your robots.txt says. Keep in mind that this only tests the user-agent string, not the IP verification a real WAF rule may apply.
User-agents are trivially spoofed, so for real analysis you want logs verified against published IP ranges. Promptwatch Data's AI crawler traffic report counts only requests whose source IP matches the operator's published ranges. In that data, OpenAI made up 79.8% of verified AI crawler requests in Aug 31 to Sep 6, 2026, down from 94.8% in June 8 to 14. Their advice is useful: if a provider is a large share of the chart but near zero in your own logs, you are probably blocked or unreachable for that provider.
Tools that read crawler logs include Promptwatch, whose Agent Analytics shows IP-verified AI crawler hits, errors and the path from crawl to citation.
Other options for seeing citation outcomes include Qwairy and Botify, depending on whether you want visibility tracking or enterprise technical SEO.
What the references actually look like
The three source guides I leaned on are worth a skim if you want the vendor-by-vendor detail. Digital Applied's decision matrix is the clearest on the WAF-versus-robots.txt order:
Qwairy's walkthrough covers the three-jobs framework and sample files for two strategies:
Quick comparison of compliance
| Crawler | Honors robots.txt? | Hard-block method |
|---|---|---|
| GPTBot, OAI-SearchBot | Yes | Published IP ranges |
| ChatGPT-User | Partly, per OpenAI | WAF |
| ClaudeBot, Claude-SearchBot, Claude-User | Yes, per Anthropic | Published IP ranges (with the caveat above) |
| PerplexityBot | Yes, with stealth-crawler dispute | WAF |
| Perplexity-User | Disputed | WAF |
| CCBot, Google-Extended | Yes | robots.txt is enough |
| Bytespider | Documented non-compliance | WAF or IP |
| Meta fetchers | Unverified, check Meta docs | WAF |
A short checklist before you ship
- Decide your training policy and your search policy separately, and write both down.
- Make sure every vendor's search and user-fetch tokens have their own lines.
- Audit your CDN and WAF for managed AI rules that contradict robots.txt.
- Add WAF rules for bots that ignore robots.txt (Bytespider, stealth crawlers).
- Test with curl, then confirm in logs that OAI-SearchBot, Claude-SearchBot and PerplexityBot are actually getting 200s.
- Revisit every quarter. Vendors add bots several times a year, and Meta's jump shows how quickly the mix can change.
Allowing the crawlers is step one. Getting cited is step two, and that depends on your content and whether those pages are retrievable. If you want to see how crawler access relates to citations across more tools, the GEO software directory at bestgeosoftware.com lists options.
The bottom line
Block the bots that only take, allow the bots that can send readers, and make sure your firewall agrees with your robots.txt. For most sites that means disallowing GPTBot, ClaudeBot, CCBot and Google-Extended while keeping OAI-SearchBot, Claude-SearchBot and PerplexityBot open. Meta is the one to decide on this quarter, with a log check on your own traffic before you do.