Key takeaways
- AI crawlers fall into three jobs: training, search indexing, and user-triggered fetching. Each deserves a different robots.txt decision.
- Meta-WebIndexer indexes content for Meta AI search and citations. Per Promptwatch Data, its share of tracked AI crawler requests rose from about 2.2% in mid-July 2026 to 37.8% on August 9.
- User-triggered fetchers (Meta-ExternalFetcher, ChatGPT-User and similar) may ignore robots.txt, so hard blocking has to happen at the CDN or server.
- Meta's own robots.txt sample only mentions meta-externalagent and meta-externalads. Do not assume your Meta rules cover the indexer.
- Robots.txt only expresses intent. Your server logs show whether crawlers actually arrive.
Why robots.txt suddenly matters again
For years robots.txt was a file most teams set once and forgot. Then AI crawlers showed up, and every provider shipped three or four user agents with slightly different jobs. Block the wrong one and you vanish from an AI answer engine. Allow the wrong one and your content trains a model you never agreed to feed.
The Meta numbers are what made me rewrite my own checklist. Promptwatch tracks IP-verified AI crawler requests, and its Meta-WebIndexer report shows the crawler going from roughly 1 in 50 AI crawler requests in mid-July 2026 to more than 1 in 3 by August 9. That made it the heaviest AI crawler in their logs at the time. One crawler, under a month.

A caveat on the cause. The report links the surge to claims that Meta is building its own search index so Meta AI does not depend on Google or Bing. That claim traces back to an August 6 post by @levelsio, who said Meta staff told him so, and who also reported load alerts on his VPS. Meta has not confirmed it. The crawl volume is measured. The motive is not.

The three jobs an AI crawler can do
Before you touch the file, sort every bot into one of three buckets.
| Type | What it does | Examples | What blocking it changes |
|---|---|---|---|
| Training crawler | Collects content that may train foundation models | GPTBot, ClaudeBot, meta-externalagent, Applebot-Extended | Whether your content feeds future models |
| Search crawler | Indexes pages so an AI search product can retrieve and cite them | OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer | Eligibility to appear in current AI answers |
| User-triggered fetcher | Loads a URL live because a person asked | ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher | Whether an assistant can read your page on request |
The mistake I see most is treating the whole list as one thing. "Block AI" sounds like a policy, but it is really three separate decisions.
Meta's crawlers, one by one
Meta documents four relevant user agents on its web crawlers page, last updated May 21, 2026.

Meta-WebIndexer
The user agent is meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler). The research firm 51Degrees classifies it as used purely for indexing Meta AI search results, and says Meta notes that allowing it supports Meta AI citing and linking to your content. If you want to be a source in Meta AI answers, this is the one to allow.
One gap in my research: I could not confirm from Meta's own documentation whether Meta-WebIndexer honors robots.txt. A third-party index page claims it does, and one aggregator list marks it as obeying the file, but check your logs after changing rules rather than trusting either.
Meta-ExternalAgent
Meta describes it as crawling the web for training foundation AI models or improving products by indexing content directly. This is the training-flavored bot. Blocking it is the closest thing to a Meta training opt-out.
Meta-ExternalFetcher
It fetches individual links at a user's request and supports agentic features, including helping AI navigate sites to complete tasks. Meta says it may bypass robots.txt. A Disallow here is a request, not a wall.
FacebookExternalHit
This builds link previews when someone shares your URL on Facebook, Instagram or Messenger. It can bypass robots.txt for security and integrity checks such as malware scans. Blocking it breaks your share cards, which is rarely what anyone wants.
| Meta crawler | Job | Honors robots.txt? | Typical stance |
|---|---|---|---|
| meta-webindexer | Meta AI search index and citations | Not confirmed by Meta docs | Allow if you want citations |
| meta-externalagent | Training and direct indexing | Yes, per Meta's sample | Your call on training |
| meta-externalfetcher | User-requested fetches | May bypass | Allow, enforce at CDN if needed |
| facebookexternalhit | Link previews | May bypass for security checks | Allow |

Note that 51Degrees warns a blanket disallow of every Meta user agent can mean missing discovery inside Meta's services. That matches what the Promptwatch report suggests: treat Meta as an emerging AI search surface, not only a social network.
The rest of the list
Meta is one of several operators. The set Promptwatch tracks in its crawler logs gives a decent working inventory:
- OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot
- Anthropic: ClaudeBot, Claude-User, Claude-SearchBot, claude-web
- Google: Google-Extended, Google-Agent
- Perplexity: PerplexityBot, Perplexity-User
- xAI: GrokBot, xAI-Grok, Grok-DeepSearch
- Mistral: MistralAI-User, MistralAI-Index
- Cohere: cohere-ai
- DeepSeek: DeepSeekBot
Two details matter in practice. Google-Extended is a control token for Gemini training and grounding, and it does not affect Googlebot or ordinary Search. Secondary guides note that it combines training and grounding, so Disallow rejects both. And several operators, including OpenAI, Perplexity, Meta and Amazon, say in their docs that user-triggered fetchers may not follow robots.txt.
For a longer reference, the AI Crawler URL Inspector and the AI bots directory cover 400+ agents with robots.txt guidance.
Decide your policy first
Pick the row that matches your business, then write the file.
| Your goal | Allow | Block |
|---|---|---|
| Maximum AI visibility | Everything documented | Nothing |
| Visible in AI answers, no training | Search crawlers and user fetchers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, meta-webindexer, meta-externalfetcher) | GPTBot, ClaudeBot, meta-externalagent, Applebot-Extended |
| Paywalled or licensed content | Little or nothing | Training and search bots, plus CDN-level enforcement |
| Local business or ecommerce | Search and fetcher bots, so products and locations can be cited | Training bots, if you prefer |
I lean toward the second row for most publishers and brands. Training opt-outs are a values and licensing decision. Search inclusion is a revenue decision. Mixing them up is how brands end up absent from answers they could have won.
It is worth knowing what the citation data says before you block anything. Promptwatch Data shows product pages are the most cited content type in ChatGPT at 22.4% of citations (Sep 28 to Oct 4, 2026), and that landing pages and docs stay cited longest. If those are the pages you most want cited, make sure search crawlers can read them.
A starter robots.txt
This version allows search and user fetchers and blocks training. Adapt it to your row in the table above.
# OpenAI
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /
# Anthropic
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /
# Perplexity
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
# Meta
User-agent: meta-webindexer
Allow: /
User-agent: meta-externalfetcher
Allow: /
User-agent: meta-externalagent
Disallow: /
# Google: keep Allow for Gemini grounding, Disallow rejects training and grounding
User-agent: Google-Extended
Allow: /
# Apple
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Disallow: /
Policies change, and I did not re-verify every operator's current documentation for this guide, so check each vendor's page before shipping. You can also generate a file with the Robots.txt Generator for AI and sanity check it against the Agentic Readiness Check.
robots.txt is not enforcement
Three things trip people up.
Fetchers can ignore you. If a Disallow must hold, use your CDN, WAF or web server. Verify by IP range, because user agent strings are trivially spoofed.
Spoofed bots exist. Promptwatch only counts requests that match each operator's published IP ranges. Your analytics probably does not, so raw user agent counts overstate real crawler traffic.
Load is real. The Promptwatch report notes Meta's volume was high enough to trigger load alerts on independent sites. If your hosting bills per request, budget for sustained crawl traffic rather than a one-week spike. A Crawl-delay line is not honored by everyone, so rate limiting at the edge is safer.
Check your logs against the market
A clean robots.txt tells you nothing about who shows up. Promptwatch's AI crawler traffic report (IP-verified weekly data, June 8 to September 6, 2026) put OpenAI at 79.8% of tracked requests in the week of August 31, down from 94.8% in the week of June 8. Its advice is simple: if a provider is big in the market mix but close to zero in your logs, you are probably blocked or unreachable for it. Check robots.txt, CDN rules and WAF settings for that provider.
Note that this report covers OpenAI, Anthropic, Google, Perplexity and Mistral. The Meta report's denominator also includes xAI, Cohere, DeepSeek and Meta. Do not compare the two percentages directly.
Here are tools that can help you see this in your own data:
| Need | Option |
|---|---|
| IP-verified crawler logs plus citation tracking | Promptwatch |
| Enterprise technical SEO and bot log analysis | Botify |
| Broad AI visibility monitoring | Profound, Peec AI, Otterly.AI |
| Crawl-based scraping for your own testing | Firecrawl, Apify |
Most monitoring tools tell you whether a brand was mentioned. Promptwatch's crawler logs go a step earlier: they show which pages bots read, what errors they hit, and whether a crawled page later gets cited, which separates an access problem from a content problem. It is also why I would pair a robots.txt change with a log check a week later.
If you are comparing options, the GEO software directory lists more of them.
A short audit routine
- List every AI user agent in your logs from the past 30 days.
- Sort each into training, search or fetcher.
- Decide your policy per bucket, not per vendor.
- Write the file, then add CDN rules for anything you truly need blocked.
- Re-check logs after a week. Did the allowed search bots come? Did the blocked ones stop?
- Repeat quarterly. New agents appear constantly, and Meta alone has changed its crawler page this year.
What I would do today
Allow meta-webindexer if Meta AI citations could matter to you, and keep meta-externalagent as your explicit training decision. Do not rely on Disallow for meta-externalfetcher or facebookexternalhit. Then watch your logs. If Meta really is building an index, content crawled during that build is what gets cited at launch, and a robots.txt line written in 2024 will not tell you whether you are in it.


