Key takeaways
- Meta-WebIndexer went from about 2.2% of AI crawler requests in mid-July 2026 to 37.8% on August 9, according to Promptwatch Data. It was the heaviest AI crawler in their logs that day.
- Meta documents five crawlers with different jobs. Each has its own user-agent token, so blocking one does not block the others.
- Meta says allowing Meta-WebIndexer helps it cite and link to your content in Meta AI answers. Meta-ExternalAgent is the one tied to AI training and indexing.
- Meta-ExternalFetcher and FacebookExternalHit may bypass robots.txt. If you need to stop them, you need server or CDN rules, and that breaks link previews and user-requested fetches.
- For most sites the sensible default is selective: decide on training separately from search citations, then watch your logs.
What actually happened
A 17x jump in under a month is hard to ignore. In mid-July 2026, requests from the meta-webindexer user agent were about 2% of all AI crawler traffic that Promptwatch tracks. The share sat there until July 17, spiked to around 23% between July 20 and 22, passed 26% on August 6, and peaked at 37.8% on August 9. Promptwatch put it as going from 1 in 50 AI crawler requests to 1 in 3.
A caveat on the number, because it gets repeated loosely. The numerator is only meta-webindexer requests. The denominator is all AI crawlers Promptwatch identifies by user agent. Googlebot, Bingbot, link-preview fetchers and humans are excluded. So "38%" is a share of AI bot traffic, not of your total traffic.
Why is Meta doing this? Reports say Meta is building its own web index so Meta AI does not have to lean on Google or Bing. The only thing on record is a post from @levelsio on August 6, 2026, saying Meta staff told him in DMs that Meta was "allegedly" building an index. He also said the scraping triggered load alerts on one of his servers. Treat that as unconfirmed. Meta's own documentation says only that the crawler improves Meta AI search results.
There is some anecdotal pain too. A r/webdev thread reports 7.9 million hits from a Meta AI crawler on one site in 30 days, and commenters suggested blocking at Cloudflare or in .htaccess. One site, one report, but it shows why you should look at your own logs rather than guess.
Meta's five crawlers, and what each one does
Meta's Web Crawlers documentation (updated May 21, 2026) lists five. Note that Promptwatch also sees a user agent called FacebookBot that is not on Meta's page, so check for it in your logs too.
| Crawler | What Meta says it does | Respects robots.txt? | Block it and you lose |
|---|---|---|---|
| Meta-WebIndexer | Crawls to improve Meta AI search quality and helps cite and link your content | Not listed as a bypass crawler | Potential citations and links in Meta AI answers |
| Meta-ExternalAgent | Training foundation AI models or improving products by indexing content directly | Yes | Participation in training and direct indexing |
| Meta-ExternalAds | Improves advertising and other business products | Yes | Ad-related crawling |
| Meta-ExternalFetcher | Fetches individual links at a user's request, supports agentic AI tasks | May bypass | Rule out with server rules only; breaks user-requested fetches |
| FacebookExternalHit | Builds link previews for Facebook, Instagram, Messenger shares | May bypass for security checks | Link previews (title, description, thumbnail) |
The user-agent strings you will see are meta-webindexer/1.1, meta-externalagent/1.1, meta-externalads/1.1, meta-externalfetcher/1.1 and facebookexternalhit/1.1.
Two things Meta's docs do not say. They do not state whether Meta-WebIndexer data feeds model training, only that it supports Meta AI search and citations. And they give no crawl-delay or rate-limit guidance, so do not assume Crawl-delay is honored.
The decision: training versus search
Most of the debate collapses two questions into one. Do you want your content used to train Meta's models? And do you want Meta AI to be able to cite and link to you? Those map to different crawlers, so you can answer them differently.
Single Grain's analysis of Meta-ExternalAgent lands on a selective default: block Meta-ExternalAgent if training is your worry, keep FacebookExternalHit and Meta-WebIndexer open. A blanket block breaks Facebook link previews and removes a path to Meta AI citations. I think that is right for most brands, with one honest caveat. Meta AI is a smaller referral source than ChatGPT or Google today, so the citation upside is real but modest. If your server is struggling, load is a legitimate reason to act regardless.
Pick the scenario closest to yours:
| Your situation | Suggested approach |
|---|---|
| You want AI visibility and are fine with training | Allow all Meta crawlers |
| You object to training but want citations | Disallow Meta-ExternalAgent, allow Meta-WebIndexer and FacebookExternalHit |
| Server load is the problem | Keep rules as above, then rate-limit at the CDN |
| You want Meta out entirely | Disallow each token, plus CDN rules for the ones that may bypass robots.txt |
| Publisher with paywalled content | Disallow by path, not by site |
robots.txt examples
The key pitfall: each crawler is its own token. Blocking Meta-ExternalAgent does not block Meta-WebIndexer.
Allow search citations, block training
User-agent: meta-externalagent
Disallow: /
User-agent: meta-webindexer
Allow: /
User-agent: facebookexternalhit
Allow: /
Block Meta's AI crawlers, keep link previews
User-agent: meta-externalagent
Disallow: /
User-agent: meta-webindexer
Disallow: /
User-agent: meta-externalads
Disallow: /
Allow indexing but keep private areas out
Meta's own sample uses this pattern for the crawlers it shows:
User-agent: meta-webindexer
Allow: /
Disallow: /private/
Disallow: /account/
Meta says changes can take up to 24 hours to apply because crawlers may cache robots.txt. Do not panic if requests continue the same day you edit the file.
What robots.txt will not stop
Meta-ExternalFetcher fetches pages when a person asks Meta AI about a link, and Meta says it may bypass robots.txt. FacebookExternalHit may do the same for security and integrity checks such as malware scans. If you must block these, you need firewall or CDN rules, and you should know the cost: users who paste your URL into Meta AI get nothing, and shared links lose their preview cards.
If you do keep FacebookExternalHit open, Meta lists requirements worth checking: gzip and deflate support, Open Graph tags within the first 1 MB, a response within a few seconds, and correct handling of the Range header. You can simulate it with Meta's own command:
curl -v --compressed -H "Range: bytes=0-524288" -H "Connection: close" -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" "$URL"
The Sharing Debugger forces a re-crawl once your content is available.
Verifying the traffic is really Meta
User agents are trivially spoofed. Verisoul reports that Meta publishes no IP file for these crawlers, only that they come from Meta's own network. Meta's FacebookExternalHit advice does mention allow-listing IP addresses as the more secure option, so the two sources do not fully agree. A common route is to check requests against the ranges announced by Meta's ASN (AS32934), but verify that approach yourself before building firewall rules on it.
If you use a CDN, a bot-management layer is easier than hand-maintaining lists. Anything you rate-limit should be tested first so you do not accidentally throttle link previews.
Check your logs before you change anything
The order I would work in:
- Pull a week of server or CDN logs and count requests for meta-webindexer, meta-externalagent, meta-externalfetcher, facebookexternalhit and FacebookBot.
- Compare against your total bot traffic and your hosting bill. If you pay per request, the sustained load is a real cost.
- Decide on training and citation separately, and write the robots.txt rules above.
- Add CDN rate limits only if load is actually a problem.
- Re-check in two weeks. Crawl patterns changed 17x in a month, and they can change again.
Tracking this by hand gets tedious. Promptwatch has IP-verified AI crawler logs that show which pages each AI bot reads, how often and where it errors, and whether the crawled page later gets cited. Verification against operators' published ranges filters out spoofers, though Promptwatch notes Meta does not publish a list, so treat Meta lines in any tool with some care.
For a quick read without setup, the AI Crawler URL Inspector and the Robots.txt Generator for AI are free. The bots directory lists 400+ AI crawlers with robots.txt guidance.
Other crawler data worth knowing
Meta is not the only crawler moving fast. Promptwatch's AI crawler traffic report tracks weekly verified requests from OpenAI, Anthropic, Google, Perplexity and Mistral for June 8 to September 6, 2026. If you are rewriting robots.txt for Meta, review those rules at the same time rather than in three separate edits.
Also keep in mind that crawl volume is not the same as visibility. A crawler hitting your site millions of times does not mean Meta AI will cite you. Compare crawls with citations: that gap tells you whether you have an access problem or a content problem. If you want to explore tools that track AI search presence, the GEO software directory at bestgeosoftware.com lists options.
What I would do
For a typical marketing or content site: block Meta-ExternalAgent if you do not want training use, keep Meta-WebIndexer and FacebookExternalHit open, and watch logs for a couple of weeks. If you are a publisher with licensed or paywalled material, block by path and consider the CDN. If your server is buckling, rate-limit first and argue about policy afterward.
The one thing I would not do is copy a blanket block from a forum post. It is two lines, it feels decisive, and it quietly removes you from a search surface that Meta is clearly investing in. Whether that surface becomes large is still unknown, which is the strongest argument for choosing rules you can easily reverse.
Meta's contact for crawler questions is [email protected].