Key takeaways
- Meta-WebIndexer is Meta's search-indexing crawler. It fetches public pages to build the index behind Meta AI's answers, and it is separate from Meta's training and link-fetching bots.
- In Promptwatch's crawler logs, its share of tracked AI crawler requests went from roughly 2.2% in mid-July 2026 to 37.8% on August 9, 2026, about a 17x jump in under a month.
- The surge lines up with an unconfirmed report that Meta is building its own web index so Meta AI doesn't have to lean on Google or Bing.
- Blocking it in robots.txt probably removes you from Meta AI citations. Allowing it costs server load, and some of the traffic claiming to be Meta is spoofed.
- Check your logs, verify IPs, and decide deliberately. Don't leave it to default.
What Meta-WebIndexer is
Meta-WebIndexer is a web crawler that identifies itself as meta-webindexer/1.1. Meta's crawler documentation says it navigates the web to improve Meta AI search result quality. The robots.txt product token is meta-webindexer.
The practical reading: this is the bot that decides which pages are available for Meta AI to cite when it answers a question with live web information. It is the Meta equivalent of OAI-SearchBot for ChatGPT or PerplexityBot for Perplexity. It's about retrieval and citation, not model training.
That distinction matters because Meta runs several crawlers, each with its own token. A site can permit one and refuse another.
| Crawler | What it is for | Robots.txt token |
|---|---|---|
| Meta-WebIndexer | Indexing for Meta AI search results | meta-webindexer |
| Meta-ExternalAgent | Meta's AI training and content ingestion crawler | meta-externalagent |
| Meta-ExternalFetcher | User-triggered fetching of specific links | meta-externalfetcher |
| FacebookBot | Facebook-related crawling | facebookbot |
FacebookExternalHit, the link-preview fetcher that runs when someone pastes your URL into a post, is yet another thing again. It's not part of the AI crawler picture here.
The numbers: 2% to 38% in under a month
Promptwatch Data published the clearest measurement of the surge in its report Meta Is Crawling the Web Like It's Building a Search Index (August 10, 2026). The headline: in mid-July 2026, Meta-WebIndexer was about 2.2% of all AI crawler requests in its logs. By August 9 it was 37.8%.
The timeline came in stages rather than as a smooth ramp:
- The share hovered around 2% until July 17.
- It spiked to roughly 23% between July 20 and 22.
- It surged again from August 5 and peaked at 37.8% on August 9.
- August 10 was a partial day, so ignore it when reading the chart.
In plain terms, it went from about 1 in 50 AI crawler requests to more than 1 in 3, and on August 9 it was the heaviest AI crawler in those logs. One bot. That's the part I keep coming back to.
How the percentage is calculated
The method is worth understanding before you quote the number anywhere. The share is requests with the meta-webindexer user agent divided by requests from all AI crawlers Promptwatch tracks, identified by user agent. It is not total site traffic.
Meta's other crawlers (meta-externalagent, meta-externalfetcher, FacebookBot) sit in the denominator but not the numerator. Googlebot, Bingbot, link-preview fetches and human visitors are excluded. The tracked list includes OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot and Claude-User, Perplexity's bots, Google-Extended, xAI, Mistral, Cohere and DeepSeek crawlers.
Two caveats, and I'd rather state them than hide them:
- The share is user-agent based, so spoofed traffic claiming to be Meta-WebIndexer could inflate it.
- I found no later Promptwatch update confirming whether the 37.8% level held after August 9.
Why would Meta build its own index?
The cause is not confirmed by Meta. Here is what is known.
On August 6, 2026, the developer @levelsio posted that Meta staff had messaged him saying Meta is allegedly building its own web index. The stated aim was that Meta AI web searches wouldn't end up at Google, which could learn from them. He also said Meta was scraping his sites hard enough to trigger load-average alerts on a VPS, and was hitting his url2og screenshot service. That's an alleged, secondhand report. Promptwatch's own wording is careful: the surge "lines up with" it.
The logic is easy to follow though. A PPC Land explainer notes that Meta AI has depended on Google and Bing for live web answers, which exposes query data to competitors and carries a cost. Owning the index fixes both. Crawling at this scale and speed is what building an index from scratch looks like.
A PPC Land summary also says the crawler overtook Meta's training crawler in June 2026. So the shift from training-heavy crawling to retrieval-heavy crawling was already underway before the August spike.
How this compares with the earlier 52% figure
You may have seen a bigger Meta number elsewhere. Fastly's Q2 2025 Threat Insights Report, covered by SDxCentral, found Meta generating 52% of AI crawler requests from mid-April to mid-July 2025. Google was at 23% and OpenAI at 20%, together making 95% of the volume.
Don't stack that on top of the 37.8% figure. They are different datasets, different periods (2025 versus 2026) and different methods. The Fastly number covers all Meta AI crawlers, mostly the training and ingestion side, as measured on Fastly's network. The Promptwatch number is one specific indexing crawler's share of the AI crawlers in Promptwatch's logs.
Other sources disagree wildly too. HUMAN Security data reported by PPC Land put OpenAI at about 69% of observed AI-driven traffic, Meta at roughly 16% and Anthropic at about 11%. Promptwatch's own verified-IP report shows OpenAI at 79.8% for August 31 to September 6, 2026, down from 94.8% in early June, but that report only covers OpenAI, Anthropic, Google, Perplexity and Mistral. Meta isn't in its provider list, so it can't be compared with the 37.8%.
The lesson: every number depends on who is measuring, which bots they count, and whether they verify IPs. Treat them as directional. Your own logs are the only number that applies to your site.
| Source | Period | Meta's reported share | Basis |
|---|---|---|---|
| Fastly Threat Insights | Mid-April to mid-July 2025 | 52% of AI crawler requests | Fastly network, all Meta AI crawlers |
| HUMAN Security via PPC Land | Not specified in our notes | About 16% of AI-driven traffic | Observed AI-driven traffic by volume |
| Promptwatch Data | Mid-July to Aug 9, 2026 | 2.2% rising to 37.8% | meta-webindexer user agent only, share of tracked AI crawlers |
What it means for your site
The upside: Meta AI citations
Meta states that allowing the crawler makes your content eligible for citation and linking in Meta AI responses. If Meta AI becomes a meaningful answer surface, being missing from its index means being missing from its answers. Meta AI sits inside apps with a huge audience, so this is not a rounding error for brands that care about discovery.
It's the same dynamic as ChatGPT and Perplexity: no crawl, no citation. If you want to see how citation behavior varies by platform, Promptwatch's AI crawler traffic report is a useful companion, and its ChatGPT citation share by content type data shows what kinds of pages tend to get picked up once a crawler can read you.
The downside: load and cost
The infrastructure complaints are real. A Reddit thread in r/webdev titled around Meta's AI crawler hitting a site 7.9 million times in 30 days describes CPU pegged at 300% and memory pressure, with a caveat from another commenter that some of it was likely bad bots spoofing Meta. A Cloudflare Community thread describes excessive repeated requests, and the complaint there is a good one: Cloudflare rate limiting is IP-based, while these crawlers rotate IPs constantly, so classic rate limiting mostly misses.
If your hosting bills by request or bandwidth, a sustained crawler at this scale shows up on an invoice. Promptwatch's advice is to budget for sustained Meta crawler traffic instead of treating it as a one-off spike. Fair.
The ads angle
PPC Land also points out that the underlying interactions feed Meta's advertising personalisation. I'd treat that as the publication's interpretation rather than something to build a policy on, but it explains why some publishers are wary of the robots.txt decision.
How to check whether Meta-WebIndexer is hitting you
Start with your raw access logs. Search for the user agent string.
# Count requests by Meta crawlers in an nginx access log
grep -iE "meta-webindexer|meta-externalagent|meta-externalfetcher|facebookbot" /var/log/nginx/access.log \
| awk -F'"' '{print $6}' | sort | uniq -c | sort -rn
Then compare against total requests to see the share on your own site. If you use a CDN, pull the same data from Cloudflare, Fastly, CloudFront or whatever you run.
Verify before you trust the user agent
User agents are trivially faked. Both the Reddit and Cloudflare threads suggest a chunk of the heavy "Meta" traffic may be impostors. Before you block or celebrate, check that the source IPs belong to Meta's published ranges. Real Meta crawler traffic resolves to Meta infrastructure; spoofed traffic doesn't. If the IPs don't match, that is a bot management problem, not a Meta policy decision, and robots.txt won't stop it because impostors ignore it.
Tools that log AI crawler activity with IP verification can save you the manual work. Promptwatch does this through its AI crawler logs and filters spoofed bots, and the logs show which pages each crawler read and which errors it hit.
For enterprise technical SEO teams that already live in log files, Botify is another option.
Controlling it in robots.txt
Each Meta crawler has its own token, so you can be selective. Here are three common setups.
Allow indexing for Meta AI citations but refuse training:
User-agent: meta-webindexer
Allow: /
User-agent: meta-externalagent
Disallow: /
Block the indexer entirely:
User-agent: meta-webindexer
Disallow: /
Slow it down on a sensitive section without blocking:
User-agent: meta-webindexer
Disallow: /search/
Disallow: /api/
Disallow: /cart/
I'd be careful with Crawl-delay. Support is inconsistent and I wouldn't assume Meta honors it. If load is the problem, enforce limits at the edge by verified IP range or ASN rather than by trusting a directive.
A quick decision guide
| Your situation | Sensible approach |
|---|---|
| Publisher or brand that wants AI citations | Allow meta-webindexer, verify IPs, watch load |
| Content you sell or license | Block meta-webindexer and meta-externalagent, enforce at the edge |
| Small site on metered hosting | Allow, but rate limit by verified range and cache aggressively |
| Unsure | Allow the indexer, block training, review in 30 days |
Blocking has a cost. If Meta is building its own index, as reported, a blocked site is simply not in it at launch. Promptwatch makes the same point: whether your content was crawled during this build-out may determine whether it can be cited.
Treating Meta as an AI search surface
Most teams track ChatGPT, Gemini, Perplexity and Google AI Overviews. Meta AI is the one that tends to be missing from the dashboard. I'd put it on the watch list now, mostly because the crawl data suggests Meta is serious about retrieval, and the 2 to 38 jump happened fast enough that most people noticed only when their servers complained.
A reasonable first month:
- Pull 30 days of logs and measure Meta-WebIndexer's share of your bot traffic.
- Verify the IPs against Meta's ranges and separate real from spoofed.
- Decide your robots.txt policy per Meta token, not for "Meta" as a whole.
- Check which pages the crawler actually reads and whether it hits errors like 403s or timeouts.
- Track whether Meta AI starts citing you. That part is hard to measure with most tools today.
If you want a broader view of platforms that monitor AI visibility, the GEO software directory at bestgeosoftware.com lists options. Most of them track mentions and citations; fewer show crawler behavior, which is the part this story is really about.
What we don't know yet
- Whether Meta has confirmed an independent search index. The report comes secondhand from a public post.
- Whether the 37.8% share held, fell back, or climbed after August 9. A spike like this can be a bulk initial crawl that tapers once the index is seeded.
- How much of the measured traffic is spoofed.
- How prominently Meta AI will cite sources once the index is live.
If the initial crawl was the big push, the share could settle well below the peak. If it's continuous, expect sustained load. Either way, the numbers in your own logs will tell you before any industry report does.
Bottom line
Meta-WebIndexer is the crawler that feeds Meta AI search. It went from a minor bot to more than a third of tracked AI crawler requests in about three weeks, and the likeliest explanation is an index build. Verify the traffic is real, pick a robots.txt policy for each Meta token separately, and watch your server load. Treat Meta AI as a citation surface alongside ChatGPT and Perplexity, because the crawl data says Meta is treating it as one.