Meta-WebIndexer Went From 2% to 38% of AI Crawler Traffic: What Site Owners Should Do Now (2026)

Key takeaways

• Meta-WebIndexer rose from roughly 2.2% of tracked AI crawler requests in mid-July 2026 to 37.8% on August 9, 2026, per Promptwatch Data. That is about 17x in under a month.
• It is the indexing crawler for Meta AI search. It is separate from Meta's training bot (Meta-ExternalAgent) and its link fetcher (Meta-ExternalFetcher), and each has its own robots.txt token.
• Check your logs first. Then decide per crawler whether you want to be in Meta AI answers, and watch your infrastructure bill if you pay per request.
• Block by user agent or token, not by IP. Meta's addresses rotate heavily, especially IPv6.
• Meta has not confirmed a search engine launch. The index is reported and alleged, so treat this as a bet on an emerging surface, not a settled one.

What actually happened

In mid-July 2026, Meta-WebIndexer accounted for about 2.2% of all AI crawler requests in the logs Promptwatch tracks. The share sat near 2% until July 17, spiked to roughly 23% between July 20 and 22, then climbed from August 5 to a peak of 37.8% on August 9. The report, written by Klaas Foppen, calls it "1 in 50 to 1 in 3" AI crawler requests and says it made Meta-WebIndexer the heaviest AI crawler in the dataset.

Two caveats on the number. First, it is a share of AI crawler requests only. The numerator is the meta-webindexer user agent. The denominator is every AI crawler Promptwatch tracks, including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Mistral, Cohere, DeepSeek and Meta's other bots. Googlebot, Bingbot, FacebookExternalHit and human visits are excluded. So this is not 38% of your traffic. Second, August 10 was a partial day, so the peak is August 9.

The surge is also specific. It comes from the indexing crawler, not from Meta's AI-training or link-fetching bots.

For context, Radware's April 2026 telemetry (before the surge) had AI crawlers at around 30% of legitimate bot traffic, with meta-externalagent at 11.49% of AI crawler requests. Radware uses a different methodology, so don't compare its percentages directly with Promptwatch's.

Why Meta is crawling like this

On August 6, 2026, @levelsio posted that Meta staff told him Meta is "allegedly" building its own web index. He also reported heavy scraping that triggered load-average alerts on his VPS, with Meta hitting his url2og screenshot service. That fits an older report: The Information wrote on October 28, 2024 that Meta was building an AI search engine to reduce its reliance on Google and Bing.

What Meta itself documents is narrower. Per its crawler page (as summarized by PPC Land), Meta-WebIndexer exists to improve Meta AI search result quality, and allowing it in robots.txt helps Meta cite and link to your content in Meta AI answers. I could not find confirmation of a public search engine launch, or any official crawl-rate guidance. Everything beyond "it feeds Meta AI search" is inference.

The demand side makes the bet plausible. Similarweb data cited by IAB Australia put Meta AI's US monthly active user growth at 435% between June 2025 and May 2026, the fastest of the assistants measured. IAB also said Meta-WebIndexer overtook Meta's training crawler in June 2026.

Meta's five crawlers are not the same bot

The single most useful thing to know: Meta runs several user agents with different jobs, and you can treat them differently.

That split means you can allow the search index but refuse training, or the reverse. It also means blocking only one token will not stop Meta traffic overall. Many site owners who blocked meta-externalagent last year are now seeing a different Meta user agent in their logs.

Step 1: check your own logs

Before changing anything, find out how much Meta is actually hitting you. Search your access logs for these strings, case-insensitively:

meta-webindexer
meta-externalagent
meta-externalfetcher
facebookbot

The user agent looks like meta-webindexer/1.1 (+link to Meta's webmaster docs) or the short form meta-webindexer/1.1. Count requests per day, bytes served, and which paths get hit. Look for patterns such as faceted navigation, search result pages, or image and screenshot endpoints, because crawlers waste the most of your bandwidth there.

Some anecdotes show what the worst case looks like. One r/webdev poster reported 7.9 million Meta AI crawler requests and over 900 GB in 30 days, another reported 741,900 hits in a month. Both are single-site reports, not benchmarks, but they explain why the topic got loud. Cloudflare community threads in March 2026 also describe Meta's crawlers generating repeated requests for the same URLs.

If your site sits behind a CDN, pull the logs there rather than at the origin, since cached hits never reach your server. Promptwatch can also ingest crawler logs from Cloudflare, CloudFront, Fastly, Vercel, Netlify and others, and filters out spoofed bots using IP verification, which is useful because anyone can fake a user agent string.

Compare against the wider mix

Promptwatch's AI crawler traffic report tracks IP-verified requests by provider. OpenAI made up 79.8% of verified requests in the week of August 31 to September 6, 2026, down from 94.8% in June 8 to 14. That report names OpenAI, Anthropic, Google, Perplexity and Mistral, and I did not see Meta in the page text, so you can't line it up with the Meta-WebIndexer share. Still, the method is handy: if a provider is large in the overall mix but near zero in your logs, you are probably blocked or unreachable for it.

Step 2: decide whether you want in

This is a business call, not a technical one. Roughly:

• Publishers who monetize with ads and don't expect Meta AI referral traffic may reasonably block.
• Brands, SaaS companies, ecommerce stores and local businesses that want to be recommended in AI answers have a reason to allow it. Meta AI is growing fast, and being absent from its index means being absent from its answers.
• Sites with expensive dynamic pages may want to allow it but limit what it can reach.

I lean toward allowing the indexer for most commercial sites and being stricter about training. But I would not make that choice blind. Nobody has published numbers on how much referral traffic Meta AI sends to cited sites, so you are paying real bandwidth for an unproven channel. Measure what you get before and after.

One more thing to weigh: the October 30, 2026 change to Meta's commercial terms. Nieman Lab, via a Michelle Gifford newsletter post, reports the terms include language allowing Meta to send bots to advertisers' websites to copy content for product training, including Meta AI, with no opt-out inside the terms. Legal experts quoted there treat it as permission for scraping. This is separate from robots.txt, and the practical effect on crawler behavior is not yet visible in the data I have.

Step 3: write robots.txt rules that match your decision

The token is meta-webindexer. Meta matches it case-insensitively per RFC 9309 and says it prefers robots.txt over non-standard signals like NoAI tags.

Allow the indexer, block training:

User-agent: meta-webindexer
Allow: /

User-agent: meta-externalagent
Disallow: /

Block the indexer entirely:

User-agent: meta-webindexer
Disallow: /

Allow it, but keep it out of expensive areas:

User-agent: meta-webindexer
Disallow: /search
Disallow: /cart
Disallow: /*?filter=
Disallow: /*?sort=

Two practical notes. Meta caches robots.txt, and changes can take up to 24 hours to register, so do not read same-day logs as non-compliance. And I did not find any official crawl-delay guidance from Meta, so don't count on a Crawl-delay line being honored.

If you want a quick starting point for AI bot rules, Promptwatch has a free robots.txt generator for AI crawlers, and its bots directory at promptwatch.com/bots lists 440+ crawlers.

Step 4: enforce at the edge if you need to

Robots.txt is a request. If load is the problem, enforce it.

Blocking by IP is the wrong tool here. Meta publishes no per-crawler JSON of IP ranges. It tells operators to query the RADb routing registry for prefixes originating from AS32934. Observed ranges include 157.240.14.0/24, 31.13.73.0/24 and IPv6 under 2a03:2880::/32, and Meta says the addresses change often. Site owners in Cloudflare's forum in March 2026 described the crawler arriving from many rotating IPv6 addresses at once. Address-level rules become a chore.

Use user-agent or token matching in your CDN or WAF instead. In Cloudflare, that means a custom rule matching the user agent string meta-webindexer, with an action of block or managed challenge. Equivalent rules work in Fastly, Akamai, CloudFront with WAF, or an Nginx map. I did not find primary-source rule snippets from Meta or Cloudflare, so test any rule in log-only mode first and confirm it matches before switching to block.

Open-source proxies are also moving: an Anubis feature request on April 16, 2026 asked for a default block rule for Meta-WebIndexer.

Verify before you trust a user agent

A user agent string proves nothing. If you are blocking, a spoofer just changes it. If you are allowing, a spoofer may be draining your server while pretending to be Meta. Compare source IPs against AS32934 prefixes for anything high-volume.

The robots.txt trust problem

To be fair about the limits: robots.txt only works if the crawler honors it. The ai.robots.txt registry, a volunteer project and not independent testing, lists Meta-WebIndexer's compliance as "unclear" with a crawl frequency above one request per second. Documents leaked to Drop Site News in August 2025 claimed Meta harvested content from 6 million sites, ignored robots.txt and reached content through CDNs. Those are allegations, not findings.

I would not build a legal or revenue-critical strategy on robots.txt alone. If your content is licensed or paywalled, enforce at the edge and keep logs.

What the shift means for your costs

Cloudflare figures cited by IAB Australia put automated requests at 57.5% of web page requests as of June 2026, the first time past human traffic. IAB also found mixed-purpose crawling fell from about 49% to about 33% of AI requests in the first half of 2026 as operators split composite bots into narrower tokens. That is why the Meta change matters: purpose-specific tokens are becoming normal, and your robots.txt needs entries per purpose, not one blanket AI rule.

If you pay per request (serverless functions, image resizing, origin egress, or metered CDN), budget for sustained crawler load. Cache aggressively, serve static HTML where you can, and rate limit expensive endpoints.

Step 5: track whether being indexed leads to anything

Allowing a crawler is only half of it. The question is whether Meta AI cites you. Right now there is no standard way to see Meta AI referral data, so you have to cobble it together:

• Watch referrers and landing pages for traffic from Meta AI surfaces.
• Check crawl-to-citation patterns: which pages get fetched, and which get cited.
• Run a fixed set of prompts through Meta AI regularly and record whether your brand appears.

Platforms that cover this include Promptwatch, which monitors Meta AI's real interface alongside ChatGPT, Google AI Overviews, Perplexity and others. Others in the category are in the table below. For a broader list, the GEO software directory at bestgeosoftware.com is a good place to compare.

Most of the monitoring-only tools stop at "were you mentioned." The ones worth paying for connect crawler activity, citations and traffic, so you can tell whether allowing a bot changed anything.

A short checklist

1. Grep your logs for the four Meta user agents and record a baseline.
2. Decide separately on Meta-WebIndexer, Meta-ExternalAgent and Meta-ExternalFetcher.
3. Update robots.txt, then wait 24 hours before judging compliance.
4. If load is a problem, add a user-agent rule at your CDN or WAF, not an IP list.
5. Cache and rate limit expensive endpoints either way.
6. Re-check in two to four weeks. This share moved from 2% to 38% in under a month, and it could swing again.

What we still don't know

Meta has not confirmed a public search engine. I found no official crawl-rate guidance, no sample WAF rules from Meta, and no data on referral traffic from Meta AI. The crawl surge is measured. The reason behind it is reported secondhand. Treat your decision as reversible: pick a setting, measure for a month, and change it if the numbers say so.

For more first-party data on how AI systems fetch and cite pages, see the research hub at Promptwatch Data. If you want help sorting out technical SEO and AI crawler policy across a larger site, the team at 1001 SEO Media, which publishes this site, handles that kind of audit.