Key takeaways
- Promptwatch's crawler logs show Meta-WebIndexer at about 2.2% of tracked AI crawler requests in mid-July 2026 and 37.8% on August 9. That is roughly 17x in under a month.
- It is Meta's indexing crawler for Meta AI search. Meta says allowing it helps Meta AI cite and link to your content. It is separate from Meta's training, ads, and link-preview bots.
- Decide per crawler, not per company. Blocking the training bot while allowing the indexer and link previews is a sensible default for many sites.
- New Meta Commercial Terms take effect October 30, 2026 and add a clause letting Meta crawl websites tied to business accounts. Legal experts say the wording is broad.
- Write the robots.txt rule, then check your logs to confirm it holds. Plan for crawl load if your hosting bills per request.
What happened to Meta-WebIndexer in 2026
For most of 2026 Meta-WebIndexer was background noise. Then it was not.
According to Promptwatch Data's report on Meta's crawling surge, the bot made up about 2.2% of all AI crawler requests in mid-July 2026. By August 9 it was at 37.8%. About 1 in 50 requests became more than 1 in 3. In the same logs it was the heaviest AI crawler at that point.
The timeline in that data is bumpy:
- Share hovered around 2% until July 17.
- It spiked to roughly 23% between July 20 and 22.
- It passed 26% around August 6.
- It surged from August 5 to the 37.8% peak on August 9.
Two caveats, and I would not skip them. First, this is a share of tracked AI crawler requests, not of all web traffic. Googlebot, Bingbot, FacebookExternalHit and human visitors are excluded from the calculation. Second, I could not find an updated figure after August 9, so nobody outside Meta can say whether that level held. Treat 37.8% as a snapshot, not a steady state.
PPC Land's explainer adds that the indexer overtook Meta's training crawler in June 2026. That matters, because it means the growth is in the indexing bot specifically, not in Meta's AI-training or link-fetching bots.
Why would Meta do this?
On August 6, 2026, the developer @levelsio posted that Meta staff had told him Meta is allegedly building its own web index so Meta AI searches don't go through Google. He also said the crawling was heavy enough to trigger load-average alerts on his VPS. That is anecdotal and Meta has not confirmed it. But the traffic pattern fits the story: a crawler that suddenly dominates looks a lot like someone building a search index from scratch.
Meta's crawlers are not one thing
The most common mistake I see is treating "Meta" as a single bot. Meta's own crawler documentation (updated May 21, 2026) lists several, with different jobs.
| User agent | What Meta says it does | Robots.txt behaviour |
|---|---|---|
| meta-webindexer | Improves Meta AI search quality; helps Meta AI cite and link to your content | Honours robots.txt per Digital Applied; Meta's doc shows no explicit statement |
| meta-externalagent | Training foundation AI models or improving products by indexing content directly | Honours robots.txt |
| meta-externalads | Crawls the web to improve advertising and business products | Honours robots.txt |
| meta-externalfetcher | Fetches individual links at a user's request, supports agentic AI tasks | May bypass robots.txt |
| facebookexternalhit | Builds link previews on Facebook, Instagram, Messenger | May bypass robots.txt for security or integrity checks |
Note the gap: Meta's documented robots.txt example only shows meta-externalagent and meta-externalads. For Meta-WebIndexer, disallowing it by its user-agent token is the implied method, and Digital Applied's reading of the docs says it is honoured. Verify in your own logs rather than trusting any of us.
The October 30 commercial terms complicate things
There is a second Meta story running in parallel, and it is easy to conflate with the crawler surge. Don't. They are related in spirit, but they are different things.
Meta's revised Commercial Terms take effect October 30, 2026 for businesses that run ads, manage a Page, or use Meta's business tools. A new clause authorises Meta, or anyone acting for it, to crawl and analyse websites associated with those businesses. Nieman Lab reported on October 15 that three legal experts thought the wording could be used to argue Meta can scrape news sites and use the content to train AI products. One of them said the broadest reading would build "a comprehensive archive of all content" Meta could serve as it wanted.
The next day Meta revised the text. According to Digital Applied's breakdown, the word "train" was removed and the purpose narrowed to advertising and business products. The lawyers' objections remain, as summarised there:
- "Develop" is broad enough that AI training for business products could arguably fit.
- "Business product" is not a defined term.
- The "associated with your products or services" scope is unchanged and could cover entire sites.
- Robots.txt states a preference, not an enforceable right, so the terms could be argued to supersede it.
Meta says nothing has changed in how its crawlers behave and that robots.txt still works. There is no opt-out short of not using Meta's business products. This is not legal advice, and if you run a publishing business with licensing ambitions, get a lawyer to read the actual terms.
Should you block or allow Meta-WebIndexer?
There is no universal answer. It depends on what you want from Meta AI.
The case for allowing it
Meta says allowing the indexer helps it cite and link to your content in Meta AI's responses. If Meta AI is becoming a real discovery surface, blocking its indexer decides whether you are in the index at launch. Promptwatch's advice is to treat Meta as an emerging AI search surface with the same citation dynamics as ChatGPT and Perplexity. For brands that live on discovery, that is a reason to stay visible.
The case for blocking it
Three reasons come up:
- Load. The crawler can be heavy, and some site owners in Cloudflare's community forum described it arriving from many rotating IPv6 addresses at once in March 2026.
- Cost. If your infrastructure bills per request, sustained crawling is a line item.
- Licensing. Publishers who want to be paid for content may prefer to hold back.
A sensible default
Blocking the AI-training crawler while allowing link previews and ad crawling is a reasonable default for most businesses. Publishers with licensing plans should decide deliberately instead of by default.
| Your situation | Suggested stance |
|---|---|
| Brand or SaaS site that wants AI citations | Allow meta-webindexer, decide on meta-externalagent separately |
| Publisher with licensing goals | Block training and indexing bots until you have a deal or a policy |
| Small site on metered hosting | Allow, but rate-limit and watch the logs |
| Ecommerce or social-driven site | Keep facebookexternalhit working at all costs |
How to set the rules
Blocking works through a Disallow rule for the user agent in robots.txt. A minimal example that blocks the training bot but allows the indexer:
User-agent: meta-externalagent
Disallow: /
User-agent: meta-webindexer
Allow: /
To block the indexer instead, swap the directives. Meta says to allow up to 24 hours for changes to take effect because crawlers may cache the file. Meta also prefers robots.txt over non-standard signals such as NoAI tags, so don't rely on the tags.
Don't break link previews
FacebookExternalHit generates the title, description and thumbnail when someone shares your link. If you block it, your shares look broken. Meta's requirements are specific:
- Support gzip and deflate encodings.
- Put Open Graph tags within the first 1 MB.
- Make content crawlable within a few seconds.
- Handle or ignore the Range header.
- Allowlist by user agent, or more securely by IP.
You can test with the Sharing Debugger, or simulate a request:
curl -v --compressed -H "Range: bytes=0-524288" -H "Connection: close" -A "facebookexternalhit/1.1" $URL
Remember that two bots may ignore robots.txt
Meta-ExternalFetcher and FacebookExternalHit may bypass robots.txt, according to Meta's docs. If you need hard enforcement, use server or CDN rules, not just a text file. Because the indexer has been seen on rotating IPv6 addresses, token-based rules (matching the user-agent) are less painful than address-level blocking.
Verify with your logs
A robots.txt rule is a request. Logs are the proof. Spoofing is real too: DataDome's Q2 2026 AI traffic report warns that any allowlist granting automatic access by trusted agent name, including Meta-ExternalAgent, is exposed, and recommends assuming spoofing of well-known identities in your design.
A practical routine:
- Pull 30 days of server or CDN logs and filter for meta-webindexer, meta-externalagent, meta-externalfetcher, and facebookexternalhit.
- Compare request counts per day against your baseline. A spike in the 20x range is not a typo.
- Check whether bots you disallowed are still arriving after 24 hours.
- Verify source IPs for anything you plan to allowlist.
- Watch for deploy timing. One TechSEO thread analysing 48 days of server logs notes that if you ship changes during off-peak US hours, AI bots may be the first to see them, and Meta was the most aggressive by volume.
If you would rather not grep logs by hand, platforms with AI crawler analytics can do the sorting. Promptwatch logs IP-verified AI crawler requests, filters out spoofed bots, and shows which pages were read and whether they errored.
For dedicated bot management and technical crawl analysis at enterprise scale, Botify is another option.
If you are comparing more options, the GEO software directory at bestgeosoftware.com lists a wider set of tools.
Check whether crawling turns into citations
A crawl is not a citation. The point of letting Meta-WebIndexer in is to show up in Meta AI answers, so measure that. Look at which of your pages get cited, not just which get fetched. Promptwatch's crawl-to-citation path shows a citation rate per page, which is the quickest way to spot pages that get read but never used. Cross-platform context helps too: Promptwatch's data on ChatGPT citations by content type shows product pages leading at 22.4% of citations (Sep 28 to Oct 4, 2026), a useful reference for which formats AI search tends to reward.
Other visibility trackers can fill the same gap if you prefer. Profound and Peec AI are common picks for prompt-level monitoring, though neither centres on crawler logs.
A checklist for this week
- Audit robots.txt for meta-webindexer, meta-externalagent, meta-externalads, and facebookexternalhit.
- Decide per crawler, and write the decision down. Someone will ask in six months.
- If you run Meta ads or a business Page, read the October 30 Commercial Terms or have counsel do it.
- Confirm Open Graph tags sit in the first 1 MB so link previews keep working.
- Set rate limits or caching if crawl load is hurting a small server.
- Re-check logs 24 to 48 hours after changing rules.
- Email [email protected] if a Meta crawler behaves unexpectedly.
What to take from the number
The 38% figure is a share of a pool of AI crawlers that Promptwatch tracks, measured over a few weeks in the summer. It will move. What will not move is the decision in front of you: Meta now has a crawler built for AI search, a set of terms that lets it read the sites behind business accounts, and a robots.txt file that you control for most, though not all, of its bots. Write the rule, then read the logs.