Meta-WebIndexer and Beyond: Which AI Crawlers Deserve a Place in Your robots.txt in 2026

Meta-WebIndexer went from 2% to 38% of tracked AI crawler requests in a month. Here is how to sort training bots, search indexers and user fetchers, and write a robots.txt that matches your goals.

Key takeaways

  • AI crawlers fall into three jobs: training, search indexing, and user-triggered fetching. Each deserves a different robots.txt decision.
  • Meta-WebIndexer indexes content for Meta AI search and citations. Per Promptwatch Data, its share of tracked AI crawler requests rose from about 2.2% in mid-July 2026 to 37.8% on August 9.
  • User-triggered fetchers (Meta-ExternalFetcher, ChatGPT-User and similar) may ignore robots.txt, so hard blocking has to happen at the CDN or server.
  • Meta's own robots.txt sample only mentions meta-externalagent and meta-externalads. Do not assume your Meta rules cover the indexer.
  • Robots.txt only expresses intent. Your server logs show whether crawlers actually arrive.

Why robots.txt suddenly matters again

For years robots.txt was a file most teams set once and forgot. Then AI crawlers showed up, and every provider shipped three or four user agents with slightly different jobs. Block the wrong one and you vanish from an AI answer engine. Allow the wrong one and your content trains a model you never agreed to feed.

The Meta numbers are what made me rewrite my own checklist. Promptwatch tracks IP-verified AI crawler requests, and its Meta-WebIndexer report shows the crawler going from roughly 1 in 50 AI crawler requests in mid-July 2026 to more than 1 in 3 by August 9. That made it the heaviest AI crawler in their logs at the time. One crawler, under a month.

Favicon of Promptwatch

Promptwatch

Track and improve your AI search visibility
View more
Screenshot of Promptwatch website

A caveat on the cause. The report links the surge to claims that Meta is building its own search index so Meta AI does not depend on Google or Bing. That claim traces back to an August 6 post by @levelsio, who said Meta staff told him so, and who also reported load alerts on his VPS. Meta has not confirmed it. The crawl volume is measured. The motive is not.

Promptwatch Data report showing Meta-WebIndexer's daily share of AI crawler requests

The three jobs an AI crawler can do

Before you touch the file, sort every bot into one of three buckets.

TypeWhat it doesExamplesWhat blocking it changes
Training crawlerCollects content that may train foundation modelsGPTBot, ClaudeBot, meta-externalagent, Applebot-ExtendedWhether your content feeds future models
Search crawlerIndexes pages so an AI search product can retrieve and cite themOAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexerEligibility to appear in current AI answers
User-triggered fetcherLoads a URL live because a person askedChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcherWhether an assistant can read your page on request

The mistake I see most is treating the whole list as one thing. "Block AI" sounds like a policy, but it is really three separate decisions.

Meta's crawlers, one by one

Meta documents four relevant user agents on its web crawlers page, last updated May 21, 2026.

Meta's developer documentation page listing its web crawlers

Meta-WebIndexer

The user agent is meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler). The research firm 51Degrees classifies it as used purely for indexing Meta AI search results, and says Meta notes that allowing it supports Meta AI citing and linking to your content. If you want to be a source in Meta AI answers, this is the one to allow.

One gap in my research: I could not confirm from Meta's own documentation whether Meta-WebIndexer honors robots.txt. A third-party index page claims it does, and one aggregator list marks it as obeying the file, but check your logs after changing rules rather than trusting either.

Meta-ExternalAgent

Meta describes it as crawling the web for training foundation AI models or improving products by indexing content directly. This is the training-flavored bot. Blocking it is the closest thing to a Meta training opt-out.

Meta-ExternalFetcher

It fetches individual links at a user's request and supports agentic features, including helping AI navigate sites to complete tasks. Meta says it may bypass robots.txt. A Disallow here is a request, not a wall.

FacebookExternalHit

This builds link previews when someone shares your URL on Facebook, Instagram or Messenger. It can bypass robots.txt for security and integrity checks such as malware scans. Blocking it breaks your share cards, which is rarely what anyone wants.

Meta crawlerJobHonors robots.txt?Typical stance
meta-webindexerMeta AI search index and citationsNot confirmed by Meta docsAllow if you want citations
meta-externalagentTraining and direct indexingYes, per Meta's sampleYour call on training
meta-externalfetcherUser-requested fetchesMay bypassAllow, enforce at CDN if needed
facebookexternalhitLink previewsMay bypass for security checksAllow

51Degrees breakdown of Meta-ExternalAgent and Meta-WebIndexer

Note that 51Degrees warns a blanket disallow of every Meta user agent can mean missing discovery inside Meta's services. That matches what the Promptwatch report suggests: treat Meta as an emerging AI search surface, not only a social network.

The rest of the list

Meta is one of several operators. The set Promptwatch tracks in its crawler logs gives a decent working inventory:

  • OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot
  • Anthropic: ClaudeBot, Claude-User, Claude-SearchBot, claude-web
  • Google: Google-Extended, Google-Agent
  • Perplexity: PerplexityBot, Perplexity-User
  • xAI: GrokBot, xAI-Grok, Grok-DeepSearch
  • Mistral: MistralAI-User, MistralAI-Index
  • Cohere: cohere-ai
  • DeepSeek: DeepSeekBot

Two details matter in practice. Google-Extended is a control token for Gemini training and grounding, and it does not affect Googlebot or ordinary Search. Secondary guides note that it combines training and grounding, so Disallow rejects both. And several operators, including OpenAI, Perplexity, Meta and Amazon, say in their docs that user-triggered fetchers may not follow robots.txt.

For a longer reference, the AI Crawler URL Inspector and the AI bots directory cover 400+ agents with robots.txt guidance.

Decide your policy first

Pick the row that matches your business, then write the file.

Your goalAllowBlock
Maximum AI visibilityEverything documentedNothing
Visible in AI answers, no trainingSearch crawlers and user fetchers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, meta-webindexer, meta-externalfetcher)GPTBot, ClaudeBot, meta-externalagent, Applebot-Extended
Paywalled or licensed contentLittle or nothingTraining and search bots, plus CDN-level enforcement
Local business or ecommerceSearch and fetcher bots, so products and locations can be citedTraining bots, if you prefer

I lean toward the second row for most publishers and brands. Training opt-outs are a values and licensing decision. Search inclusion is a revenue decision. Mixing them up is how brands end up absent from answers they could have won.

It is worth knowing what the citation data says before you block anything. Promptwatch Data shows product pages are the most cited content type in ChatGPT at 22.4% of citations (Sep 28 to Oct 4, 2026), and that landing pages and docs stay cited longest. If those are the pages you most want cited, make sure search crawlers can read them.

A starter robots.txt

This version allows search and user fetchers and blocks training. Adapt it to your row in the table above.

# OpenAI
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Disallow: /

# Anthropic
User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# Meta
User-agent: meta-webindexer
Allow: /

User-agent: meta-externalfetcher
Allow: /

User-agent: meta-externalagent
Disallow: /

# Google: keep Allow for Gemini grounding, Disallow rejects training and grounding
User-agent: Google-Extended
Allow: /

# Apple
User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Disallow: /

Policies change, and I did not re-verify every operator's current documentation for this guide, so check each vendor's page before shipping. You can also generate a file with the Robots.txt Generator for AI and sanity check it against the Agentic Readiness Check.

robots.txt is not enforcement

Three things trip people up.

Fetchers can ignore you. If a Disallow must hold, use your CDN, WAF or web server. Verify by IP range, because user agent strings are trivially spoofed.

Spoofed bots exist. Promptwatch only counts requests that match each operator's published IP ranges. Your analytics probably does not, so raw user agent counts overstate real crawler traffic.

Load is real. The Promptwatch report notes Meta's volume was high enough to trigger load alerts on independent sites. If your hosting bills per request, budget for sustained crawl traffic rather than a one-week spike. A Crawl-delay line is not honored by everyone, so rate limiting at the edge is safer.

Check your logs against the market

A clean robots.txt tells you nothing about who shows up. Promptwatch's AI crawler traffic report (IP-verified weekly data, June 8 to September 6, 2026) put OpenAI at 79.8% of tracked requests in the week of August 31, down from 94.8% in the week of June 8. Its advice is simple: if a provider is big in the market mix but close to zero in your logs, you are probably blocked or unreachable for it. Check robots.txt, CDN rules and WAF settings for that provider.

Note that this report covers OpenAI, Anthropic, Google, Perplexity and Mistral. The Meta report's denominator also includes xAI, Cohere, DeepSeek and Meta. Do not compare the two percentages directly.

Here are tools that can help you see this in your own data:

NeedOption
IP-verified crawler logs plus citation trackingPromptwatch
Enterprise technical SEO and bot log analysisBotify
Broad AI visibility monitoringProfound, Peec AI, Otterly.AI
Crawl-based scraping for your own testingFirecrawl, Apify
Favicon of Botify

Botify

Enterprise AI search optimization for technical SEO teams
View more
Screenshot of Botify website
Favicon of Profound

Profound

Enterprise AI search visibility and analytics
View more
Screenshot of Profound website
Favicon of Peec AI

Peec AI

AI visibility tracking with smart suggestions
View more
Screenshot of Peec AI website

Most monitoring tools tell you whether a brand was mentioned. Promptwatch's crawler logs go a step earlier: they show which pages bots read, what errors they hit, and whether a crawled page later gets cited, which separates an access problem from a content problem. It is also why I would pair a robots.txt change with a log check a week later.

If you are comparing options, the GEO software directory lists more of them.

A short audit routine

  1. List every AI user agent in your logs from the past 30 days.
  2. Sort each into training, search or fetcher.
  3. Decide your policy per bucket, not per vendor.
  4. Write the file, then add CDN rules for anything you truly need blocked.
  5. Re-check logs after a week. Did the allowed search bots come? Did the blocked ones stop?
  6. Repeat quarterly. New agents appear constantly, and Meta alone has changed its crawler page this year.

What I would do today

Allow meta-webindexer if Meta AI citations could matter to you, and keep meta-externalagent as your explicit training decision. Do not rely on Disallow for meta-externalfetcher or facebookexternalhit. Then watch your logs. If Meta really is building an index, content crawled during that build is what gets cited at launch, and a robots.txt line written in 2024 will not tell you whether you are in it.

Share:

AI Search Visibility Tools

© 2026 AI Search Visibility Tools · The best AI search visibility tools compared · RSS

AI Search Visibility Tools is an affiliate review site. When you click links to vendors or buy through links on our site, we may earn an affiliate commission at no extra cost to you.

AI Search Visibility Tools is a review website based on user reviews on Reddit and G2, and on publicly available information. We keep everything as up to date as possible, but pricing and features can change. Always confirm the details with the vendor before purchasing.