Bing Copilot's crawler, explained: what it reads and how to track it

Key takeaways

• Copilot Search has no standalone crawler. It retrieves from Bing's existing web index, which is built entirely by Bingbot. If Bingbot can't see your page, Copilot can't cite it.
• Three Microsoft user agents matter in practice: Bingbot (the one that counts for citations), MicrosoftPreview (link previews), and AdIdxBot (ad landing-page checks).
• Never trust a user-agent string alone. Verify Bingbot via reverse DNS against search.msn.com or check IPs against Microsoft's published range list at bing.com/toolbox/bingbot.json.
• Old NOARCHIVE tags left over from CMS security presets can silently block Copilot citations without blocking you from classic Bing results.
• Bing Webmaster Tools now has a first-party "AI Performance" dashboard (public preview since February 2026) that shows exactly which of your pages Copilot is citing and for what queries.

There's no such thing as a "Copilot crawler"

This trips people up constantly, so let's get it out of the way first. Copilot Search doesn't send out its own bot to read the live web when you ask it a question. It can't. The whole system is architected around retrieving from an index that's already been built, not fetching pages in real time.

That index is Bing's, and the bot that builds it is Bingbot. Full stop. Block Bingbot and you don't just drop out of classic Bing search results, you also disappear from Copilot's answers, because there's nothing left for Copilot to retrieve. This is a different mental model than ChatGPT Search, which does more live retrieval. With Bing and Copilot, it's indexed-first, always.

Microsoft actually frames this as a security decision rather than a limitation. Having Copilot fetch arbitrary live URLs at answer time opens the door to prompt injection and data exfiltration attacks. Retrieving only from a pre-vetted, pre-indexed corpus closes most of that off. Makes sense when you think about it that way, though it does mean your content has to clear the indexing bar before Copilot ever gets a shot at quoting it.

The three Microsoft bots you'll actually see in your logs

If you're staring at a server access log trying to figure out which Microsoft requests matter for Copilot visibility, here's the breakdown:

Bingbot (bingbot/2.0; +http://www.bing.com/bingbot.htm) is the one that counts. It builds the index Copilot's Prometheus system retrieves from. Block it, and both Bing search and Copilot lose access to your content.

MicrosoftPreview (MicrosoftPreview/2.0) generates link and page previews across Microsoft surfaces, like the thumbnail card you see when a link is shared in Teams or Outlook. Blocking this degrades how your links preview elsewhere, but it won't kill your Copilot citations.

AdIdxBot (adidxbot/2.0) checks landing page quality for Microsoft Advertising. Irrelevant to organic Copilot visibility unless you're running Bing Ads.

A realistic Bingbot request looks something like Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36. Match on the bingbot token, since the Chrome version number rotates over time and a strict string match will eventually break your rules.

One more thing worth saying plainly: GPTBot, OpenAI's training crawler, has zero relationship to Copilot. Blocking or allowing GPTBot is a completely separate decision from Bingbot. People mix these up more often than you'd expect.

How Copilot actually turns an index into an answer

Under the hood, Microsoft's system is called Prometheus, first described back in February 2023 as a model that combines "the fresh and comprehensive Bing index, ranking, and answers results with the creative reasoning capabilities of OpenAI's most-advanced GPT models." The practical mechanics work through something Microsoft calls the Bing Orchestrator, which sits between the index and the language model.

Simple queries, like "weather in NYC," go straight to the Bing index for an instant answer. Complex or comparative queries route through the LLM, which generates several internal sub-queries, fires them back through Bing's index, and synthesizes the results, all within milliseconds according to Microsoft's own description. This retrieval step is what Microsoft calls "grounding": the model reasons over data Bing hands it, rather than purely over its training data, which is also why it can answer questions about things that happened after its training cutoff and why it can attach verifiable citations to specific claims.

Three data sources feed consumer Copilot answers in practice: Azure-hosted OpenAI models (run on Microsoft's own infrastructure, not OpenAI's public API), the Bing search index, and, for enterprise Microsoft 365 Copilot specifically, Microsoft Graph data from SharePoint, OneDrive, Teams, and Exchange. The web-facing version most SEOs care about only touches the second one.

Verifying Bingbot, and why the user-agent string can't be trusted

Anyone can spoof a user-agent string. If you want to actually confirm a request came from Microsoft, you need reverse DNS: a genuine Bingbot IP resolves to a hostname ending in search.msn.com, and forward-resolving that hostname has to return the same IP. Microsoft's own documentation gives this example: the IP 157.55.39.1 resolves to msnbot-157-55-33-18.search.msn.com.

Easier option for most setups: Microsoft publishes a machine-readable list of Bingbot IP ranges at bing.com/toolbox/bingbot.json. Load that into your WAF or firewall as an allowlist and you're done, no DNS lookups needed on every request. There's also a manual Verify Bingbot tool where you can paste a single IP and get a yes/no answer.

Worth flagging, because it surprised me reading through Microsoft Q&A forum threads: being a verified Bingbot IP doesn't automatically mean the crawl behavior is polite. There are real reports of confirmed Bingbot ranges hammering sites with excessive or oddly repetitive query strings. Verification tells you who's knocking, not whether they're being reasonable about it.

Robots.txt, crawl rate, and the content-control tags that matter

Bingbot respects robots.txt, including Allow, Disallow, and wildcards. Unlike Googlebot, it also honors the old Crawl-delay directive, but Microsoft's own guidance recommends against using it. An aggressive Crawl-delay value can trigger a "crawl politeness" error inside Bing Webmaster Tools, telling you that you've accidentally throttled your own crawl rate. Use the Crawl Control feature inside Bing Webmaster Tools instead, which lets you set a custom pattern against your site's automatically calculated baseline rate.

The part that actually matters for Copilot specifically, though, is a pair of meta tags Microsoft introduced in September 2023, written by Fabrice Canel, Principal PM on Bing. Here's exactly what they do:

The syntax, straight from Microsoft's announcement: <meta name="robots" content="noarchive, nocache"> for both combined, or <meta name="robots" content="noarchive"> <meta name="bingbot" content="nocache"> to scope NOCACHE specifically to Bing while blocking others more broadly.

Here's the trap I'd flag loudest out of everything in this guide: a lot of sites have had NOARCHIVE sitting in their templates for years, often added as part of some generic CMS security hardening preset that nobody remembers enabling. It doesn't touch your classic Bing ranking at all, so nobody notices. But it silently opts you out of Copilot citations completely. If your content ranks fine in regular Bing but never shows up as a cited source in Copilot, check this tag before you check anything else.

Rendering: Bingbot's thin JavaScript budget

Bingbot's JavaScript rendering budget is described as limited, and noticeably thinner than Googlebot's. Grounding fetches lean on server-delivered HTML. If your FAQ answers, comparison tables, or key stats only render after client-side JavaScript executes, there's a real chance Copilot never sees them at all. This is reportedly a big part of why some pages rank fine in Google but never get cited by Copilot.

Quick sanity check: run curl -A "bingbot" [your-url] and read the raw response. If it's mostly empty divs and script tags rather than actual copy, that's your answer.

Another spot worth checking: bot-management defaults on Cloudflare, Akamai, and Fastly frequently score and throttle Bingbot more aggressively than they do Googlebot, sometimes serving it CAPTCHAs or outright 403s. Every challenged fetch is a page Copilot can't retrieve. Allowlist Bingbot by its verified IP ranges, not by user-agent string, and exempt it from rate limits and bot challenges.

IndexNow: skip the wait for the next crawl cycle

IndexNow is a protocol Microsoft co-authored with Yandex back in October 2021. Instead of waiting for Bingbot to naturally recrawl your page, you ping an API the instant you publish or update content, and it triggers a near-immediate recrawl. Bing, Yandex, Seznam, and Naver support it. Google does not, so don't expect it to do anything for your Google visibility.

IndexNow-pinged pages reportedly account for 17% of all clicked URLs on Bing, and outperform crawl-discovered URLs according to data cited alongside that figure. Setup is a POST request to api.indexnow.org/IndexNow with your host, a key, a key location, and a URL list, plus hosting a verification key file at your domain root. You can also configure it directly inside Bing Webmaster Tools under Configure My Site then URL Submission.

Not required for Copilot visibility, since organic crawling still works. But without it, your visibility timing depends purely on Bingbot's regular crawl schedule, which can genuinely lag days behind for lower-authority sites.

Tracking your Copilot citations: Bing's own AI Performance dashboard

Microsoft launched a dashboard called AI Performance inside Bing Webmaster Tools as a public preview on February 11, 2026, after quietly testing it since late January according to reporting from Search Engine Land. It's Microsoft's first-party answer to the question this guide is named after: what is Copilot reading, and is it citing you.

What it actually shows:

• Total citations, meaning how many times your content showed up as a source in AI-generated answers over a chosen period
• Cited pages and average cited pages, the unique URLs from your site referenced across AI experiences
• Grounding queries, sample key phrases the AI used when retrieving and citing your content
• Page-level citation activity, broken out by specific URL
• Visibility trends over time

The preview also includes Intents (classifying grounding queries by type: informational, commercial, navigational, comparison, and so on), Topics (grouping related queries into themes), and a Citation Share view showing your relative presence against others for a specific query.

A couple of honest caveats straight from Microsoft's own docs, worth repeating because they get glossed over: "AI Performance does not measure rankings, authority, performance, or importance. It simply shows which content was cited." And the dashboard only shows a sample of overall activity, not everything, so low-frequency citations may not surface at all. It also only covers content that's already index-eligible in classic Bing, same as regular search.

One other important scope note: the report covers "Microsoft Copilot, AI-generated summaries in Bing, and select partner integrations," not Copilot exclusively. So the numbers you see blend a few surfaces together.

Where third-party tracking tools fit in

Bing's own dashboard is useful but site-level only, and you have to be logged into Bing Webmaster Tools for your own property to see anything. If you want to track Copilot alongside ChatGPT, Gemini, Perplexity, and Google's AI Overviews in one place, or run competitive comparisons that Bing's tool simply doesn't offer, that's where a dedicated AI visibility platform like Promptwatch earns its keep.

Promptwatch tracks real UI responses (not just APIs) across ChatGPT, Gemini, Claude, Perplexity, Grok, Copilot, and Google's AI surfaces, and pairs citation tracking with AI crawler log analysis, so you can see both whether Bingbot is actually reaching your pages and whether Copilot is quoting them, in the same place you're watching every other engine. Most Copilot-specific coverage, including Bing's own dashboard, stops at monitoring. Tools with crawler-log and content-gap features let you act on the gap once you've found it.

It's worth noting there isn't yet a Promptwatch citation-share report specifically isolating Bing Copilot by domain, the kind that exists for ChatGPT, AI Overviews, and AI Mode. For now, Bing's own AI Performance report remains the most direct first-party source for domain-level Copilot citation data on your own site.

A practical checklist

If you want Copilot to be able to cite you, work through this in order:

1. Confirm Bingbot can actually reach your site. Check robots.txt isn't blocking it, and that your WAF or CDN isn't challenging it as a bot.
2. Audit every template for stray NOARCHIVE tags left over from old security presets. This one catches more sites than you'd think.
3. Run curl -A "bingbot" [url] on your most important pages and confirm the actual answer text is present in the raw HTML, not just rendered client-side.
4. Set up IndexNow if you publish or update content regularly, especially time-sensitive content.
5. Check your Bingbot IP ranges are allowlisted in your bot-management settings, not just permitted by user-agent.
6. Get access to Bing Webmaster Tools' AI Performance report and actually look at your grounding queries. That's the closest thing to seeing what Copilot is reading about you, directly from Microsoft.
7. Write answer-first, self-contained passages, a clear definition, a step list, a direct comparison, since that's the extractable format grounding retrieval favors.

If you're comparing this against how other AI engines crawl and cite, the GEO software directory at bestgeosoftware.com is a reasonable place to see how different platforms stack up on multi-engine coverage, Copilot included.