Why your AI Mode rankings look different in every tool: a data accuracy investigation for 2026

Key takeaways

• Google says it outright: AI Mode and AI Overviews "may use different models and techniques, so the set of responses and links they show will vary" -- there is no single ground truth to check a tool against.
• Platform-side changes, not your content, explain most sudden swings. ChatGPT's average citations per response dropped about 27% within a day of the GPT-5.3 rollout on March 4, 2026, and never recovered.
• Sample size matters more than most dashboards let on. SparkToro and Gumshoe.ai found less than a 1-in-100 chance that ChatGPT or Google's AI gives the same brand list twice in a row for the same prompt.
• Engines disagree on how many sources they even cite: Perplexity averages close to 10 a day with barely any wobble, while Microsoft Copilot has swung from under 2 to nearly 17 sources per response in a matter of weeks.
• The fix isn't finding the "accurate" tool. It's running a bigger, repeated sample, understanding each tool's methodology, and tracking trends instead of single snapshots.

If you've ever run the same brand through two or three AI visibility tools in the same week and gotten wildly different "rankings," you're not imagining it. I've seen a brand show up as dominant in one dashboard and nearly invisible in another, checked on the same day, for what was supposedly the same thing: presence in Google's AI Mode.

The instinct is to assume one tool is wrong. Usually that's not really the problem. The problem is that "ranking" in AI Mode isn't a stable thing to measure in the first place, and most tools are quietly making different choices about what to measure, how often, and from where.

AI Mode doesn't have a ranking to agree on

Classic SEO rank tracking works because there's a fixed, orderable list. Position 3 is position 3, whoever checks it. AI Mode breaks that assumption completely. Google synthesizes an answer from a pool of sources, and which sources make the cut depends on the query's wording, your location, your device, whether you're logged in, and apparently which side of an internal test bucket you landed in.

Google's own developer documentation states this directly: AI Mode and AI Overviews "may use different models and techniques, so the set of responses and links they show will vary." Both surfaces "may use a query fan-out technique," but Google doesn't promise the fan-out behaves the same way across the two. That's a vendor telling you, in writing, that there is no stable answer to check your tracker against.

It gets stranger. Google's Personal Intelligence feature in AI Mode pulls context from a logged-in user's Gmail and Photos, which means two people typing the identical query can get genuinely different responses. There is no external tool, no matter how well built, that can produce one deterministic "ranking" for a feature designed to personalize its output.

And the rollout timing itself causes confusion. AI Overviews launched in Germany on March 26, 2025. AI Mode didn't arrive there until October 8, 2025, over six months later. Any tool claiming "AI Mode coverage" in a market during that gap was, whether it admitted it or not, reporting Overviews data under an AI Mode label.

The consistency problem is worse than most people realize

Even if you ignore the AI Mode vs. AI Overviews confusion and just ask "will the same prompt return the same answer twice," the honest answer is: rarely.

SparkToro and Gumshoe.ai ran 12 prompts through ChatGPT, Claude, and Google's AI a combined 2,961 times with 600 volunteers. Their finding: there's less than a 1-in-100 chance that ChatGPT or Google's AI returns the same list of brands in any two runs of the same prompt. Ordering is worse, roughly 1-in-1,000 runs before you see the same list in the same order twice. Even the length of the list changes, sometimes 2-3 recommendations, sometimes 10 or more, for an identical question.

That's not a tool-quality problem. That's the underlying behavior of the systems being measured. A separate analysis from cloro.dev backs this up with a cleaner experiment: running the same 20 prompts through the same model (Claude Sonnet) minutes apart produced only about 40% overlap in cited domains. Sixty percent of the sources differed between two back-to-back runs of identical prompts.

So when Tool A says you're cited and Tool B says you're not, on the same day, for the same prompt, it's entirely possible both tools are telling the truth about what they saw. They just happened to catch the model on different rolls of the dice.

Sample size: the part almost nobody talks about

This is where most AI visibility reporting falls apart quietly. A 25% visibility rate measured over 72 answers carries a margin of error of roughly plus or minus 5 points at 95% confidence, according to cloro.dev's analysis. To get solid statistical power, you need somewhere around 294 answers per comparison period. A tool checking 50 prompts across 6 engines daily generates about 2,100 answers a week, which makes week-over-week comparisons meaningful but day-over-day comparisons close to noise.

Obsero.ai found something similar: tracking a single prompt daily across 3 models gives you about 21 readings a week, with a margin of error around plus or minus 16 percentage points at a realistic visibility rate. Their fix is to track clusters of roughly 20 related prompts as a "topic" instead of single prompts, which tightens the margin of error to about plus or minus 3.7 points weekly.

Graphite.io's research adds a nuance worth sitting with: the number of responses you need isn't fixed, it depends on your actual visibility rate. At 50% true visibility, getting a reasonably tight confidence interval needs 167 responses. At close to 100% visibility, you only need 22. With just 50 responses, you can reliably tell 50% visibility apart from 30%, but not from 40%. That means a lot of the small week-over-week swings showing up in dashboards aren't real movement. They're statistical noise dressed up as insight.

Evertune.ai puts a number on what "enough data" looks like: roughly 100 samples per prompt per AI model for real statistical confidence. Most tools, especially at entry-level pricing tiers, aren't running anywhere close to that per prompt.

Different engines behave like different products

This is the part that trips up anyone comparing tools without checking which engine each one is actually hitting. Promptwatch's data shows how differently the major engines behave when it comes to how many sources they cite per response:

Promptwatch's average sources per response data shows Copilot swinging from under 2 to nearly 17 sources per response within a few weeks, which its research calls "the wildcard." If a sudden change in your Copilot citations shows up, the practical read is that Microsoft changed something on its end, not that your content got worse.

The ChatGPT drop is its own cautionary tale. Promptwatch's data on the citation drop after GPT-5.3 shows average citations per search-enabled response fell from about 6.4 to 4.7-4.9 within a day of the March 4, 2026 rollout, across GPT-5.3, GPT-5.4, and GPT-5-Mini simultaneously. Not a gradual drift. A platform-side switch flipped and every brand being tracked looked less visible overnight, through no fault of their own. Any tool running a single snapshot audit right before or after that date would have reported numbers that had nothing to do with content quality.

Domain counts and the illusion of "ranking #1"

Another reason tools disagree: how many domains actually share a single AI answer. Promptwatch's domain-level data from late September 2026 found that 47.6% of all ChatGPT responses with citations name 10 or more unique domains, and 66.1% name at least 8. Only 6% cite three or fewer, and just 1% cite a single domain.

That breaks down a lot by prompt type too. Organic prompts (think "best project management software") cite 10+ domains in 52.5% of responses. Brand-specific prompts ("is [Brand] good") cite 10+ domains in only 21.3% of cases, with 40.7% citing 5 or fewer. So a tool sampling mostly brand-name prompts will report a much narrower, cleaner-looking citation share than one sampling broad category prompts, not because either tool is broken, but because the denominator is completely different.

This also kills the idea of "ranking #1" in any traditional sense. When 10+ sources routinely share one answer, being cited isn't a position, it's closer to a lottery ticket you can improve your odds on but never fully control.

Domain authority doesn't predict citations the way SEO people expect

If you're used to thinking "bigger, more authoritative sites win," AI Mode and ChatGPT citation behavior will frustrate you. Promptwatch's August 2026 domain-rank data shows DR 46-60 and DR 61-75 sites together account for roughly 46% of ChatGPT citations, while DR 91-100 sites, the biggest and most "authoritative" domains by classic metrics, fell from about 7% share in week one of August to around 3% by August 14, and stayed there. By the end of the month, DR 91-100 domains were cited less often than DR 0-15 domains (4.4% vs 6.3%).

That drop happened on the same day Reddit's ChatGPT citation share also collapsed, from roughly 4% to 0.5%, which Promptwatch's report on Reddit citations dropping in ChatGPT ties to what looks like a related platform-side filtering change. If your tracker only measures citation counts without flagging these kinds of structural shifts, you'll misread a platform event as a content problem.

Social platforms show up completely differently depending on the engine

A tool that only monitors ChatGPT will systematically undercount how much AI search actually leans on social content, because social platforms behave nothing alike across engines. Promptwatch's social media citation data by AI model shows ChatGPT citing social platforms in just 0.46% of responses, down sharply from 5.8% in an earlier snapshot, with LinkedIn now ChatGPT's top social source at a mere 0.3% share. Google AI Overviews tells a completely different story: social platforms make up 12.2% of citations there, about 26 times ChatGPT's rate, led by YouTube at 4.28%. Perplexity sits in between at 2.7%.

If your brand lives on Reddit or YouTube and you're only checking a ChatGPT-only tracker, you'll conclude social content doesn't matter for AI visibility. Check the same brand against AI Overviews and the picture flips entirely.

Query behavior itself keeps changing

Even the mechanics of how these engines search the web shift underneath tools without warning. Promptwatch's query fan-out research shows ChatGPT's fan-out volume easing from about 2.15 queries per response in early December to 1.0 by April, with average query length shrinking from around 117 characters to about 53, meaning queries now behave more like terse keyword searches than full sentences.

Then on August 8, 2026, ChatGPT started using the site: operator at scale, jumping from about 0.4% to roughly 17% of all fan-out queries overnight, with searches per response nearly doubling the same day. A tool that ran the same prompt set on August 7 and August 9 would show two completely different retrieval patterns for reasons that have nothing to do with your site.

Crawler-based tools vs. UI-scraping tools disagree too

Tools that read AI crawler logs and tools that scrape the visible chat interface aren't even looking at the same evidence. Promptwatch's crawler data found OpenAI's crawlers made up 79.8% of verified AI crawler requests in the first week of September 2026, down from 94.8% just twelve weeks earlier. That's a major mix shift in under three months. A log-based tool and a UI-scraping tool checked at different points in that window will naturally disagree about which engines are even active on your site.

It's also worth noting that scraping AI Mode's conversational interface reliably is genuinely hard. Vendors like Apify and Oxylabs sell dedicated "Google AI Mode scraper" products specifically because the interface requires session handling, proxy rotation, and headless browser automation to extract consistently. Any tool built on fragile scraping is going to produce noisier data than one reading verified crawler logs or stable API access.

There's no clean way to verify a tool against ground truth

Here's the frustrating part: even if you suspect a tool's numbers are off, Google Search Console won't settle the argument. AI Mode and AI Overview impressions are both folded into the standard "Web" search type in GSC's Performance report, with no separate bucket for AI Mode specifically. You can't pull a clean, isolated AI Mode number to check a vendor's claim against.

This is also why pricing and plan comparisons across AI visibility vendors move faster than most buyer's guides can keep up with. Scrunch AI was acquired by Sitecore for a reported $225M in June 2026. Profound pulled its self-serve plans entirely in mid-September 2026, shifting to trial-then-custom-Enterprise only. Peec AI repriced from USD to geo-priced EUR tiers in August 2026. Any comparison table older than a couple of months should be treated with suspicion on pricing specifically, even when the tracking methodology hasn't changed.

So how do you actually validate a tool?

A few concrete checks, drawn from how practitioners actually catch mislabeled or shallow tools:

• Ask the vendor to show you separate, timestamped data for AI Mode and AI Overviews specifically. If they use the terms interchangeably, or only have one dataset, that's a red flag.
• Check whether exports include full answer text and citation URLs, not just aggregated "competitor.com cited 5 times" summaries. Raw output lets you spot-check.
• Run the same 15-20 prompts yourself across two or three tools, in the same countries, and compare overlap. If one tool shows 15/20 prompts cited and another shows 8/20, the lower one is probably missing real AI Mode output, not catching something the other tool invented.
• Look at trend lines over weeks, not single-day snapshots. Given the GPT-5.3 drop and the site: operator rollout, anyone reading day-to-day swings as meaningful is going to chase ghosts.
• Confirm whether the tool tracks a large enough, varied enough prompt set (organic, brand, and comparison prompts) rather than a narrow slice that happens to flatter or deflate your numbers.

Where this leaves you

None of this means AI visibility tracking is useless, it means single numbers from single tools deserve skepticism, and trend data beats snapshot data every time. The tools worth paying for are the ones that are upfront about methodology: which engines, how many prompts, how often refreshed, and what counts as a citation.

Promptwatch is built around that transparency problem specifically. It monitors the actual user interfaces of ChatGPT, Gemini, Claude, Perplexity, AI Overviews, and AI Mode rather than relying only on APIs that can diverge from what users actually see, and its dataset of 26 billion-plus analyzed citations, prompts, and responses is large enough to separate real trend shifts (like the GPT-5.3 citation drop or the Reddit collapse) from day-to-day noise. It also goes past pure monitoring with crawler logs, content gap analysis, and automated content generation, so once you know where you're missing citations, there's a path to actually closing the gap rather than just watching the dashboard.

If you're evaluating a broader set of AI visibility and rank tracking tools, it's worth browsing a directory like the one at ai-rank-tools.com or the GEO software comparisons at bestgeosoftware.com to see how methodology claims hold up across vendors before you commit a budget line to any single one.