Key takeaways
- API calls and real UI collection often produce different answers. In Surfer's 2026 study, brand overlap between the two was only about 15% to 32%.
- The ChatGPT API named 13.8 brands per answer against 7.9 in the UI, so API-based reports tend to overstate who gets mentioned.
- Almost every published comparison comes from a vendor selling one method. Treat the numbers as directional, not settled.
- There is no Google AI Overviews API. Any tool claiming API coverage of AI Overviews is proxying through another surface.
- Ask every vendor three questions: where does the data come from, which countries and models are covered, and can you see the raw answer behind each score.
Why the collection method matters
An AI visibility tool answers a simple question: when someone asks ChatGPT or Perplexity about your category, does your brand show up? To answer it, the tool has to send prompts somewhere and record what comes back.
There are two ways to do that. One is to call the model's developer API. The other is to drive the real consumer interface, the thing a buyer actually types into, and capture the answer, the cited sources, and any shopping cards. Vendors usually call this UI scraping or real-UI collection.
The difference sounds technical, but it changes what you report to your CMO. An API call returns the raw model. It skips the interface's retrieval layer, personalization, safety filtering, and answer shortening. So a dashboard built on API calls can be confidently wrong about what real users see.
What the evidence says
The Surfer study
The most detailed comparison I found is Surfer's study of scraped answers versus API results, last updated October 4, 2026. They ran 1,000 prompts across ChatGPT, Perplexity, Gemini, AI Mode, and AI Overviews. Each prompt was collected once by scraping the UI and once through an API, producing 13,779 answers across 14 model-and-method combinations.
The headline numbers:
- Brand overlap between API and scraped answers was 15.5% to 23.8% on raw names, and 21.3% to 31.6% after name canonicalization. Gemini was the closest, at roughly 3 in 10 brands.
- The ChatGPT API named 13.8 brands per answer. The ChatGPT UI named 7.9.
- The ChatGPT API wrote 2.05 times as many words as the UI. Gemini was 1.27x and AI Mode 1.06x.
- Gemini and AI Mode were close to parity on brand count (1.10x and 1.06x), so the gap is not uniform across engines.
- Page-level source overlap was very low: 19.7% for Perplexity and only 4.8% to 6.2% for ChatGPT.
Surfer's conclusion is that API-based reports overreport brands, and the sources they show may not be the ones real users see. One caveat they state themselves: the AI Overviews "API" in the test was borrowed from AI Mode, because no AI Overviews API exists.
Other vendors say similar things
ZeroRank, which sells scraping-based tracking, reported API answers averaging 406 words against 743 for scraped answers, with only 24% of brands overlapping. seoClarity argues that UI collection mirrors what users see while the API gives a simplified, developer-facing version. A GTM Only review of AI visibility tools makes the same warning in one line: API-based monitoring often differs from real UI outputs.
Notice the pattern. Every one of these sources sells scraping or has a stake in the answer. That does not make them wrong, but it does mean you should not quote a single percentage as gospel.
The counterargument
Conductor takes the opposite side, and it deserves a fair hearing. Conductor is an API-based vendor, so it has its own conflict of interest, but its criticisms of scraping are real:
- Fragility. UI changes break scrapers, and geo-blocking, rate limits, and CAPTCHAs get in the way.
- Single-configuration bias. Results reflect one device or UI setup, not the spread of real users.
- Terms-of-service and compliance risk.
- High maintenance overhead and occasional data-quality issues.
I think these are fair points about operations, but they are arguments about cost and reliability, not about accuracy. If the question is "what does a real user see," the UI wins on definition. If the question is "what can a vendor deliver cheaply and stably," the API wins.
Why the real interface keeps moving
There is a second argument for UI collection that gets less attention: the live product changes constantly, and in steps.
Promptwatch Data's ChatGPT query fanout report tracked average fanouts per response falling from 2.15 in early December 2025 to about 1.84 by early March 2026, and average fanout query length dropping from about 117 characters to the high 80s. Data returned in April at 1.0 fanouts per response with queries around 53 characters. Separately, Promptwatch Data found that ChatGPT started using the site: operator at scale on August 8, 2026, taking it from about 0.4% to about 17% of all fanout queries overnight.
Average citation counts moved too. Citations per response dropped across GPT-5.3, GPT-5.4, and GPT-5-Mini around the March 4, 2026 rollout. And the number of sources differs by engine to begin with: Promptwatch's average sources per response data shows ChatGPT citing about 5 sources per web-search answer, while AI Overviews and Perplexity sit around 10.
A pinned API call will not reproduce any of this. If the interface decides to run twice as many searches or cite half as many pages, only a tracker watching the live product will notice.
Which tools use which method
Here is the honest bit: public information on this is uneven. Many vendors say "real data" without defining it, and a Reddit thread on r/LLMO_SaaS (anecdotal) claims most top tools run on scraping infrastructure such as Bright Data, Oxylabs, or DataForSEO. I could not verify every vendor's method independently, so the table below separates what vendors or reviewers state from what I could not confirm. Check each vendor's documentation before buying.
| Tool | Collection method (as stated publicly) | Source of claim | Worth checking |
|---|---|---|---|
| Promptwatch | Real user interfaces of ChatGPT, Gemini, Perplexity, AI Overviews, AI Mode, Copilot and more, plus API models tracked alongside | Vendor | Country and language coverage on your plan |
| Peec AI | UI scraping that simulates real user interactions | Position Digital review | Which engines are on your tier |
| AIclicks | Real user interfaces rather than APIs | Profound's agency roundup (a competitor) | Independent confirmation |
| Superlines | Real AI interfaces, not APIs | Catalog description | Engine list and refresh rate |
| Conductor | Official APIs plus web-search tool integration | Conductor and Position Digital | How closely API output matches UI for your prompts |
| Profound | Real-time front-end visibility snapshots plus crawler logs | Third-party ranking | Method per engine |
| Otterly.AI, AthenaHQ, Scrunch AI, Semrush, Ahrefs Brand Radar | Not verified in my research | n/a | Ask the vendor directly |
Tools that say they use real UI data
Promptwatch collects responses, citations, and click data from the actual user interfaces of ChatGPT, Google AI Overviews, Google AI Mode, Microsoft Copilot, Gemini, Perplexity, and Amazon Alexa, in each country and language. It tracks API models such as Claude, DeepSeek, Mistral, and Grok alongside them, which is a useful split: the UI shows what users see, the API shows what the model believes from training data. That split is the correct way to think about the whole topic. Both are legitimate data, as long as they are labeled.
Peec AI is described by Position Digital as collecting responses via UI scraping, simulating real user interactions instead of using API data.
Superlines positions itself as GEO tracking from real AI interfaces, not APIs.
Superlines
AIclicks is named by Profound as a tool that queries through real interfaces.
Tools that lean on APIs
Conductor is the clearest public case for the API route, combining API access with web-search tool integration. If your needs are stability, compliance comfort, and broad enterprise integration, that argument has weight.
Tools where the method is less clear
Profound is positioned around real-time front-end snapshots, log-level crawler data, and a large prompt-volume dataset. Its own agency roundup also says the stronger tools use real interfaces.
Otterly.AI, AthenaHQ, and Scrunch AI show up on most 2026 shortlists. I could not confirm their collection methods from public sources, so ask.
SEO suites with AI add-ons such as Semrush and Ahrefs Brand Radar are worth the same question.
If you want to browse more options, the GEO software directory at bestgeosoftware.com lists tools in this category.
What the roundups say
The vendor-published roundups are worth reading, with the usual skepticism. KIME's ranking of 11 tools focuses on engine coverage and sentiment, and ends with the sensible advice to match a tool to your stage and to how many engines your plan really includes.
Profound's agency roundup grades tools partly on where the data comes from, which is the right criterion even if the author sells a competing product.
Evertune's list is useful mainly for its feature matrix separating monitoring, optimization, and AI advertising.
Five traps in UI vs API claims
- Every study has an agenda. Surfer, ZeroRank, seoClarity, Conductor, and Profound all sell one of the two methods. Look for the direction of agreement, not the exact percentages.
- "API coverage" of AI Overviews is not real. No such API exists, so a tool reporting it is using AI Mode or another proxy.
- Engines differ. Gemini and AI Mode came close to parity in Surfer's data, while ChatGPT and Perplexity diverged a lot. A tool may be fine on one engine and misleading on another.
- UI scraping is not automatically accurate. A single device, one logged-out session, or one location is a narrow sample. Ask about geography, repetition, and how answers are sampled.
- Citations are the weakest link. Source overlap of 5% to 20% between API and UI means any API-derived citation report should be treated with suspicion.
How to test a vendor yourself
You do not have to take anyone's word for it. A 30-minute test is enough to see which side a tool falls on.
- Pick five prompts your buyers really ask. Run them in the live ChatGPT and Perplexity apps, logged out, and screenshot the answers and sources.
- Load the same prompts into the tool's trial. Compare brand lists, order, and cited URLs against your screenshots.
- Count the brands. If the tool consistently shows many more than the app does, that points to API output, based on the 13.8 vs 7.9 ChatGPT gap above.
- Check the sources. Do the cited URLs match what you saw? Page-level overlap should be much higher than the 5% to 20% seen in API-versus-UI comparisons.
- Ask for the raw answer text behind any score. If a vendor will not show it, that is an answer in itself.
- Confirm country and language settings match your market, since answers vary by location.
A free scan is a decent starting point. Promptwatch has a no-login Free AI Visibility Report, and its GEO Inspector Chrome extension shows ChatGPT's fan-outs and fetched versus cited sources, which makes a handy reference when you compare a tool's output with what the app actually did.
When an API is the right call
I do not want to oversell the UI. APIs are the better fit in some cases:
- You need to know what a model believes without live search, for example brand perception from training data.
- You are building internal tooling and need predictable, ToS-clean access.
- You want cheap, high-volume spot checks where directional accuracy is enough.
The best tools are transparent about which is which. A dashboard that mixes UI and API results in one score without labeling them is the thing to avoid.
How to decide
| If you care most about | Lean toward | Why |
|---|---|---|
| Reporting what buyers see in ChatGPT, AI Overviews, and AI Mode | Real UI collection | API output diverges on brands, wording, and sources |
| Citation and source analysis | Real UI collection | Source overlap with API output is very low |
| Stable, compliance-friendly enterprise integration | API-based, with a clear caveat | Fewer breakages and ToS questions |
| Understanding model knowledge from training data | API models | No live retrieval needed |
| Closing the loop on fixes | A platform that goes past monitoring | Accuracy of the data matters most when you act on it |
For teams that want accurate collection and the follow-through, Promptwatch pairs real-UI tracking with IP-verified AI crawler logs, visitor analytics, and an Action Board and Content Agent that turn gaps into published pages. Plans start with a free Explore tier and Essential at $95/month, so confirm current pricing at promptwatch.com/pricing. Reviewing several tools side by side in the AI rank tracking directory at ai-rank-tools.com is also worthwhile before you commit.
The bottom line
The studies are vendor-funded, but they point the same way: API answers and real UI answers are different products, and the gap is largest for ChatGPT brand counts and for cited sources. If your report claims to show what customers see, collect it from the interface. If a tool will not tell you how it collects data, run the six-step test above and let the screenshots decide.