How to Test Whether an AI Visibility Tool Uses Real UI Data or APIs (2026 Method)

Vendors rarely say how they collect AI answers. This 2026 guide gives you a hands-on test protocol to tell real UI data from API calls: citations, fanouts, shopping modules, answer length and evidence.

Key takeaways

  • AI visibility tools collect data one of two ways: calling model APIs, or running prompts through the real interfaces people use. The two can return very different answers.
  • Surfer's study of 13,779 answers found roughly 75% of the brand lists did not overlap between scraped UI and API results. Surfer sells a scraping-based tool, so read that with a grain of salt, but the direction matches what other sources describe.
  • You can test a vendor yourself in about an hour: compare citations, source counts, fanouts, shopping modules, answer length, and ask for raw evidence.
  • A vendor that cannot say, in writing, which method it uses per AI surface has given you its answer.
  • Method is not the only variable. Sampling frequency and how a tool handles non-deterministic answers matter just as much.

Why this question matters

There is no Search Console for ChatGPT. Nobody publishes rankings for AI answers. So every AI visibility tool has to generate its own data by running prompts and recording what comes back. How it runs those prompts is the biggest methodological difference between products, and it is the one vendors talk about least.

If a tool queries an API and your buyers use the consumer app, you are measuring a cousin of the product, not the product. The brand lists, the sources, even the length of the answers can differ. You might spend a quarter fixing a visibility problem that exists only in the dashboard.

This guide gives you a practical method to find out which approach a tool uses, without taking the sales deck at its word. I could not find a vendor-neutral test protocol published anywhere, so what follows is built from the observable differences between the two methods.

The two collection methods, briefly

API-based collection

The tool sends your tracked prompts to a developer endpoint, such as the OpenAI API or the Gemini API, and parses the response. It is stable, fast, cheap, and officially supported. High prompt volumes are easy.

The catch, as metehan.ai's breakdown of collection methods puts it, is that the API is not the product consumers use:

  • The model behind a consumer chat interface is often a different build than the API default, with different system instructions and tuning.
  • A consumer product decides on its own when to search the web. An API call without a search tool returns parametric memory only, so citation data can be missing or synthetic.
  • Follow-up suggestions, shopping modules, source carousels and formatting exist only in the real interface.

Real UI execution

The tool opens the actual ChatGPT web app, the actual Google results page with AI Overviews, or the actual Perplexity thread, submits the prompt, and records the rendered answer with its sources. You get the production model, live retrieval, real citations and real interface modules.

It is slower, more expensive and harder to run at scale. That is why some vendors avoid it. When a tool offers suspiciously high prompt volumes at a suspiciously low price, this is the corner that was probably cut.

Surfer's study comparing scraped UI answers with API answers across five AI products

DimensionAPI-based collectionReal UI execution
Model queriedAPI build, may differ from the consumer productThe production model users get
Web searchOptional, often off or simulatedNative, decided by the product
CitationsMay be absent or approximatedCaptured as rendered
Interface modulesNot visibleVisible and measurable
Cost and speedCheap, fast, scales easilyExpensive, slower, harder to scale
Fidelity to what buyers seeApproximateHigh

What the evidence says about the gap

Surfer ran 1,000 prompts across five AI products, once by scraping the interface and once via an API, and collected 13,779 answers across 14 model-and-method combinations. About 75% of the brand lists did not overlap, and API answers could run twice as long as what a buyer reads. Two honest caveats from the study itself: Surfer sells a scraping tool, and each prompt had one sample, so some of the disagreement is plain non-determinism. Also, the AI Overviews API method was borrowed from AI Mode, because no AI Overviews model is sold.

Peec's documentation, for the record, says it uses UI scraping rather than API collection. That is the kind of plain statement you want from every vendor.

The 7-step test protocol

You do not need a data science team. You need a spreadsheet, two or three prompts you care about, and a free afternoon. Run these steps during a trial or demo.

Step 1: Ask the written-answer question

Before testing anything, ask each vendor four things in writing:

  1. For each surface (ChatGPT, Gemini, Perplexity, AI Overviews, AI Mode), do you collect through the user interface or an API?
  2. Which model version do you query?
  3. How often do you run each prompt?
  4. How do you handle non-deterministic answers?

An answer like "we use a proprietary approach" is evasion. Also watch for mislabeling: a Gemini API observation should not be presented as an AI Overview just because both involve Google technology.

Step 2: Run the manual baseline

Pick three tracked prompts. Open the real product in a clean browser session, logged out where possible, and run each prompt yourself three times. Record the brands mentioned, their order, the sources cited and the answer length.

Three runs matter because answers vary. You are looking for the stable core, not a single sample.

Step 3: Compare the brand lists

Now look at what the tool reports for the same prompts. The lists will not match exactly, since non-determinism guarantees that. But the core brands should recur. If your manual runs consistently show five names and the tool shows a mostly different set, that is the signature of an API call to a different model or without search.

Step 4: Check citations and source counts

Count the sources. Promptwatch's average sources per response data is a useful reference point: ChatGPT cited about 5 sources per answer when web search was triggered, Google AI Overviews about 10, and Perplexity almost exactly 10 every day. Those figures date from 2025 (page dated September 23, 2025), so verify they still hold before leaning on them.

The test: does the tool's average citations per response land in the same neighborhood? An API answer with search disabled returns few or no real citations. Suspiciously uniform or invented-looking sources are another warning.

Step 5: Look for query fanouts

When ChatGPT searches the web, it runs one or more sub-queries behind the scenes. Promptwatch's ChatGPT query fanout data shows the average at about 2.15 web searches per response in early December and about 1.84 by early March, with average fanout query length falling from roughly 117 characters to the high 80s. A single prompt can trigger 3 to 8 or more searches.

An API call with no search tool produces no fanout at all. So ask the vendor to show you fanout or search-query data per response. If a tool shows fanouts, they should move in step with these public shifts. If a tool shows none for ChatGPT, or shows static numbers that never change, be skeptical. Our own free-tool pointer here: Promptwatch's ChatGPT query fan-out generator and the query fan-out academy guide explain what you are looking at.

Step 6: Test interface-only features

Some things exist only in the rendered product: shopping carousels, product cards, source panels, maps, follow-up suggestions. Pick a commercial prompt ("best running shoes under 150 dollars") and see whether the tool reports shopping results. If the real interface shows a product carousel and the tool has no concept of it, the data is not coming from that interface.

Step 7: Demand evidence

Ask to see a screenshot, rendered HTML or a raw response for a specific prompt and run. Real UI collection can produce this easily. API collection cannot produce a screenshot of something that never rendered. Vendors who store only parsed text are not necessarily lying, but ask why.

What the results look like

Use a simple scorecard. Score each signal from 0 to 2 per surface.

SignalAPI-like resultUI-like result
Brand overlap with your manual runsMostly different namesCore brands recur
CitationsFew, missing or genericCount and mix resemble the live product
Fanout queriesNone, or frozenPresent and moving over time
Shopping and interface modulesNot capturedCaptured
Answer lengthNoticeably longer or shorter than what you seeComparable
Raw evidenceParsed text onlyScreenshot or rendered capture on request

Two or three API-like signals on one surface is enough to ask hard questions. Remember that hybrid setups are common: a tool might scrape ChatGPT and use an API for Claude, which has no search interface comparable to the others. That is not automatically a problem if the vendor says so clearly.

Sampling matters as much as method

A tool that runs one prompt once a week is running a poll, not a measurement. The same prompt asked twice does not return the same answer, so a single sample tells you little. Ask how often prompts run and how many samples feed each number. Tools that run prompts daily, and on higher tiers every few hours, give steadier visibility numbers.

metehan.ai guide explaining API versus UI scraping, sampling and methodology transparency

How some tools describe themselves

I have not verified the collection method for every tool below, and you should not take my word or theirs. Treat this as a starting list of what to test, based on what vendors say publicly.

Promptwatch says it monitors the actual user interfaces of ChatGPT, Google AI Overviews, Google AI Mode, Microsoft Copilot, Gemini, Perplexity, Amazon Alexa, Meta AI and the AI coding agents, in each country and language, and tracks API models such as Claude, DeepSeek and Mistral alongside to show what models believe from training data. Its explanation of how it collects data is public, which is the kind of transparency Step 1 asks for. It also adds IP-verified AI crawler logs and visitor analytics, which let you check the tool's claims against your own server data.

Favicon of Promptwatch

Promptwatch

Track and improve your AI search visibility
View more
Screenshot of Promptwatch website

Peec AI's documentation describes UI scraping rather than API collection.

Favicon of Peec AI

Peec AI

AI visibility tracking with smart suggestions
View more
Screenshot of Peec AI website

Superlines markets itself as GEO tracking from real AI interfaces, not APIs.

Superlines

GEO tracking from real AI interfaces, not APIs
View more

Otterly.AI, Profound and Scrunch are frequent shortlist names. Put the same seven steps to each.

Favicon of Otterly.AI

Otterly.AI

Affordable AI brand visibility monitoring
View more
Screenshot of Otterly.AI website
Favicon of Profound

Profound

Enterprise AI search visibility and analytics
View more
Screenshot of Profound website
Favicon of Scrunch AI

Scrunch AI

AI search monitoring for brands and agencies
View more
Screenshot of Scrunch AI website

SEO suites that added AI tracking, such as Semrush and Ahrefs Brand Radar, deserve the same questions, particularly about fixed prompt sets and which surfaces are observed versus approximated.

Favicon of Semrush AI Visibility Toolkit

Semrush AI Visibility Toolkit

SEO and AI visibility in one platform
View more
Favicon of Ahrefs Brand Radar

Ahrefs Brand Radar

Track your brand across AI search engines
View more
Screenshot of Ahrefs Brand Radar website

To browse more options, the directories at bestgeosoftware.com and ai-rank-tools.com list many more trackers.

Cross-check against your own data

The best sanity check is not a vendor demo. It is your own site. If a tool says ChatGPT cites your pricing page often, check your server logs for OAI-SearchBot and ChatGPT-User hits on that page, and check analytics for referral traffic from AI engines. Promptwatch's AI crawler logs are one way to do this; a CDN log export works too. Note that ChatGPT stopped sending a referrer on October 8, 2026, so tag-based attribution (utm_source=chatgpt.com) matters now. The attributing AI traffic guide covers it.

Questions to ask on every vendor call

  • Per surface, UI or API? Can you put that in the contract or documentation?
  • Which model version, and how do you update when the product changes?
  • Do you capture citations as rendered, and can I see a screenshot or raw capture?
  • How many samples per prompt per day, and how do you aggregate them?
  • Do you track fanouts, shopping modules and sponsored placements?
  • Are prompt volumes and price consistent with real UI execution costs?

Common mistakes

  • Treating one run as truth. Non-determinism means you need repeated samples on both sides of the comparison.
  • Expecting identical answers. The goal is overlapping cores and similar citation behavior, not a perfect match.
  • Penalizing hybrid tools unfairly. API collection for models with no consumer search surface can be reasonable, if disclosed.
  • Trusting a demo account. Run your own prompts, in your own market and language.
  • Ignoring location. Answers shift by country and city, so compare like for like.

Bottom line

Ask the four questions in writing, run three prompts by hand, and check brand overlap, source counts, fanouts, shopping modules and raw evidence. If a vendor fails more than one of those on a surface it claims to cover, treat that surface as an approximation and price it accordingly. If you want help auditing AI visibility vendors for your brand, 1001 SEO Media runs this kind of evaluation as part of its GEO work.

Share:

AI Search Visibility Tools

© 2026 AI Search Visibility Tools · The best AI search visibility tools compared · RSS

AI Search Visibility Tools is an affiliate review site. When you click links to vendors or buy through links on our site, we may earn an affiliate commission at no extra cost to you.

AI Search Visibility Tools is a review website based on user reviews on Reddit and G2, and on publicly available information. We keep everything as up to date as possible, but pricing and features can change. Always confirm the details with the vendor before purchasing.