Key takeaways
- Many AI visibility tools send prompts to a model API and read the reply. The consumer app is a different product with different defaults, so the numbers can describe something your buyers never see.
- Windsor.ai found ChatGPT's API named 5.3 tools per answer against 1.1 in the app, and 7% of API answers named no tool compared with 57% in the app.
- Surfer reported that roughly 75% of the brand list did not overlap between scraped UI answers and API answers across 13,779 answers.
- API calls also cannot return interface-only elements like ads, shopping carousels, and citation ordering.
- Ask every vendor how they collect data, whether you can see the raw answer behind a number, and whether the method differs per engine.
The short version of the problem
Most AI visibility trackers work the same way. They send a prompt to the model's API, read the response, and log which brands and URLs show up. It is the only practical way to run thousands of prompts a day, so I understand why vendors do it.
The trouble is that an API call returns the raw model. A person typing into ChatGPT, Gemini, or Google AI Overviews gets a product wrapped around that model: a system prompt, retrieval behavior, formatting, citation display, and sometimes ads or shopping cards. An API-only tracker describes the API. Whether that matches what your customers see is a separate question, and the evidence says often it does not.
What the 2026 tests found
Two vendors published experiments this autumn. Neither is neutral (Surfer sells a scraping-based tracker, and Windsor.ai is writing about its own visibility), and I have not independently verified their figures. Still, the direction is consistent.
Windsor.ai: three rounds of ChatGPT testing
Windsor.ai compared the ChatGPT API (gpt-5-mini with web search on) against the real ChatGPT app. Their note that the app has no web-search toggle, while the API does, is an important detail. The app decides for itself when to search.
The headline numbers from their open-prompt round:
- Tools named per answer: 5.3 on the API against 1.1 in the app.
- Answers naming no tool at all: 7% of API answers against 57% of app answers.
- In round 3, API answers to the same prompt agreed with each other only 29% of the time.
That last figure matters beyond the API-versus-app gap. Even a single method is noisy, so one query per prompt tells you very little. Sampling repeatedly is not optional.
Surfer: 13,779 answers across five AI products
Surfer ran 1,000 prompts across five AI products, once by scraping the interface and once through an API, collecting 13,779 answers over 14 model-and-method combinations. Their claim is that overlap is about 30% at best, and that roughly 75% of the brand list did not overlap in the comparison they highlight.
One caveat they state themselves: Google AI Overviews has no sold API, so they borrowed the AI Mode method for it. That is a useful reminder that for some engines an API-based approach is a proxy by necessity.
What an API call cannot see
Beyond different answers, there are whole categories of things the API never returns.
Retrieval and fan-out behavior
When ChatGPT searches the web, one prompt can trigger several separate searches. Promptwatch's query fanout data shows a single prompt triggering 3 to 8+ searches, with the average falling from 2.15 per response in early December to about 1.84 by early March. Query length shifted too. Your content has to match one of the generated fanout queries, not the user's literal words.
Behavior also changes in steps. On August 8, 2026, ChatGPT started using the site: operator at scale, jumping from about 0.4% to about 17% of fanout queries, per Promptwatch Data. A tracker that does not replicate live search behavior sees different retrieval than real users do.
Citation counts that move with the product
Citations per response are not a constant. Promptwatch's citation drop report shows average citations falling across GPT-5.3, GPT-5.4, and GPT-5-Mini around the March 4, 2026 rollout. Engines also differ enormously. In Promptwatch's sources-per-response data (September 2025), ChatGPT cited about 5 sources per web-search answer, Google AI Overviews about 10, and Perplexity almost exactly 10. Copilot swung from under 2 to nearly 17. Measuring each engine in its own interface is the only way to know which regime you are in.
Ads and shopping
ChatGPT ads and shopping cards are interface features. Promptwatch publishes data on how often ChatGPT shows ads and shopping feature usage. An API call would never return either, so an API-only tool is blind to them by design.
Source order and formatting
Cognizo makes a related point: API sampling can miss citation formatting, source ordering, and platform-specific presentation that changes how a brand appears to a buyer. Position in the source list is a real signal, and it only exists in the rendered answer.
Is API data ever fine?
Yes, with limits. API collection is cheaper, cleaner, and easier to scale. It also tells you something genuine: what the model believes from training data without live retrieval. That is useful for brand perception checks. The mistake is treating it as a stand-in for what buyers see in the app.
The sensible setup is both. UI collection for the engines your customers actually use, API models alongside for the training-data view, and clear labeling of which is which. Olostep puts it neatly: API-only collection can be cleaner but may miss elements added by retrieval or the interface, so ask how each vendor collects and how often they refresh.
Some limits apply to every method. As Searchable notes, no engine publishes prompt volumes, so tracking cannot tell you how many people saw an answer, and answers vary by user and run.
Five questions to ask any vendor
- Do you collect from the real consumer interface or from the API, per engine? Mixed methods are fine if disclosed.
- Can I open the raw answer behind any metric?
- How many times do you run each prompt, and how often? Windsor.ai's 29% self-agreement figure is the reason to ask.
- Do you capture citations with their order, plus ads and shopping elements?
- Can you track by country, language, and location? Answers differ by region, especially for AI Overviews and local queries.
Tools that collect from real interfaces
Here is where the catalog stands. I have flagged what each vendor claims about collection; confirm details in a demo, since I did not verify pricing or methodology independently.
Promptwatch
Promptwatch monitors the actual user interfaces of ChatGPT, Google AI Overviews, Google AI Mode, Microsoft Copilot, Gemini, Perplexity, Amazon Alexa, and Meta AI in each country and language, and also tracks API models like Claude, DeepSeek, Mistral, and Grok alongside to show what models believe from training data. That split is the one I argued for above. It goes beyond collection: IP-verified AI crawler logs show whether bots can read your pages, ChatGPT Shopping and Ads Radar cover the interface-only elements, and the Content Agent writes and publishes pages against the gaps it finds. Plans start at $95/mo (Essential); check pricing and the free AI visibility report to start.
For the methodology, the how Promptwatch collects data guide explains the process in detail.
Superlines
Superlines describes itself as GEO tracking from real AI interfaces, not APIs. That is the whole pitch, and it is the right one for this topic.
Superlines
Searchable
Searchable says it captures answers the way users see them and cross-checks them against API responses, which is how it covers Google AI Overviews and AI Mode alongside chat engines.
Cognizo
Cognizo argues for UI scraping over API-only sampling and publishes a comparison of tools on that basis. Worth reading with the usual caution that it is a vendor ranking.
Surfer
Surfer's own AI visibility tracking uses scraped interface results, per its research above. Surfer is best known for content optimization.
Others to evaluate
These appear in the catalog as AI visibility trackers. I have not confirmed their collection methods, so ask the five questions above.
Browse more options in the GEO software directory at bestgeosoftware.com.
Comparison table
| Tool | Collection approach (vendor-stated) | Also tracks API models | Beyond monitoring |
|---|---|---|---|
| Promptwatch | Real interfaces across ChatGPT, AI Overviews, AI Mode, Gemini, Copilot, Perplexity, Alexa, Meta AI | Yes | Crawler logs, content agent, shopping, ads, visitor analytics |
| Superlines | Real AI interfaces, not APIs | Not stated | Tracking focus |
| Searchable | Captured answers cross-checked against API | Yes, as cross-check | Scoring and insights |
| Cognizo | UI scraping | Not stated | Tracking plus optimization |
| Surfer | Scraped interface results | Not stated | Content optimization |
How to test a tool yourself
Do not take any vendor's word, including mine. A quick audit takes an afternoon:
- Pick 10 prompts your buyers would plausibly type.
- Run each one by hand in the real ChatGPT app and Google AI Overviews, a few times each, and note which brands appear.
- Compare with what the tool reports for the same prompts.
- Check whether the tool shows the raw answer so you can see where the difference comes from.
If the tool consistently lists far more brands than you saw in the app, you are probably looking at API output. Windsor.ai's 5.3 versus 1.1 gap is exactly the pattern to watch for.
What to do with the data once it is right
Correct collection is the start. The useful part is what you change. Make sure AI crawlers can reach your pages, since blocked pages cannot be cited, and map content against the fanout queries models actually run. If you manage this for clients and want help, an agency like 1001 SEO Media works on GEO alongside technical SEO, and 1001 SEO Media itself uses Promptwatch for this tracking.
Bottom line
An API-only tracker measures the API. If your buyers use the apps, that is a proxy at best, and the published tests suggest the gap is large: 5.3 versus 1.1 tools per answer, and about 75% brand-list non-overlap. Pick a tool that collects from real interfaces for the engines you care about, keeps API models as a labeled secondary view, and lets you open the raw answer behind every number.