Key takeaways
- Agent analytics reads your server or CDN logs, because AI crawlers don't run the JavaScript tags GA4 depends on. If it isn't in the logs, you can't see it.
- Crawl frequency tells you whether bots can find your pages. Errors tell you whether they can read them. Citation rate tells you whether the content won.
- High crawl volume with no citations is usually a content problem. Low crawl volume or lots of 403s is an access problem. Fix the right one.
- Watch bot-level data, not provider totals. A training crawler and an answer-time fetcher mean very different things.
- Treat any shares you see as shares. A growing share of traffic doesn't prove absolute requests grew.
Why you need a different kind of analytics
GPTBot, ClaudeBot and PerplexityBot request your pages server to server. They fetch raw HTML and leave. No script runs, so GA4 records nothing. When ChatGPT fetches your pricing page mid-conversation to answer a buyer, from GA4's point of view that visit never happened.
There is one partial exception. When a person clicks a link inside a ChatGPT answer, that click arrives as a normal browser session and GA4 can see it. The crawl and the live fetch that earned the citation stay invisible. Note also that ChatGPT stopped sending a referrer on October 8, 2026, so even the clicks are easy to misattribute unless you tag them (for example with utm_source=chatgpt.com).
Agent analytics fills the gap by reading the one source that does record everything: your logs. Nick Lafferty's overview of the category frames it as three questions. Which AI systems read my site? What are they reading it for? Did the reading turn into anything?

The three numbers that matter
Most dashboards give you dozens of charts. I'd start with three and ignore the rest until these make sense.
| Metric | Question it answers | Failure looks like |
|---|---|---|
| Crawl frequency | Can bots reach and revisit my pages? | Zero hits on key directories, or a sudden drop |
| Error rate | Can bots read what they reach? | 403, 404, 429 and 5xx responses on important URLs |
| Citation rate | Did the page get used in an answer? | Many fetches, few or no citations |
Read them in that order. There's no point optimizing content for a page that returns a 403 to every bot.
Reading crawl frequency
Start with which bot, not which provider
Provider totals hide the useful part. OpenAI runs GPTBot (training), OAI-SearchBot (search indexing) and ChatGPT-User (live fetches on behalf of a person). Anthropic runs ClaudeBot, Claude-SearchBot and Claude-User. Perplexity has PerplexityBot and Perplexity-User. A training crawl, an indexing crawl and a live citation fetch are different events with different business value.
A rough rule: training bots tell you about the future, indexing bots tell you whether you're eligible, and user-triggered fetches tell you that someone is asking about you right now. A burst of ChatGPT-User hits on your pricing page is worth far more than ten thousand GPTBot hits on old blog posts.
Crawl frequency by directory
Break requests down by folder. If /blog gets 4,000 weekly agent hits while /pricing and /integrations get none, answer engines will omit or guess your commercial details. Neuracite's agent analytics framework makes the same point. Bots reaching only the homepage usually signals a discovery problem: weak internal linking or a thin sitemap keeping crawlers from your deep pages, which is where most citations come from.
Compare against the market mix
Promptwatch Data tracked IP-verified AI crawler requests from June 8 to September 6, 2026. In its weekly AI crawler traffic report, OpenAI made up 79.8% of verified requests in the week of Aug 31 to Sep 6, down from 94.8% in Jun 8 to 14. That is a share of tracked logs, not absolute volume, so don't read it as OpenAI crawling less. The report's practical advice is useful for your own logs: if a provider is a big share in the market data but near zero in yours, you're probably blocked or unreachable for that provider.
Expect surprises
Promptwatch's Meta-WebIndexer report showed that crawler's share going from about 2.2% in mid-July 2026 to 37.8% on August 9. If you saw a new bot suddenly dominate your logs that month, that's why. It's worth checking logs and robots.txt for Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher and FacebookBot, and budgeting for crawl load if your host bills per request.
Reading errors
Errors are the cheapest wins in agent analytics. A bot that keeps hitting 404s on important content, or that suddenly stops requesting your pages, is a visibility problem in the making. Logs surface it weeks before it shows up as a missing citation.
| Status | Usual cause | What to do |
|---|---|---|
| 403 | WAF or bot-fight rules blocking AI user agents | Allowlist verified bots by IP range, review CDN rules |
| 404 on old URLs | Legacy links in model memory or third-party citations | 301 redirect to the current canonical page |
| 429 | Rate limiting too aggressive for live agents | Raise limits for verified AI bots |
| 5xx or slow TTFB | Server strain, especially under heavy crawling | Fix capacity or caching; live agents abandon slow pages |
| 200 with empty body | Content rendered client-side only | Server-side render the critical copy |
Two things people forget. First, spoofed user agents. Anyone can claim to be GPTBot, so counts that don't verify the request IP against the operator's published ranges are inflated by fakes. Second, robots.txt compliance. Your logs are the ground truth for whether a bot respects your directives, and they show whether your file says what you think it says.
When you fix something, your logs are the fastest feedback loop you have. You'll see within days whether crawlers picked up the change, instead of waiting for citation data to move.
Reading citation rate
A crawl means a bot fetched your content. It does not mean any answer used it. Joining crawl data with citation data turns two disconnected numbers into a funnel: crawled, then cited.
Some teams express this as a retrieval-to-citation ratio: prompt runs citing a URL divided by verified retrieval requests for that URL. Treat the formula as a convenient label, not a standard. The idea is what matters. Otterly made the same observation about its own site, noting that its pricing page was among its most-crawled URLs yet barely appeared in citation data.

The diagnostic table
| What you see | Likely meaning | First move |
|---|---|---|
| 200s, healthy crawl volume, no citations | Content problem. The page works but its answer loses to competitors | Add a direct answer near the top, tables, clearer entities |
| Heavy crawling, few human clicks on citations | The answer was complete without a click | Add something worth verifying: calculators, original data |
| Bots only on the homepage | Discovery problem | Fix internal links and sitemap |
| Low crawl, many 403s | Access problem | Audit robots.txt, WAF, CDN rules |
| ChatGPT-User hitting 404s | Stale URLs being fetched live | 301 redirects to current pages |
The key mental move: high crawl volume with no citations looks like a win when it isn't, and low citations without log context look like a content failure when the real problem is access.
How long should a citation take?
Profound reported tracking roughly 900 newly published marketing pages over 60 days. Its median time from publish to first ChatGPT or Claude citation was 6.81 days, with 75% cited within 18.68 days and 90% within 37.1 days. That's a single vendor's sample, so use it as a rough yardstick, not a law. The practical use: a new page that is still uncited well past five weeks is more likely a technical block than a content miss. Check robots.txt, WAF rules and response codes before rewriting anything.
Content type matters too. Promptwatch Data's content half-life report tracked every page ChatGPT cited on July 6, 2026 for 90 days: landing pages and docs stayed cited longest, while news lost half its pages within 36 days. So a citation that disappears isn't always a fault. Compare against your content type before you panic. And since product pages made up 22.4% of ChatGPT citations in the week of Sep 28 to Oct 4, 2026, your commercial pages should be among the most scrutinized in your logs.
A weekly reading routine
Here's a routine that takes about 20 minutes.
- Check the bot mix. Did any bot appear, vanish, or jump? Compare against the market mix.
- Check errors on your top 20 commercial URLs. Anything that isn't a 200 gets a ticket.
- Check directory coverage. Are pricing, product and docs pages being fetched at all?
- Pull citation data for the same URLs. Sort by high crawl, low citation. Those are your content rewrites.
- Note what changed on your site last week and see whether bot behavior moved.
A caution on one idea you'll see floated: that serving markdown to bots boosts crawling. One test in the research showed about 16% higher mean bot visits for a markdown group, but it was not statistically significant and concentrated in pages that were already high-traffic. Promptwatch Data also found markdown files make up just 0.05% of AI search citations. Measure before you re-platform.
Tools that give you this data
You can analyze raw logs by hand with grep and a spreadsheet, and for a one-off audit that works. For ongoing monitoring, a dedicated platform saves a lot of pain, mostly around IP verification and joining crawl data to citations.
Promptwatch reads IP-verified logs of GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and others through Cloudflare, Vercel, Fastly, AWS CloudFront, Netlify, Akamai or the WordPress plugin. It shows which pages bots read, the errors they hit with plain-language explanations, and a citation rate per page, so you can tell an access problem from a content problem. See its AI crawler logs feature for setup details.

Otterly added an Agent Analytics view to its plans, with an overview of total agent visits, pages visited and top engines.

Profound is the enterprise option behind the citation timing benchmark above, and tracks newer personal agents too.
| Tool | Crawler log view | Joins crawl to citations | Best for |
|---|---|---|---|
| Promptwatch | IP-verified, per bot and per page, with crawl failure explanations | Yes, with citation rate per page | Teams that want to diagnose and then fix |
| Otterly.AI | Agent visits, pages, engines, trends | Yes, in combined views | Smaller teams wanting affordable monitoring |
| Profound | Bot tracking including personal agents | Yes | Enterprise brands |
| Raw logs plus spreadsheet | Whatever you build | Manual | One-off audits |
For more options, the GEO software directory at bestgeosoftware.com lists others in the category.
Common misreadings
- Treating a rising share as rising volume. Shares are relative. Always check absolute requests.
- Counting spoofed bots. Without IP verification your numbers are inflated.
- Lumping training and retrieval together. They answer different questions.
- Rewriting content when the page returns a 403.
- Ignoring referral data. Agent analytics shows which pages bots read, but referral data shows where humans land. Some reports suggest a large share of AI referral traffic lands on the homepage, in which case your homepage is now an AI landing page and probably needs to be written for strangers.
What to do next
Start by getting any log source connected, then run the routine above for a month. You'll quickly learn what normal looks like for your site, and after that anomalies stand out on their own. If you want to see how bots currently perceive your site before connecting anything, Promptwatch's free AI visibility report includes a crawler access check.
