Key takeaways
- Citation trackers answer "did AI mention me?" Crawler logs answer "could AI even read my pages?" You need both, joined at the page level.
- OpenAI made up 79.8% of verified AI crawler requests in the week of Aug 31 to Sep 6, 2026, down from 94.8% in early June, so the bot mix is shifting fast.
- If a provider has a big share of the platform-wide mix but near zero in your logs, you are probably blocked or unreachable for it.
- Cloudflare's AI audit and raw server logs are free and show bot activity, but not citation outcomes. Platforms like Promptwatch and Profound join the two.
- No public source I found publishes crawl-to-citation conversion rates by bot, so build your own baseline.
Why crawl-to-citation data matters
Most GEO tools work the same way. They run prompts against ChatGPT, Perplexity, Gemini and others, then parse the answers for your brand. That tells you the outcome. It tells you nothing about the cause.
If you are missing from answers, there are roughly four possible reasons. The bots never visit you. They visit and hit errors. They read your pages but pick other sources. Or they cite you and you simply are not tracking the right prompts. Only crawler logs separate the first two from the rest.
One roundup from Foundry CRO suggests an order of operations that I agree with: start with crawler analytics (are AI bots reading your content?), then citation monitoring (are they citing you?), then content optimization. Skipping step one is how teams spend months rewriting pages that a WAF rule was blocking the whole time.
Ansvisor's October 2026 tools list splits the market into six groups, and crawler tools get their own category. That matches what I see: it is still the thinnest part of the market.
What AI bot logs actually contain
A log line is simple. A timestamp, a URL, a status code, a user agent and an IP address. The value comes from three things you can derive.
Which bots visit, and which do not
AI companies run several bots each, and they do different jobs. Promptwatch's bot list in its crawler data report names these user agents:
- OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot
- Anthropic: ClaudeBot, Claude-User, Claude-SearchBot, claude-web
- Google: Google-Extended, Google-Agent
- Perplexity: PerplexityBot, Perplexity-User
- xAI: GrokBot, xAI-Grok, Grok-DeepSearch
- Mistral: MistralAI-User, MistralAI-Index
- Meta: meta-webindexer, meta-externalagent, meta-externalfetcher, FacebookBot
- Cohere: cohere-ai, and DeepSeek: DeepSeekBot
The split that matters is training crawlers versus answer-time fetchers. GPTBot collecting training data is not the same event as ChatGPT-User fetching your page because someone just asked a question. The second one is much closer to a citation. Provider totals hide this, so group by crawler, not by company.
Errors and blocked paths
A 403 from a bot protection rule, a 429 from rate limiting, a robots.txt disallow, a JavaScript-only page that renders empty. All of these show up in logs and nowhere else. A tracker will just tell you that you are not mentioned.
Which pages get read
Bots do not crawl evenly. Seeing which pages get fetched, and how often, shows what AI systems consider worth reading on your site. Compare that to the pages you actually want cited. The gap is usually embarrassing.
What the platform-wide data says
Promptwatch publishes IP-verified crawler data, counted from raw server and CDN logs, with spoofed user agents excluded. Some of what Promptwatch Data shows for June 8 to September 6, 2026:
- OpenAI's share of verified AI crawler requests fell from 94.8% (June 8 to 14) to 79.8% (Aug 31 to Sep 6). Other providers are taking a bigger slice.
- These are shares of the weekly mix, not absolute volume. Do not read them as OpenAI crawling less.
Two other movements are worth knowing about.
Promptwatch's Meta-WebIndexer report shows that bot going from about 2.2% of tracked AI crawler requests in mid-July 2026 to 37.8% on August 9, roughly 17x in under a month. It became the heaviest AI crawler in their logs. The report says the surge lines up with chatter that Meta is building its own search index, but that is unconfirmed (it cites a @levelsio post saying "allegedly"). The practical lesson is load: that post said the crawling triggered server alerts, so if your hosting bills per request, budget for it.
Claude's growth is the other one. Promptwatch's Claude crawler data shows citation crawler visits going from about 30 a day in mid-December 2025 to several thousand a day by mid-April 2026, more than 100x, with a peak daily share of 1.73%. Over the whole 130-day window the share was 0.45%. Small, but growing quickly. Check that Claude-User and ClaudeBot are not blocked.
How to read your logs against the benchmark
Here is the most useful single check from the crawler report. Compare your own provider mix against the platform-wide mix. If a provider holds a large share overall but sits near zero in your logs, you are likely blocked or unreachable for it.
A working routine:
- Pull 30 days of logs filtered to verified AI bot IPs.
- Group by crawler, not company.
- List which bots from the list above have zero hits.
- For each missing bot, check robots.txt, CDN rules and WAF or bot-fight settings.
- Look at status codes per bot. Anything above a few percent of 4xx or 5xx deserves a look.
- Sort pages by fetch count and compare with your priority pages.
One warning on spoofing. User agent strings are trivially faked, so a log analysis that trusts them will overcount. IP verification against each operator's published ranges is what keeps the numbers honest.
Where citation data comes in
Once bots can reach you, the question becomes whether fetches turn into citations. Two things from Promptwatch Data help set expectations.
- ChatGPT's unique domains per response data shows almost half of responses cite 10 or more domains (Sep 28 to Oct 4, 2026), so the competition for a slot is real, but there are many slots.
- Promptwatch's content half-life report found landing pages and docs stayed cited longest among pages ChatGPT cited on July 6, 2026, while news lost half its pages within 36 days. A crawl log showing steady fetches of your docs, with citations decaying slowly, is a healthy pattern. News pages need constant refreshing.
I will be upfront: I could not find any public dataset of crawl-to-citation conversion by bot. The reports give crawler mix, not a ratio. So you have to compute your own: for each page, count answer-time fetches and count citations over the same period. Even a rough ratio per page type tells you where to focus.
Tools that show crawler data
The market splits into three levels. Free log analysis, crawler-only dashboards, and platforms that join crawls to citations.
Free options
Cloudflare's AI audit dashboard shows which AI bots crawl your site, requests per day, pages fetched and bytes transferred. Foundry CRO's review is blunt about the limit: it shows bot activity, not citation outcomes. Raw server log parsing is the other free route. Foundry CRO also claims a free stack covers about 80% of what paid tools do. That is the vendor's claim and I would treat it as plausible for crawl monitoring, less so for citation tracking at scale.
Platforms that join crawls to citations
Promptwatch ships IP-verified crawler logs (Agent Analytics) for bots like GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Meta-WebIndexer, with plain-language crawl failure explanations and a crawl-to-citation path with a citation rate per page. aiclicks.io describes it as joining crawler logs to the pages they cite instead of giving raw log data. Logs come in through Cloudflare, Vercel, Fastly, AWS CloudFront, Netlify, Akamai, a custom endpoint or the WordPress plugin. It then connects findings to action: the Action Board, content gap analysis and the Content Agent that publishes to WordPress, Webflow or Framer.
Profound also offers agent analytics, per the same aiclicks.io roundup, at a higher price point.
Botify comes at this from the technical SEO side, which suits enterprise teams that already live in log files.
Trackers without crawler logs
Pure citation trackers are fine for the outcome side. Examples in the catalog include Otterly.AI, Peec AI and AthenaHQ. I did not verify their current feature sets or pricing, so check each vendor before buying.
Quick comparison
| Approach | Shows bot visits | Shows citations | Joins the two | Cost |
|---|---|---|---|---|
| Server logs | Yes, manual | No | No | Free |
| Cloudflare AI audit | Yes | No | No | Free tier available |
| Otterly.AI / Peec AI | No | Yes | No | Paid, entry-level |
| Profound | Yes | Yes | Yes | Paid, enterprise-leaning |
| Promptwatch | Yes, IP-verified | Yes | Yes, per page | Free Explore plan, paid from $95/mo |
Confirm current pricing at promptwatch.com/pricing and on each vendor site. For more options in the category, the GEO software directory lists a wide set of tools.
Diagnosing problems with joined data
Once you can see crawls and citations together, the patterns are fairly readable.
| Pattern in your data | Likely cause | First fix |
|---|---|---|
| A provider has zero hits in your logs | Blocked by robots.txt, CDN or WAF | Allow the bot, re-check in a week |
| Frequent fetches, errors on key pages | Server errors, rate limits, rendering | Fix status codes, serve plain HTML |
| Many fetches, few citations | Content does not answer the prompt | Rewrite or add pages for the prompts |
| Few fetches, some citations | Cited from third-party pages or old index | Improve internal links, sitemap coverage |
| Page cited, then dropping off | Content decay | Refresh the page |
The third row is where most teams get stuck. A bot reading your page and not citing it is a content problem, and no log analysis will fix it. That is where gap analysis against the prompts people actually ask comes in.
A practical setup in an afternoon
- Decide where logs live. If you are behind Cloudflare, Logpush is the simplest route. Otherwise use your CDN or host logs.
- Send logs to your chosen tool, or export a month of raw logs to analyze yourself.
- Run the missing-bot check against the list above.
- Review robots.txt and WAF rules for every provider, not just OpenAI.
- Set a monthly review. Baseline the bots, especially fast-growing ones like Claude and Meta.
- Add citation tracking for your 20 to 50 most important prompts and compare per page.
If you want a quick outside view before connecting anything, free checks exist, such as Promptwatch's Agentic Readiness Check and AI Crawler URL Inspector, which test which AI crawlers can read your site.
Bottom line
Citation tracking without crawl data is half a diagnosis. Start with free logs or Cloudflare to find out whether bots can reach you at all. Then, once the basics are clean, pick a platform that joins crawls to citations so you can see which fetched pages turn into answers. The research is honest about one gap: nobody publishes crawl-to-citation ratios by bot yet, so your own baseline is the benchmark that matters. For wider data on how AI answers are built, Promptwatch Data is the place to start.