Key takeaways

Key takeaways¥n¥n- Most AI visibility tools are prompt trackers: they tell you whether you got mentioned, not why you didn't. Crawler logs are the only data source that explains the why.¥n- Ranked by crawler log depth, the order is Promptwatch, Profound, Scrunch AI, Semrush, and Botify. Most of the rest of the market offers no log data at all.¥n- The crawler landscape moved fast in 2026. OpenAI's share of verified crawler requests fell from 94.8% to 79.8% between June and early September, Meta's crawler grew roughly 17x in a month, and Claude's citation crawler grew about 100x in four months (Promptwatch Data).¥n- llms.txt is a dead end for AI search citations. Three independent log analyses found near-zero bot requests for the file. It still has real value for coding agents, which is a different job.¥n¥n## Why crawler logs are the honesty check¥n¥nA prompt tracker tells you your brand appeared in 4 of 50 prompts. It cannot tell you that PerplexityBot requested your comparison page on Tuesday and got a 403 from your WAF rule. That is the gap between monitoring and diagnosis, and it is the entire reason crawler log depth is worth ranking tools by.¥n¥nThe volume of this activity is no longer hypothetical. Cloudflare's CEO Matthew Prince shared in June 2026 that bots now generate 57.5% of HTML web traffic, the first machine majority on record. And the payoff per crawl is still lopsided: Q1 2026 measurements put ClaudeBot at roughly 23,951 pages crawled per referral visit, against about 4.9 pages per visit for traditional Google search. AI crawlers read a lot and send very little traffic back, which is exactly why you need to know which pages they read and what happened when they did.¥n¥nA visibility score says you are at 12%. Logs answer the useful questions underneath it: Is the bot blocked? Did it hit errors? Did it crawl the page ten times and never cite it? Those are three completely different problems with three completely different fixes. Only one class of tool can tell them apart.¥n¥n## How I ranked these tools¥n¥nI scored platforms on four levels of crawler log depth:¥n¥n- Depth 4, full ingestion. The tool ingests your real server or CDN logs, identifies hundreds of bot user agents, verifies them against published IP ranges, tracks errors, and connects crawl events to citations.¥n- Depth 3, gated server-level tracking. The tool reads actual crawl behavior for specific providers, but the capability is locked behind higher plan tiers.¥n- Depth 2, simulation and audit. The tool simulates how a bot would access your site, or audits robots.txt against a fixed bot list, or reads crawler visits out of Google Analytics. Useful, but it is not your logs.¥n- Depth 1, none. The tool counts prompt mentions and nothing else.¥n¥nSources: vendor documentation, third-party pricing reviews verified in August 2026, and Promptwatch Data for crawler behavior. This is not hands-on testing of every platform, so where a capability is tier-gated or unverified, I say so directly.¥n¥n## The ranking: five tools with real crawler data¥n¥n| Rank | Tool | Crawler log depth | What you actually get | Entry price |¥n| --- | --- | --- | --- | --- |¥n| 1 | Promptwatch | Full ingestion, 400+ bots, IP-verified | Real-time logs, errors, crawl-to-citation rate per page | $95/mo (logs from $245/mo) |¥n| 2 | Profound | Server-level, tier-gated | Agent Analytics for how ChatGPT, Gemini, Claude, Perplexity crawl you | $99/mo (crawlers from $399/mo) |¥n| 3 | Scrunch AI | Named crawler tracking plus GA4 | GPTBot, ClaudeBot, PerplexityBot visits plus AI referral traffic | ~$250/mo billed annually |¥n| 4 | Semrush | Simulation and audit | Crawler Profiles, site audit against 8 AI crawlers | $99/mo add-on |¥n| 5 | Botify | Enterprise log analyzer | Server log analysis heritage extended to AI search | Custom |¥n¥n### 1. Promptwatch: full log ingestion with crawl-to-citation attribution¥n¥n¥n¥nPromptwatch is the only platform on this list that ingests real, real-time crawler logs at scale: ChatGPTBot, ClaudeBot, PerplexityBot, GoogleOther, Google-Agent, Meta's agents, and 400-plus others, with the pages they read, the errors they hit, and a crawl-to-citation path that gives you a citation rate per page. Logs flow in from Cloudflare, CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, or a custom HTTP endpoint. Capacity is 25 million logged requests a month on the $245 Professional plan and 100 million on Business at $579.¥n¥nThe detail I rate most is the attribution. A page crawled 500 times and cited once is a content problem. A page crawled zero times is an access problem. Those need opposite fixes, and Promptwatch's Agent Analytics is the only place I've seen both in one view. The crawler traffic dataset also verifies requests against operators' published IP ranges instead of trusting user agents, which matters because user agents are trivial to spoof.¥n¥nThe honest caveat: the free and Essential tiers track prompts, but meaningful log capacity starts at Professional. If you only want mention counts, you would be buying the wrong tool. The logs are the point.¥n¥n### 2. Profound: server-level tracking, if you pay for the right tier¥n¥n¥n¥nProfound's Agent Analytics tracks how ChatGPT, Gemini, Claude, and Perplexity actually crawl your site, and the platform carries real enterprise weight: API access, SSO, and analyst-grade reporting. Data-Mania's August 2026 roundup named it the best enterprise option, and for governance-heavy organizations I'd agree.¥n¥nThe tier gating is the catch. Starter at $99 a month tracks ChatGPT only, and per third-party pricing reviews, crawler analytics is a paid add-on on the $399 Growth plan and only included from the Scale tier up, with enterprise pricing on top. If your budget starts at four figures a month, none of this matters. If it doesn't, you're buying a prompt tracker with a log-shaped upgrade path.¥n¥n### 3. Scrunch AI: named crawler tracking with a fix-it bent¥n¥n¥n¥nScrunch AI tracks named crawlers (GPTBot, ClaudeBot, PerplexityBot) and maps AI referral traffic through a Google Analytics 4 integration. Third-party reviews list its Core plan around $250 a month billed annually, with Claude coverage and the full Agent Experience Platform gated to Enterprise.¥n¥nWhat makes Scrunch interesting is less the logs and more what it does next: the Agent Experience Platform serves AI crawlers a clean, lightweight version of your site at the CDN level, which sidesteps the JavaScript problem entirely (more on that in the pitfalls section). As a log product it is narrower than the two above. As a