Key takeaways
- Screaming Frog is the default desktop crawler: £199 per licence per year, free up to 500 URLs, and the best fit for hands-on investigation.
- Lumar, Oncrawl and Botify are cloud platforms built for large sites, log files, crawl budget and release workflows. All three are quote-based.
- In 2026 the new job for these tools is AI crawler visibility: checking whether GPTBot, ClaudeBot and similar bots can actually reach your pages.
- Do not buy an enterprise platform for a site with a few hundred URLs. A desktop crawler plus Search Console covers it.
- Technical tools tell you what AI bots can read. They do not tell you what AI says about your brand, so pair them with a visibility tracker.
What a technical audit tool needs to do in 2026
The core job has not changed. You need to know whether search engines can discover, render, index and understand your important pages. Status codes, canonicals, hreflang, redirects, internal links, structured data and render problems still make up most of the work.
What has changed is the audience. Googlebot is no longer the only bot that matters. Promptwatch Data's AI crawler traffic report covers IP-verified requests from Jun 8 to Sep 6, 2026. OpenAI made up 79.8% of verified AI crawler requests in the week of Aug 31 to Sep 6, down from 94.8% in Jun 8 to 14. Anthropic, Google, Perplexity and Mistral are taking a bigger share, and the mix keeps moving.
That report gives a simple check. If a provider is a large share in the report but near zero in your own logs, you are probably blocked or unreachable for that provider. Running that check on your own site is a log-file job, and it is where the cloud platforms earn their price.
The four tools at a glance
| Tool | Type | Pricing | Log analysis | Best for |
|---|---|---|---|---|
| Screaming Frog SEO Spider | Desktop crawler | £199/user/year, free to 500 URLs | Separate Log File Analyser | Consultants, in-house SEOs, migrations |
| Lumar | Cloud crawler | Quote only (third-party estimates around $2,667/month) | Yes | Engineering-led teams, pre-deploy checks |
| Oncrawl | Cloud crawler + logs | Quote only | Core strength | Data-heavy teams, custom integrations |
| Botify | Enterprise platform | Quote only (third-party estimates around $5,000/month) | Yes, with AI bot tracking | Millions of URLs, marketplaces, big ecommerce |
Pricing for Lumar and Botify comes from a third-party comparison at prerender.io, not from the vendors. Treat it as a ballpark and get a real quote.
Screaming Frog SEO Spider
Most technical SEOs buy this first. It runs on your machine, you control every crawl setting, and it pulls back titles, status codes, redirects, canonicals, hreflang, structured data and anything you can reach with XPath or regex.
On the official pricing page a licence is £199 per user per year. Volume discounts bring it to £189 for 5 to 9 licences, £179 for 10 to 19 and £169 for 20 or more. Some third-party roundups still quote £259 or $279, so check the official page before budgeting. The version at the time of writing is SEO Spider 24.3, and the Log File Analyser is a separate product at 7.1. One older review puts the Log File Analyser at £99 per year with a free tier of 1,000 log events, but I could not confirm that on the vendor site.
What is new
The feature list now includes crawling with OpenAI and Gemini, custom JavaScript, accessibility auditing, segmentation, a Looker Studio crawl report, crawl comparison and near-duplicate detection. It also connects to Google Analytics, Search Console and PageSpeed Insights, so one crawl can blend your data with Google's.
Limits
The free version caps at 500 URLs. You will see claims that the free tier has no JavaScript rendering and cannot save crawls. The official feature matrix lists both, so that advice looks stale. Verify in the user guide before you rely on either claim.
The paid crawl limit is listed as unlimited, with an asterisk. The real ceiling is the memory and disk you allocate. A big site on a laptop gets painful. It also lives on one desktop, so sharing results means exports, not a link.
A Reddit thread on r/bigseo makes the common point: if you already pay for Semrush, Screaming Frog is still the one people reach for when they need granular detail on broken links, duplicates and redirects.
Lumar
Lumar is the platform formerly known as DeepCrawl. It is a cloud crawler built around automation. Its pitch is fitting SEO checks into engineering workflows, including its Protect product, which tests staging or pre-production builds so regressions are caught before a deploy. Reviewers also say it has added accessibility monitoring and AI-search features in 2026. I could not verify the current details of either on the vendor's site, so ask for a demo that shows them.
Pricing is quote-only. For a small site it is overkill. For a team that ships weekly and keeps breaking canonicals or noindex tags, a pre-deploy check is worth more than a prettier report.
The CMO Club lists Lumar as best for enterprise-level crawl scalability, which matches how most teams use it.
Oncrawl
Oncrawl is the one I would shortlist if your analysts want to work with raw data. Reviewers describe it as strongest on log file analysis and on integrations with other SEO data sources. It combines crawl data with logs so you can segment pages by template or directory and see which ones bots actually hit.
Two things I could not confirm in this research: official pricing, and the extent of its AI bot monitoring. One aggregator mentions it, but that is a weak source. Ask the vendor which AI user agents its log parser identifies out of the box, because that list ages fast.
Botify
Botify is the heaviest option. It unifies crawl data, server logs and Search Console data, and its Activation layer pushes fixes live rather than just reporting them. One reviewer picked it for the depth of its log analysis, specifically the gap between what Googlebot crawls and what your sitemap says it should crawl. That gap is where most large-site problems live.
AI features in 2026
Botify has moved hard toward AI visibility. Per its own release notes:
- AI Visibility sits inside Botify Analytics, launched in beta in October 2025, and is now generally available.
- The April 2026 release added coverage for Google Gemini and Google AI Overview, plus CSV export, a Discoverability Report comparing AI and traditional search down to URL level, and a Deep Think Mode in Botify Assist for diagnosing things like traffic drops.
- LogAnalyzer now tracks Claude-User, Claude-SearchBot and Google-Agent.
- The August 2026 release listed competitor comparison for AI visibility, topic grouping and structured-data features. I did not read that page in full.
The cost is real. Third-party estimates run around $5,000 per month, and another source cites mid-size deployments at $75,000 to $150,000 a year. Both are unofficial, and Botify is sales-gated. It assumes a dedicated team to run it.
Which one should you pick
| Situation | Pick |
|---|---|
| Site under roughly 500 URLs | Screaming Frog free |
| Consultant or in-house SEO, up to a few hundred thousand URLs | Screaming Frog paid |
| Weekly releases, regressions keep shipping | Lumar |
| Analysts who want logs and crawl data blended and segmented | Oncrawl |
| Millions of URLs, marketplace or large catalog | Botify |
| Small local business | Screaming Frog plus Search Console |
Two rules of thumb. First, log analysis only matters if you can get the logs. Many teams on shared hosting or locked-down CDNs cannot, and then a cloud log platform is shelfware. Second, if the site has 80 URLs, a cloud enterprise crawler is a waste of money.
Running an AI crawler check
Whatever tool you pick, the AI-era audit has a few steps:
- Pull 30 days of server or CDN logs and filter to verified AI user agents such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Agent. Verify by IP, because spoofed user agents are common.
- Compare your share per provider to the mix in Promptwatch Data's AI crawler report. A provider that is big there and near zero on your site is a red flag.
- Check robots.txt, CDN and WAF rules for each major provider. Bot-protection settings block AI crawlers more often than robots.txt does.
- Separate training crawlers from answer-time fetchers. Blocking the first is a policy choice. Blocking the second costs you citations.
- Crawl the pages that get fetched and fix render, status code and canonical problems on them first.
Also keep expectations straight about file formats. Promptwatch Data's markdown in AI search report found markdown files make up just 0.05% of AI search citations, so do not rebuild your site around a markdown feed to impress a crawler.
Where AI visibility tools fit
Crawlers and log analyzers answer whether bots can read your site. They do not answer whether ChatGPT or Perplexity mention you, which pages they cite, or what traffic that sends. For that you want a separate layer. Promptwatch does both sides in one place, with IP-verified AI crawler logs, citation tracking and AI traffic analytics, and it also helps produce content for the gaps it finds.
If you are comparing options in that category, the GEO software directory at bestgeosoftware.com lists more of them. Teams that would rather hand the whole workflow to specialists can talk to 1001 SEO Media, which runs technical SEO and GEO work for clients.
Bottom line
Buy Screaming Frog first, almost regardless of size. Add a cloud platform only when you hit a real wall: logs you must analyze at scale, releases that keep breaking SEO, or millions of URLs. Within that tier, Lumar suits engineering workflows, Oncrawl suits data teams, and Botify suits the biggest sites with budget to match. Get quotes from all three, ask each how it identifies AI bots in your logs, and run the five-step check above before you sign.