Key takeaways
- Two different problems get lumped together: crawlers that can't fetch or parse your pages, and browser agents that can't use your interface. Each needs its own fix.
- Most failures come from boring causes: robots.txt, CDN or WAF rules, heavy client-side JavaScript, and unclear markup.
- HTML is 99.94% of AI search citations. Markdown mirrors of marketing pages are not the fix.
- Server logs tell you which bots reach you. Without them, you are guessing.
- Test agent task completion on your own critical journeys. Public readiness scanners do not tell you that.
Two kinds of failure, one confusing symptom
When someone says "AI can't read my site," they usually mean one of two things. Either a crawler (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) never got your content, or a browser agent got there and could not finish a task. The first costs you citations. The second costs you conversions.
MERJ, a technical SEO consultancy, calls the second problem the interaction layer. For years sites were built for people and made discoverable for crawlers. Browser agents are a third visitor that has to actually use the interface. MERJ began its research in November 2025 and ran 100 purpose-built tests plus live production pages across multiple models. Its own disclaimer matters: a scenario that failed in December may pass today, so treat findings as version-specific.

One test stuck with me. MERJ gave two buttons misleading ARIA labels and asked an agent to save a document. It clicked Cancel first in 25 of 25 runs. No ticket gets filed for that. Analytics shows a session that looks roughly human. You would never know.
A note on scope: you will find plenty of 2026 articles titled "why AI agents fail" that are about enterprise agent architecture (monolithic "God agents," infinite API loops, orchestration debt). That is a different problem, about building agents. This guide is about why agents fail on your website.
The 8 causes at a glance
| # | Cause | Affects | Quick check |
|---|---|---|---|
| 1 | robots.txt blocks AI bots | Crawlers | Read robots.txt per user agent |
| 2 | CDN/WAF challenges or rate limits | Crawlers, agents | Compare logs against expected provider mix |
| 3 | Client-side rendering | Crawlers | View source vs rendered DOM |
| 4 | Weak heading and semantic structure | Crawlers, agents | Audit headings and landmarks |
| 5 | Misleading or missing accessible names | Browser agents | Inspect ARIA labels and button text |
| 6 | Login walls, CAPTCHAs, interstitials | Browser agents | Run the journey in a clean session |
| 7 | Silent state and feedback failures | Browser agents | Confirm the state change actually happens |
| 8 | No visibility into agent behavior | Both | Add crawler logs and task-level tests |
1. robots.txt rules that block the wrong bots
The simplest cause and still the most common. A blanket Disallow: / for a user agent, an old rule copied from a template, or a plugin that added AI-bot blocks without anyone noticing.
The complication is that providers run several bots with different jobs. Training crawlers, search indexers, and user-triggered fetchers are separate user agents. Blocking one does not block the others, and allowing one does not allow the rest.
The fix
- List every AI user agent you care about and decide on each one deliberately.
- Allow the search and user-fetch bots if you want citations and visibility. Block training crawlers only if that is your actual policy.
- Re-check after CMS, security plugin, or hosting changes.
A directory like promptwatch.com/bots lists 400+ AI crawlers with robots.txt guidance, and the Robots.txt Generator for AI helps you write the file without guessing at user agent strings.
2. CDN and WAF rules that challenge or drop AI traffic
This is the one that bites teams who think robots.txt is fine. Your file says "come in," and your CDN says "prove you're human." Bot management defaults, rate limits, and managed challenge pages can return 403s, 429s, or a JavaScript challenge that a crawler cannot solve.
Promptwatch Data's AI crawler traffic report (IP-verified requests, June 8 to September 6, 2026) gives you a benchmark to compare against. OpenAI made up 79.8% of verified AI crawler requests in the week of August 31 to September 6, down from 94.8% in the week of June 8 to 14, with Anthropic, Google, Perplexity, and Mistral making up the rest. The report's advice is blunt: if a provider is a large share in the benchmark but near zero in your logs, you are probably blocked or unreachable for that provider.
The fix
- Check robots.txt, CDN, and WAF rules for every major AI provider, not just the one you hear about most.
- Allowlist verified AI bot IP ranges where your security setup permits it. Verify against each operator's published ranges, because user agent strings are trivially spoofed.
- Look for 403, 429, and 5xx responses in logs, not just absence of traffic.
3. Client-side rendering hides your content
If your HTML ships as an empty shell and content appears after JavaScript runs, many retrieval systems will see little or nothing. Some crawlers execute JavaScript, some do not, and none of them promise to wait for your slow API calls.
Promptwatch Data's markdown in AI search report is direct on this: fix heavy client-side rendering first, because a page a retrieval system can't parse loses citations simply by being unreadable.
The fix
- Server-side render or statically generate pages that need to be cited: product pages, docs, landing pages, comparisons.
- Put the primary content, headings, and key facts in the initial HTML response.
- Test with
curland view-source. If your main copy isn't there, a bot may not see it either.
curl -s -A "OAI-SearchBot" https://example.com/pricing | grep -i "your key phrase"
If that returns nothing, you have a rendering problem. (Run it as a quick sanity check, then confirm with real logs, since some firewalls treat spoofed user agents differently.)
4. Weak structure: headings, landmarks, and semantic HTML
A page can be technically fetchable and still be hard to use. Div soup, skipped heading levels, and key facts buried in images or tabs all make extraction harder, for crawlers and for browser agents.
The same report recommends putting effort into the HTML itself: a clear heading hierarchy, semantic markup, and content that answers a question directly. HTML accounts for 99.94% of AI search citations in that data, markdown for 0.05%, images for 0.015%. That is roughly 2,000 to 1 in favor of HTML.
The fix
- One H1, logical H2 and H3 nesting, descriptive headings.
- Use real elements:
<button>,<nav>,<main>,<table>,<label>. - State the answer near the top of the section. Don't make a parser dig through a carousel.
Do not build markdown mirrors for marketing pages
It is tempting to publish .md versions of every page for AI. The data says that effort belongs elsewhere. Markdown makes sense for developer documentation, where coding agents like Claude Code and Codex fetch specific docs, READMEs, and API references directly. Promptwatch logs show anthropic-claudebot and openai-searchbot requesting /docs/.../*.md paths with thousands of requests on top paths. For a blog or product page, spend the time on the HTML.
5. Misleading or missing accessible names
This is where browser agents diverge from crawlers. An agent navigating your UI relies heavily on the accessibility tree: roles, names, states. If a button says "Save" visually but carries an ARIA label of "Cancel," the agent trusts the label. That is exactly what MERJ's 25-out-of-25 result shows.
MERJ's broader conclusion is that a large and commercially important share of browser-agent failures come from the same structural, naming, state, and feedback defects that accessibility engineering already targets. And its caveat is worth repeating: accessibility exists so people with diverse abilities can use the web. Agent compatibility is a side effect, not the reason.
The fix
- Make visible text and accessible names agree. Audit
aria-labelandaria-labelledbyfor leftovers. - Give icon-only buttons real names.
- Prefer native elements over custom clickable divs.
- Run an accessibility audit and fix the findings, then re-test with an agent.
6. Login walls, CAPTCHAs, and interstitials
Cookie banners that block the page, forced sign-ups before pricing, age gates, newsletter modals, and CAPTCHAs all stop an agent cold. A person dismisses a modal without thinking. An agent may click the wrong thing, loop, or give up.
The research notes I had did not cover vendor-specific behavior here (for example, how each crawler handles Cloudflare's default challenges), so test your own stack rather than assuming.
The fix
- Keep pricing, docs, specs, and policies reachable without an account.
- Make modals dismissible with a clearly labeled close control, and make sure they don't trap focus or cover required content.
- Do not put a CAPTCHA in front of read-only content you want cited. Reserve it for actions that need it, like form submission.
7. Silent state and feedback failures
An agent clicks "Add to cart." Nothing visibly changes for 800ms, then a small toast appears with no accessible announcement. The agent concludes the click failed and clicks again, or moves on. MERJ's monitoring approach checks exactly this: whether the intended state change actually occurred, not just whether a click happened.
The fix
- Expose state changes in the DOM: updated cart counts,
aria-liveregions for confirmations and errors, disabled states that are really disabled. - Make errors specific and attached to the field they refer to.
- Avoid actions that only succeed after hover, long-press, or timing tricks.
8. You can't see what agents are doing
The last cause is a meta one. Most sites have no idea which AI bots fetch them, which pages fail, or whether an agent completed a journey. Analytics lumps agent sessions in with human ones, and crawler requests never show in JavaScript-based tools at all.
Watch the crawlers
Server or CDN logs verified against provider IP ranges are the ground truth for crawler access. You can read raw logs yourself, or use a platform. Promptwatch has IP-verified AI crawler logs that show which pages bots read, how often, and what errors they hit, plus a crawl-to-citation view that helps separate an access problem from a content problem. It works with Cloudflare, Vercel, Fastly, and raw server logs. See the AI crawler logs feature page for setup details.

Other tools worth knowing for the technical side of this problem: Botify for enterprise crawl analysis, and the free Agentic Readiness Check for a quick look at which AI crawlers can read your site and which agent standards you support.
Watch the tasks
Crawler logs will not tell you whether a browser agent can check out. MERJ's point is that public benchmarks and readiness scanners don't tell a company whether agents can complete its critical journeys. Only task-level measurement on your own site does. Define two or three journeys that matter (find pricing, book a demo, add to cart, save a document), run agents through them repeatedly across more than one model, and record whether the intended state change happened. Browser infrastructure like Browserbase or open-source Skyvern can drive these runs.

Diagnosing which problem you have
| Symptom | Likely cause | First move |
|---|---|---|
| One provider shows near-zero in your logs | Robots rule or WAF block | Check rules for that user agent and IP range |
| Bots hit the page, but it is never cited | Thin initial HTML or unclear content | View source, check rendering and headings |
| 403 or 429 for AI user agents | Bot management or rate limits | Adjust challenge rules, allowlist verified ranges |
| Agent starts a task and abandons it | Modal, login wall, CAPTCHA | Run the journey in a clean session |
| Agent clicks the wrong control | Bad accessible names | Audit ARIA labels and button text |
| Agent repeats an action | No visible or announced feedback | Add state changes and live regions |
A practical order of operations
- Read your robots.txt and list which AI user agents are allowed or blocked.
- Check CDN and WAF logs for blocked or challenged AI requests.
- Compare your crawler mix with a benchmark such as Promptwatch Data's AI crawler traffic report.
- View source on your ten most important pages. If the content isn't there, fix rendering first.
- Clean up headings and semantic HTML on those pages.
- Audit accessible names, modals, and form feedback on your key journeys.
- Set up recurring agent task tests. Re-run them when a major model releases, since behavior shifts.
- Only then think about extras like llms.txt or markdown endpoints for developer docs.
For more tools in this space, the directory at bestgeosoftware.com is a reasonable place to compare options.
What to expect
None of this is glamorous. A blocked bot rule fixed in five minutes can matter more than a month of content work, and a mislabeled button can quietly send agents the wrong way on every run. The fixes overlap heavily with things you should already be doing for accessibility, performance, and crawlability, so the cost of getting them right is mostly attention. Start with the logs, because they will tell you which of the eight you actually have.

