Fixing a Website AI Assistants Cannot Read

The fault. A retrieval crawler receives something other than the page: a robots group refusing it, an edge rule answering with a challenge, a login or consent gate returning a stub, or a redirect chain spending the fetch before a live URL arrives. Three retrieval crawlers carry the citations that matter — OAI-SearchBot, PerplexityBot and Claude-SearchBot — and two training crawlers, GPTBot and ClaudeBot, carry none of them. OpenAI states the separation directly: "Each setting is independent of the others — for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot." Google draws the same line on its own control token, documenting that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."

How it is detected. Every template class is requested once per named agent from an address outside the client network, and the status code, the final URL after redirects and the response length are recorded against each agent-template pair. The robots file is read for group precedence, because one crawler obeys the single most specific group naming it and ignores the rest. Edge and bot-protection rules are read alongside it, since a firewall refusal never appears in robots.txt.

The repair. The retrieval agents are named in their own robots group with every site-wide restriction repeated inside it. Google-Extended is permitted. The published address ranges of the named agents are allowed at the edge layer, ahead of the generic bot rule that refused them. Redirect chains collapse to a single hop, and gates returning a stub page to a non-browser client are removed from the tested templates.

How the repair is verified. The agent-by-template matrix re-runs unchanged. The repair passes on two conditions together: every named agent receives HTTP 200 on every tested template, and each agent's own user-agent string appears in the server access log after the date of the edit. A log entry proves arrival, which a status code alone does not.

The fault. The main content assembles in the browser, so a human reader sees a complete page and an agent receives an empty shell. Vercel and MERJ measured this across five agents in December 2024 — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot — and stated the finding without qualification: "None of the major AI crawlers currently render JavaScript." Navigation built from click handlers rather than anchors fails the same way, and it removes every page beyond the entry URL at once.

How it is detected. Each template is fetched twice: once with scripting enabled and once with scripting disabled. The two responses are reduced to their text and their anchor lists, then compared element by element. Every element present in the scripted render and absent from the unscripted response is recorded with the script that produces it. A boilerplate ratio is recorded beside it — main-content bytes measured against total response bytes.

The repair. Main content moves into the response the server sends. Static generation, server-side rendering and prerendering all satisfy that condition, and the choice belongs to the existing stack rather than to a preferred one. Navigation is rewritten as <a href> anchors present in the raw HTML at every breakpoint. Interstitials, consent overlays and age gates are removed from the templates an assistant retrieves.

How the repair is verified. The scripts-off diff re-runs against the same templates. The repair passes on one condition: zero elements present in the scripted render and absent from the unscripted response. The H1 string, the full H2 set, the body text of every section and every navigation anchor are checked by name inside the raw response, and the before-and-after element counts are published in the re-test report.

The fault. A page with no <main> offers a parser no boundary between the article and the navigation, footer and promotional blocks wrapped around it, so no passage segments cleanly and no centerpiece forms. A page with no <h1> states no subject. A heading sequence that jumps from an H2 to an H4 breaks the parent-child relationship the outline depends on. The nine Kansas City competitor pages audited in September 2026 show how common the fault is in this market: three emit no <main> at all, two emit no <h1>, and the median page ships 133 KB of HTML against 229 KB of CSS.

How it is detected. The raw response is parsed and four values are counted: the number of <main> elements, the number of <h1> elements, the heading levels in document order, and the byte share of main content against the whole document. Page-builder templates fail more than one count at once, because the builder wraps content in generic containers carrying no semantic role: in the September 2026 competitor audit the Divi build emits neither <main> nor <h1>, and the Elementor build emits no <h1> on its homepage.

The repair. One <main> wraps the main content and nothing else. One <h1> carries the page subject in words a reader recognises. Heading levels descend without a gap, and each H2 opens a section that stands on its own. The <header>, <nav> and <footer> landmarks mark the chrome that <main> now excludes. Unused builder stylesheets and duplicated media queries are stripped, which lowers the boilerplate ratio measured during detection.

How the repair is verified. The landmark count re-runs on the raw response. The repair passes on three conditions together: exactly one <main>, exactly one <h1>, and zero skipped heading levels across the document. The boilerplate ratio is recorded a second time and both figures are published, because a lower ratio is the number that shows main content gained share against template chrome.

The fault. JSON-LD asserts a rating, a price, a service or an address the rendered page never shows. Two Organization blocks describe one business. The telephone string in the markup differs from the telephone string in the footer by a space, and one business becomes two records. Google Search Central sets the honest ceiling on this work — "Structured data isn't required for generative AI search" — which removes schema from the list of magic fixes and leaves it doing the job it does well: stating an identity and a set of facts without ambiguity. Identity is the open gap in this market. Nine Kansas City competitor pages were audited in September 2026, none emits a Wikidata identifier, and the two that reference a Wikipedia article omit the QID that would disambiguate it.

How it is detected. Every JSON-LD block is extracted and flattened to a list of string values. Each value is searched for in the rendered text of the same URL. Values with no visible counterpart are listed as unmatched. Alongside that list, the @id values are counted for duplicate nodes, and every sameAs target is requested to confirm it resolves.

The repair. Properties with no visible counterpart are deleted rather than illustrated, because markup describing invisible content carries a manual-action risk and no benefit. One Organization node is published sitewide and referenced elsewhere by @id. The legal name, the postal address and the telephone number are written byte-identically in the markup and in every visible instance. The sameAs set points at records that resolve — the industry directories and review platforms the business already holds, and the Wikidata item where one exists.

How the repair is verified. The string-match pass re-runs across the repaired templates. The repair passes on three conditions together: every string value in the markup appears byte-identically in the rendered text of the same URL, every sameAs URL returns HTTP 200, and the structured-data validator reports zero errors. The count of unmatched strings before repair and after repair is published in the re-test report, and the target after repair is zero.

The fault. A page asserts leadership, decades of experience, or a result with no figure, no date and no source behind it. A second page on the same domain states a different founding year, a different service area or a second telephone number. Both faults produce the same outcome: the sentence is unusable as a quotation and the contradiction lowers confidence in every neighbouring claim. The practice itself dates the ceiling on one common boast — Wikidata records the inception of generative engine optimization as 2023-11-16, so a decade of experience in it exists nowhere.

How it is detected. Claim sentences are extracted from every repaired template and sorted into two groups: claims carrying a figure, a date or a named source, and claims carrying none of the three. A second pass compares facts across URLs — founding year, legal name, address, telephone number, service area and service list — and lists every fact stated with two different values anywhere on the domain.

The repair. Each retained claim gains a figure, a date or a named source, written into the sentence rather than into a footnote. Claims that survive none of the three are deleted. Contradictions resolve to one value, chosen by the client in writing, and that value propagates to every URL, every schema block and every profile inside the client's control. Hedged constructions are rewritten as declarations, since a hedged sentence carries no fact for a model to attribute.

How the repair is verified. The claim ledger re-runs across the repaired templates. The repair passes on two conditions together: every retained claim carries a figure, a date or a named source, and no fact returns two values when re-extracted from every URL that states it. The ledger lists each claim, the evidence attached to it, and the URL it sits on, so the check reproduces without the auditor.

Read the table as five paired statements. Blocked crawlers are proved repaired by the agent-by-template status matrix, which passes when every named agent receives HTTP 200 on every tested template and its user-agent string appears in the access log after the edit. Script-dependent content is proved repaired by the scripts-off diff, which passes when zero elements present in the scripted render are missing from the unscripted response. Absent landmarks are proved repaired by the landmark and heading count, which passes at exactly one <main>, exactly one <h1> and zero skipped heading levels. Contradictory markup is proved repaired by the string-match pass, which passes when every markup string is byte-identical to visible text, every sameAs URL returns 200, and the validator reports zero errors. Unverifiable claims are proved repaired by the claim ledger, which passes when every retained claim carries a figure, a date or a named source and every fact holds one value across the domain.

Two documents close the stage. The re-test report names every stage-one check that changed state and every check that did not, with the before value and the after value beside each one. The declined list records the items the client refused, the reason given and the date, so a check that stays failed stays visible rather than disappearing from the record. Neither document is a status call, and no meeting closes the stage.

Repair changes retrieval and extraction, which are the two layers a website controls. Selection stays with the assistant, and measurement of it belongs to stage four: seven assistants carry the citation log, four of them weekly — Google AI Overviews, ChatGPT, Perplexity and Gemini — and three of them monthly — Claude, Grok and Meta AI.