Perfect HTML, zero citations: the header AI crawlers check first

If an AI crawler cannot parse your page, nothing else about your content strategy matters. In August 2026 we audited our own site and found 46 pages serving flawless, server rendered HTML that no AI crawler could read, because the server was labelling them as the wrong kind of file.

The pages had correct titles, canonical tags and structured data. They ranked fine. They were also, as far as any crawler honouring the HTTP Content-Type header was concerned, binary downloads rather than web pages.

This is the layer beneath everything the current wave of GEO advice talks about, and almost nobody audits it.

What the wrong content type actually does

The Content-Type header tells a client what it just received. A page served as text/html gets parsed as a document. A page served as application/octet-stream is a stream of bytes with no declared meaning.

Browsers hide this from you. They sniff the content, notice it looks like HTML, and render it anyway. Your page looks perfect in Chrome.

Crawlers are stricter, and they have every reason to be. A parser that trusts a declared type does not waste compute guessing at binary blobs. As MDN puts it, when a server reports an incorrect MIME type the client has no way of knowing the author intent.

So the title tag is there. The JSON-LD is there. The FAQ schema is there. None of it is read.

Why every standard audit misses this

SEO tools check what a page contains. This is a fault in how the page is delivered, so the usual checks all pass.

• View source shows correct HTML, because the browser sniffed and rendered it.
• Rich results tests pass, because you paste the markup in by hand.
• Rankings hold, because Google sniffs content types too.
• Sitemap coverage looks complete, because the URLs return 200.

We only caught it by fetching every URL in our sitemap and extracting the title from each response. Forty six came back empty. The empty titles were the symptom. The header was the cause.

The bug, precisely

Ours was four characters of Express. It is worth naming because the same shape of mistake exists in most server frameworks.

Express res.type(x) treats x as a literal MIME type only if the string contains a slash. Without one, it runs a lookup against known file extensions.

So res.type("text/html; charset=utf-8") works, because it has a slash. res.type("html") works, because the lookup succeeds. But res.type("html; charset=utf-8") has no slash, the lookup fails against the whole string, and Express falls back to application/octet-stream.

That single line silently removed 46 pages from our entire public citation surface. They had been live for weeks.

Why this matters more in 2026 than it did in 2023

Two things changed. The first is that AI crawlers do not behave like Googlebot.

Research published through 2026 found that none of the major AI crawlers execute JavaScript, including GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot. They fetch raw HTML and read what is there. Google, by contrast, passes pages to a rendering service running headless Chrome.

That difference is why server side rendering became the standard GEO recommendation. It is correct advice. It is also incomplete, because rendering your content into the HTML accomplishes nothing if the transport layer then mislabels the result.

The second change is that the traffic now justifies the attention. Ahrefs measured referral traffic on its own site and found that visitors arriving from an AI assistant converted at roughly 23 times the rate of a typical Google visitor.

Most teams are working on layers one, three and four. Layer two is where our 46 pages died.

How to check your own site in ten minutes

You do not need a tool. You need the response headers for pages you actually care about, fetched the way a crawler fetches them rather than the way a browser does.

1. Take every URL from your sitemap, or at minimum your highest value template types. Product, comparison and documentation pages carry the most citation value.
2. Request each one and read the Content-Type response header. Anything that is not text/html on an HTML page is a defect.
3. Extract the title from each response body. Blank titles on pages that clearly have titles indicate the response was not returned as text.
4. Fix it in the handler that sets the type, not in a proxy or CDN rule. Header rewrites downstream mask the bug and it returns with the next route someone adds.

Do this per template rather than per page. One broken handler takes out every page that uses it, which is exactly why our count was 46 and not one.

The wider point about technical GEO

The current GEO conversation is mostly about content shape: answer capsules, structured headings, schema markup, entity clarity. That work is real and it matters. We have written about what GEO actually is and published a step by step AI search visibility checklist.

But content optimisation assumes delivery works. Plenty of brands have no GEO strategy at all, and among those that do, the effort concentrates almost entirely on what the page says rather than whether the page arrives in a readable state.

The checks are unglamorous and they are cheap. Serve the right status code. Serve the right content type. Put the content in the raw HTML. Then optimise the words.

We found this on our own site, which is the only reason we can write about it with specifics. If you want the same audit run against yours, that is where a Revenue Engine Diagnosis starts. Book one here.