How AI crawlers read your site (and why JavaScript sites lose)
Assistant crawlers mostly fetch raw HTML and move on. If a page assembles itself in the browser, the machine reads a near-empty shell. Hayley on the render gap, and how to fix it.
Two visitors, two very different pages
When a person opens your site, their browser does a lot of quiet work on their behalf. It downloads a small HTML file, runs the JavaScript that file references, fetches data from your APIs, and assembles the finished page in front of them. A few hundred milliseconds later they see headings, prices, articles and links, and they have no idea any of that happened. The page simply appeared. An AI crawler is a different kind of visitor, and it usually stops at the first step. It requests the HTML and reads what comes back. Many of the crawlers now feeding AI assistants do not run the JavaScript at all, or run it inconsistently and on their own schedule. So the question that decides whether your content exists to a machine is brutally simple: what is actually in the HTML your server sends, before a single line of script has run? If the answer is a loading spinner and an empty div, that is the page the assistant reads.
Why assistant crawlers skip the JavaScript
Executing JavaScript is expensive. To render a modern page a crawler has to run a full browser engine, wait for scripts and network calls to finish, and hold that process open long enough for the page to settle. Doing that once is cheap. Doing it across billions of URLs is not, and the crawlers pulling content for AI answers are optimised for breadth and speed rather than for faithfully rendering every single-page app they meet. Google's own search crawler will render, but it does so in a second pass that can lag the first by anything from seconds to days, and even that is a best effort rather than a promise. The newer assistant crawlers are less generous still. Many take the raw HTML and move on. This is not a bug you can report or a setting you can switch. It is a deliberate trade-off on their side, which means the responsibility for being readable sits entirely on yours.
The render gap, and why your browser hides it
We call the distance between those two pages the render gap: the fully built page a user sees versus the thin shell a non-rendering crawler receives. The reason it goes unnoticed for so long is that every human on the team, including everyone who signs off the site, only ever sees the rendered version. It looks immaculate. Design approves it, the client loves it, and nothing on screen hints that a machine is getting something else entirely. The symptoms show up later and in the wrong place. Rankings that never arrive for pages that look perfect. An assistant that describes a competitor accurately but gets your offering vague or wrong. Product and article pages that seem to be indexed but carry none of their real content. Teams burn weeks chasing content quality and backlinks when the actual fault is that the words were never in the HTML to begin with. The front end gives no warning, so the diagnosis rarely lands where the problem is.
How to see what a machine actually receives
You do not need special tooling to catch this, only the discipline to stop trusting the rendered view. View the page source rather than inspecting the live DOM, because source shows you the raw HTML as delivered while the inspector shows you the assembled result after JavaScript has run. If your headings, body copy, navigation links and metadata are present in that source, a crawler can read them. If the source is a near-empty shell and the words only appear in the inspector, you have found your render gap. For a firmer test, fetch the page with JavaScript disabled, or pull it with a plain command-line request that does no rendering at all, and read what comes back. That is much closer to what an assistant crawler sees than any browser tab. Do it for your highest-value pages first: the ones you most want quoted, ranked or recommended. The pages that matter most commercially are exactly the ones where an invisible render gap costs the most.
Putting your content back in the raw HTML
The fix is to make sure the meaningful content is present in the server's initial response, before the browser does anything. In practice that means server-side rendering or static pre-rendering, so the HTML that leaves your server already contains the headings, the copy, the links and the metadata. Frameworks like Next.js, Nuxt and their peers support this directly. The goal is not to abandon JavaScript, it is to stop depending on the client to construct the parts of the page a machine needs to read. This is rarely a full rebuild. Often the highest-value content can be moved into the server response while the interactive layer stays client-side, and the metadata that quietly breaks first, titles, descriptions and canonicals written by script, moves into the server response where a crawler cannot miss it. The principle to hold onto is that the version a machine indexes should match the version a person sees. When those two pages agree, most of the mysterious ranking and citation problems that trace back to rendering simply stop happening.
Why this matters more in the AI era than it did
For years the render gap was mostly a search problem, and Google's rendering pass papered over a lot of it. AI assistants have changed the stakes. When someone asks an assistant to recommend a supplier or summarise what your business does, the model draws on content its crawler could actually read. If your pages arrived as empty shells, you are not in the running, and there is no ranking report to tell you why. You are simply absent from the answer, and a competitor whose content sits in plain HTML is present. This is why rendering strategy has stopped being a front-end preference and become a core question for answer engine optimisation and for entity recognition. An assistant cannot attribute a fact to your business, or connect you to the concepts your buyers search for, if the text that establishes those facts never reached it. Getting your content into raw HTML is the unglamorous foundation underneath all of it. Do that, and everything downstream, from technical SEO to how confidently an assistant names you, has something real to work with.