Log file analysis that shows what search engines really do on your site.
A crawling tool tells you what it found. Your server logs tell you what Googlebot, Bingbot and the AI crawlers actually did: which URLs they hit, how often, what they got back, and what they never touched. We read those logs line by line and turn them into a short list of fixes that put crawl budget back on the pages that earn enquiries.
Log file analysis for SEO is the practice of reading a server's raw access logs to see how search engine and AI crawlers genuinely behave on a site, rather than relying on what a crawling tool infers from the front end. Every request a bot makes leaves a line in the log: the URL, the user agent, the HTTP status returned, the timestamp and often the bytes served. Analysed properly, those lines reveal wasted crawl budget on parameter noise and dead redirects, important pages that are crawled rarely or never, and status-code and bot-behaviour patterns no simulated crawl can surface. It is the only source of ground truth about crawling, which is why it sits at the centre of serious technical SEO on large or complex sites.
Log file analysis, in full.
Almost every technical SEO tool works by guessing. It starts at your homepage, follows the links it can see, renders what it can, and builds a picture of your site from the outside in. That picture is useful, but it is an assumption about how a search engine might behave, not a record of how one did. Server logs are the record. They are the one dataset where you are not modelling Googlebot's behaviour, you are reading it, request by request, exactly as your infrastructure saw it.
This matters most on the sites where technical SEO matters most: large ecommerce catalogues, publishers with deep archives, marketplaces, and any site where the number of URLs runs well ahead of the number that actually need to rank. On a fifty-page brochure site, crawl budget is not a constraint and logs tell you little you could not infer elsewhere. On a site with hundreds of thousands of URLs, Google is making constant decisions about what to crawl, how often, and what to quietly ignore, and those decisions are invisible until you open the logs.
The commercial stakes are concrete. If Googlebot is spending the bulk of its visits re-crawling filtered category variants, expired listings and redirect chains, it is not spending them discovering your new products or re-assessing the pages you have just improved. Content gets indexed slowly or not at all. Fixes take weeks longer to register than they should. None of this appears as a single alarm in a traffic report; it shows up as a site that feels sluggish to respond no matter how good the work upstream is.
There is a further reason logs have become essential. It is no longer only Googlebot and Bingbot on your server. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and a growing list of AI crawlers now request your pages, and their behaviour differs from the traditional search bots. Logs are where you see who is crawling, how aggressively, and whether your infrastructure is serving or throttling them. If you care whether assistants can reach your content, the logs are where that question is answered.
Log file analysis, broken down.
Crawl budget and frequency mapping
We aggregate every bot request by section, template and URL to show where crawl budget genuinely goes. Which directories are crawled daily, which monthly, which never. The gap between where budget is spent and where your revenue sits is usually the whole story.
Explore →Status code and redirect diagnosis
Logs expose the 404s, 500s, soft 404s and redirect chains that bots hit repeatedly, not just the ones a front-end crawl stumbles across. Persistent server errors and long redirect hops waste budget and erode trust in the affected paths, and we trace each back to its cause.
Explore →Orphan and rarely-crawled page discovery
By comparing log data against a full crawl and your sitemaps, we find pages that exist and should rank but are crawled rarely or never, and pages Google visits that you did not know were exposed. Both distort how authority flows through the site.
Crawl versus render reality
Logs show whether bots are fetching your JavaScript and API responses at all, which is the first question behind any rendering problem. Where the two intersect, we hand off to a deeper render investigation rather than guessing at the front end.
Explore →Bot verification and fake-crawler filtering
A meaningful share of traffic claiming to be Googlebot is not. We verify crawlers by reverse DNS and IP range so the analysis rests on genuine search and AI bots, and so you can see where scrapers are impersonating them and eating resource.
AI and LLM crawler behaviour
We separate out GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest, so you can see which assistants are reaching your content, how often, and whether robots rules or rate limits are quietly shutting them out.
The way we run Log file analysis.
Get the logs, in full
We work with you or your hosting team to pull raw access logs covering a representative window, ideally several weeks and across peaks, from every layer that serves requests: origin, CDN and any load balancers. Partial logs give partial truth, so we are particular about completeness up front.
Parse, verify and normalise
We process the raw lines into a clean dataset, verify claimed search bots against their real IP ranges to strip out impostors, and normalise URLs so parameters and variants group sensibly. This is the unglamorous step that decides whether every later number can be trusted.
Cross-reference against crawl and Search Console
Logs on their own tell you what was crawled; joined to a full crawl, your sitemaps and Search Console data, they tell you what it means. This is where wasted budget, orphaned pages and indexation gaps become visible as specific, named URLs.
Diagnose and prioritise
A senior practitioner reads the patterns, separates the findings that affect traffic and revenue from the long tail of curiosities, and traces each real issue to its cause. You get an ordered plan, not a raw export of everything a bot ever touched.
Fix at source
We implement the changes or specify them clearly for your developers: robots and canonical rules, redirect consolidation, internal-linking changes and parameter handling, all applied in templates and configuration so the fix holds across the whole URL set.
Re-pull and confirm
After the fixes land we take a fresh log sample and check that crawler behaviour actually changed: budget shifting toward the pages that matter, error patterns clearing, important URLs crawled more readily. We confirm against the record rather than assuming.
How we approach Log file analysis.
Ground truth, not simulation
A crawler like Screaming Frog models how a bot might traverse your site. Logs record what Googlebot and the AI crawlers actually did. When the two disagree, and on real sites they routinely do, the logs win. We start from the record, then use the crawl to explain it.
Crawl budget is finite, so we spend it deliberately
On a large site Google will not crawl every URL as often as you would like. Every request spent on a faceted duplicate, a tracking parameter or a chain of redirects is a request not spent on a page that makes money. We find where budget leaks and close the leaks at source.
Senior reading, not an automated export
Parsing a million log lines is trivial. Knowing which patterns matter is not. A senior practitioner reads the aggregates and the anomalies, weighs them against the revenue behind each section, and tells you what is worth acting on and what is safe to leave.
What Log file analysis puts on your desk.
Why log analysis is senior work
Anyone can load a log file into a parser and produce charts. The value is in reading them: knowing that a spike in Googlebot 304s on one template is fine while the same pattern on another is a problem, or that a section crawled hard but converting nothing is a budget leak worth closing. That judgement comes from having done this on real sites for years, not from a tool's defaults. The four of us share roughly forty years of technical SEO between us, and the person who reads your logs is the one who scopes and specs the fix. Nothing is handed down a chain, so the context behind a finding is never lost before it is acted on.
We read the record ourselves
Verified bot data, status patterns and crawl frequency are interpreted by a senior practitioner, because the findings that matter rarely sit at the top of an automated severity list.
Joined to the wider picture
Logs are read alongside a full crawl, your sitemaps and Search Console, so a finding is a named URL with a cause and a cost, not an isolated number on a chart.
Honest about scale
On a small site we will tell you logs add little and point you at cheaper wins. We recommend this work where crawl budget is genuinely a constraint, not as a default line item.
Log file analysis: common questions.
What is log file analysis in SEO, and why does it matter?
It is the practice of reading a server's raw access logs to see how search engine and AI crawlers actually behave on your site: which URLs they request, how often, and what status codes they receive. It matters because logs are the only dataset that records real crawler behaviour rather than simulating it. On large or complex sites, that record reveals wasted crawl budget, rarely-crawled pages and error patterns that no front-end crawling tool can see.
How is log file analysis different from a crawling tool like Screaming Frog?
A crawling tool starts at your homepage and follows links to model how a bot might move through your site. It is a simulation. Log file analysis reads what Googlebot and the AI crawlers genuinely did, request by request, as your server recorded it. The two are complementary: we cross-reference them, and where a simulated crawl and the real logs disagree, the logs are the ground truth.
What server logs do you need, and how do I get them to you?
We need raw access logs covering a representative window, ideally several weeks, from every layer that serves requests: your origin server, your CDN, and any load balancers in front of them. Most hosting platforms and CDNs can export these, and we will give your team precise instructions for the setup you run. Completeness matters more than anything here, because partial logs produce partial conclusions.
How does log file analysis help with crawl budget?
By showing exactly where crawler requests go. When we aggregate bot activity by section and template, it usually becomes obvious that a large share of budget is spent on faceted duplicates, tracking parameters, expired pages or redirect chains, rather than on the pages that earn enquiries. We close those leaks at source so more of a finite budget lands on the URLs you actually want crawled and indexed.
Does log file analysis help with AI crawlers and assistants, not just Google?
Yes, and increasingly it is one of the clearest reasons to do it. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others now request your pages, and their behaviour differs from traditional search bots. Logs are where you see which assistants are reaching your content, how aggressively, and whether robots rules or rate limits are quietly blocking them. If AI visibility matters to you, the logs answer questions nothing else can.
How big does my site need to be for log file analysis to be worthwhile?
It earns its place when the number of URLs runs well ahead of the number that need to rank: large ecommerce catalogues, publishers with deep archives, marketplaces, and anything with heavy faceting or pagination. On a small brochure site, crawl budget is not a real constraint and logs tell you little you could not learn more cheaply, and we will say so rather than sell you the work anyway.
Will log file analysis on its own improve my rankings?
It removes a specific class of blocker: crawl budget wasted on the wrong URLs, important pages crawled too rarely, and errors that erode trust in a section. On a crawl-constrained site, fixing those can help content get discovered and re-assessed faster. But it is foundational work that lets your content and authority register properly, not a substitute for them, and we will not promise rankings that depend on factors outside the technical layer.
What do you actually deliver from a log file analysis engagement?
A crawl budget allocation report, a status-code and error breakdown traced to cause, a list of orphaned and rarely-crawled URLs, a verified summary of genuine bot activity with any blocking flagged, and a prioritised roadmap tying each finding to a specific fix. After the fixes land, we take a fresh log sample and confirm crawler behaviour actually moved the way it was meant to.