GSC Crawl Stats Report: What Googlebot Does on Your Site

Most SEOs Never Open This Report

Google Search Console has a report that tells you exactly how often Googlebot visits your site, which file types it requests, and what HTTP status codes it encounters on every crawl. It's called the Crawl Stats report, and the vast majority of site owners have never opened it.

That's a mistake. When pages take weeks to get indexed, when a site redesign tanks rankings, or when Google keeps crawling the wrong URLs, the Crawl Stats report is often the first place the answer shows up. Default dashboards won't surface these problems. You have to go looking.

This guide walks through every section of the report, what each metric actually means, and the specific patterns that should prompt you to take action.

Where to Find It

In Google Search Console, navigate to Settings in the left sidebar. Under “Crawling,” click “Open report” next to Crawl Stats. The report is property-specific, so you need to be viewing the right site before you navigate there.

The report covers the last 90 days of crawl activity. You can't adjust the date range, which is worth knowing when you're trying to correlate crawl changes with a specific deployment or content update.

The Three Core Charts

The top of the Crawl Stats report shows three line charts: total crawl requests, total download size, and average response time. Each covers the full 90-day window.

Total Crawl Requests

This is the number of URLs Googlebot requested from your server each day. A healthy site shows a relatively stable line with minor fluctuations. Sudden spikes or steep drops are both worth investigating.

A spike often means Googlebot discovered a large batch of new URLs - from a sitemap update, a new site section going live, or an influx of external links pointing to new pages. That's usually fine. A spike followed by a sharp drop can indicate Googlebot hit server errors and backed off.

A sustained drop in crawl requests is more concerning. It typically signals that Google is reducing crawl frequency because it's finding too many errors, too little fresh content, or pages that duplicate what it already has indexed. If total crawl requests trend down over weeks, cross-reference with your Index Coverage report to see if indexing is slowing too.

Total Download Size

This measures how many bytes Googlebot downloads from your server per day. It correlates closely with crawl request volume, but divergences are meaningful.

If crawl requests stay flat but download size jumps, Google is downloading larger files - often because HTML pages have grown heavier, JavaScript bundles are being returned to the crawler, or unoptimized images are slipping into crawlable URLs. If download size drops while crawl count stays constant, pages may be getting lighter or Googlebot is receiving more 304 Not Modified responses, meaning content hasn't changed and the full page body doesn't need to be re-downloaded.

Average Response Time

This is the number Google cares about most in this report. Response time measures how long your server takes to start returning data to Googlebot, not how long the page takes to fully load in a browser. Think of it as server time to first byte.

Google's documentation suggests that response times consistently above 500ms are worth investigating, and anything regularly above 2,000ms can cause Googlebot to slow its crawl rate to avoid overloading your server. Slow response times on a crawl budget perspective are distinct from the user-facing Core Web Vitals metrics, but both ultimately stem from the same infrastructure bottlenecks.

Breaking Down Crawl Requests by Type

Below the summary charts, the report splits crawl requests into categories. This breakdown is where most of the actionable information lives.

By Response

This shows what HTTP status codes Googlebot received. The categories are: Success (2xx), Redirect (3xx), Not Found (4xx), Server Error (5xx), and Other.

A healthy site has the vast majority of requests in the Success bucket. But the ratios matter as much as the absolute numbers. If 15% of all crawl requests return 404s, Google is wasting significant crawl budget on dead URLs. If you're seeing a meaningful percentage of 5xx responses, your server is returning errors to Googlebot - which can cause indexing slowdowns and eventually ranking drops.

Redirects warrant attention too. A moderate number of 301 redirects is normal and fine. But if a large share of crawl requests are redirects, Googlebot is spending time following chains rather than discovering and indexing new content. Redirect chains - where page A redirects to B which redirects to C - multiply this waste. The technical SEO audit process should include mapping and collapsing these chains.

By File Type

This breakdown shows what kinds of files Googlebot is requesting: HTML, JavaScript, CSS, images, and other resource types.

The interesting question here isn't just what file types exist, it's what proportion of your crawl budget they consume. If JavaScript files make up 40% of your crawl requests, Google is spending nearly half its time on your site downloading scripts rather than indexing content pages. CSS files and images rarely affect SEO directly, but they do consume crawl bandwidth.

Sites that rely heavily on client-side rendering sometimes see an unusual JavaScript-to-HTML ratio. That's worth flagging because Googlebot processes JavaScript in a deferred second wave - it renders the HTML first, queues JavaScript execution for later, and may not return to render some pages for days or weeks. If your content lives inside JavaScript components, this lag directly affects how quickly new pages appear in the index.

By Purpose

Google categorizes crawl requests by purpose: discovery (finding new URLs), refresh (revisiting known pages), and sitemap (crawling URLs submitted via sitemap).

The balance between discovery and refresh tells you something about where Google thinks your site is in its crawl cycle. A site with lots of new content publication should see a healthy discovery share. A site where most crawls are refreshes may not be generating enough new signal to prompt Google to explore unknown pages.

If sitemap-driven crawls are proportionally low, either your sitemap isn't submitted, it contains errors, or it's not being processed regularly. All three are fixable.

Crawl Stats vs. Crawl Budget: Understanding the Relationship

Crawl budget is the number of pages Googlebot is willing to crawl on your site within a given timeframe. The Crawl Stats report is essentially a readout of how that budget is being spent.

For most small to mid-size sites under 10,000 pages, crawl budget isn't the constraint. Google crawls all their pages regularly regardless. The report still matters for these sites because it reveals response time issues, error patterns, and redirect waste. But prioritization becomes more urgent at scale.

Large sites - e-commerce catalogs, news publishers, programmatic pages - can genuinely run into crawl budget problems. Googlebot has a finite capacity per site per day. If that capacity gets eaten by paginated archives, URL parameters generating near-duplicate pages, or staging environment content that leaked into the public index, important new pages wait in the queue. Understanding crawl budget in depth helps frame what the Crawl Stats numbers mean for your specific scale.

Common Crawl Patterns and What They Mean

Pattern: Crawl Requests Spike Then Drop

This usually follows a large content deployment or a new sitemap submission. Google detects a batch of new URLs, crawls aggressively, then settles back to a baseline. If the baseline after the spike is higher than before, that's good - Google found the new content valuable. If it drops below the pre-spike level, the new content may have triggered quality signals that reduced overall crawl frequency.

Pattern: Response Time Spikes on Specific Days

Periodic response time spikes - say, every Saturday - often indicate server maintenance windows, scheduled tasks that spike database load, or traffic patterns that compete with Googlebot. Audit your server logs for the same time windows. If a cron job or backup process is running when Googlebot shows slow response times, stagger the schedules.

Pattern: High 404 Rate in Crawl Requests

A persistent high 404 rate almost always traces back to one of three sources: old URLs that were deleted without redirects, links on external pages pointing to URLs you've changed, or internal links that reference non-existent pages. The GSC Links report can help identify external sources. A site crawl with a tool like Screaming Frog catches internal 404 references. Both problems are fixable and worth fixing - they directly waste the crawl visits Google allocates to your site.

Pattern: Growing Download Size Without Growing Crawl Count

Pages are getting heavier. Run a page weight audit on your most-crawled URLs. Unoptimized images served in legacy formats, bloated JavaScript in the HTML head, and inline CSS that could be cached externally are common culprits. Lighter pages load faster for users and download faster for Googlebot - fixing page weight helps both dimensions simultaneously.

Using Crawl Stats Alongside Other GSC Reports

The Crawl Stats report is most useful when read alongside other data. On its own, it tells you what Googlebot did. Combined with the Index Coverage report, it tells you whether that activity produced results. Combined with the URL Inspection tool, it helps diagnose why specific pages aren't being discovered or indexed.

A workflow that works well: check Crawl Stats weekly for anomalies in the three core charts, then drill into the breakdowns when you spot something unusual. If crawl requests are down, check Index Coverage for a parallel drop in indexed pages. If response time is spiking, use server logs to identify the specific URLs causing slowdowns.

For sites that depend heavily on fresh content - news, job listings, product pages - consider tracking organic traffic trends alongside crawl data. If crawl rate drops before a traffic drop appears, you have an early warning that lets you act before rankings are affected.

The site audit tools in platforms like MeasureBoard can surface the technical issues the Crawl Stats report flags - broken redirects, slow server response paths, and crawlable junk - without requiring you to manually cross-reference multiple reports.

What You Can and Can't Control

You can't directly control how often Googlebot visits your site. Google determines crawl frequency based on signals it computes independently: PageRank of your pages, historical server reliability, freshness of content, and site authority in general. What you can control is what Googlebot encounters when it arrives.

Fast server response times, clean HTTP status codes, efficient redirects, and well-structured sitemaps all make each crawl visit more productive. A site that serves 200ms responses with clean 200 status codes will see Googlebot return more often than one that serves 1,500ms responses with 20% error rates - even if both sites publish new content at the same rate.

That's the practical takeaway from the Crawl Stats report. You're not just watching numbers. You're identifying friction points that make Google's job harder, and removing them one by one.