What crawl budget means and why it matters to your site

Crawl budget is the number of pages on your website that a search engine will actually visit and read in a given time period. Search engines do not look at every page on every website every day — they decide which pages are worth their time based on how big your site is, how often you update it, and how important the pages seem to be. If you have 10,000 pages but a search engine only crawls 500 of them each month, those 5,000 uncrawled pages will not show up in search results, no matter how good they are.

Think of crawl budget like a librarian with limited hours. The librarian cannot catalog every book in existence, so they focus on the libraries that get the most visitors and the books that get checked out most often. If your library is small and quiet, the librarian might only visit once a year. If your library is large and busy, they might visit every week. Search engines work the same way — they send out automated visitors called crawlers to read your pages, but they do not send them everywhere equally.

The two biggest search engines that crawl websites are Google and Bing. Google's crawler is called Googlebot. Bing's is called Bingbot. Both have limits on how many pages they will crawl on any single website in a day, and both adjust those limits based on how they think your site performs.

Key Takeaways

  • Search engines crawl only a portion of your website each month, so pages they do not crawl will not appear in search results.
  • Your crawl budget depends on your site's size, how often you update it, and how much traffic it gets — not on how much you pay.
  • Broken links, redirect chains, and duplicate content waste crawl budget by sending crawlers to pages that do not help your rankings.
  • You can see which pages Google has crawled by checking Google Search Console, a free tool that shows you crawl activity on your site.
  • Fixing crawl errors and removing low-value pages frees up budget for search engines to find and rank your best content.

How search engines decide what to crawl on your site

Search engines use two main signals to set your crawl budget: crawl demand and crawl capacity. Crawl demand is how much the search engine thinks your site is worth visiting — based on how popular your pages are, how often you publish new content, and how many other websites link to you. Crawl capacity is the technical limit of how much the search engine can crawl across all websites at once.

A new website with five pages might get crawled once a week. A large news site that publishes hundreds of articles daily might get crawled multiple times per day. A site that has not been updated in two years might get crawled once a month or less. Google and Bing both adjust their crawl frequency based on what they observe — if they crawl your site and find new content, they will come back sooner next time. If they crawl it and find nothing has changed, they will wait longer before returning.

The search engine also looks at your site's structure. If your homepage links to your best pages, crawlers will find those pages quickly. If your best pages are buried deep in your navigation or only linked from old pages, crawlers might not reach them before their budget runs out.

What wastes your crawl budget and how to stop it

Several common problems eat up crawl budget without helping your search rankings. Broken links — links that point to pages that no longer exist — send crawlers to dead ends. Redirect chains — where page A redirects to page B, which redirects to page C — force crawlers to follow multiple hops to reach one page. Duplicate content — the same page accessible from multiple URLs — makes crawlers waste time reading the same thing twice.

Parameter-heavy URLs also drain budget. If your site creates a new URL every time someone filters products by color, size, or price, you might have thousands of nearly identical pages that all need crawling. Search engines will crawl some of them, but many will never be seen.

Soft 404 errors are another hidden drain. These are pages that return a "page not found" error code but still display content — crawlers waste time on pages that should not exist. Crawlers also waste budget on pages that serve no purpose: auto-generated pages, test pages, old staging versions, or pages with almost no text.

To fix these problems, start by checking Google Search Console. Go to the "Coverage" report to see which pages Google has crawled and which it has not. Look for pages marked "Excluded" — these are pages Google found but chose not to index, often because they are duplicates or have no real content. The "Errors" section shows broken pages that wasted crawl budget.

How to check your crawl budget in Google Search Console

Google Search Console is a free tool that shows you exactly how much crawl activity is happening on your site. To use it, you need to own the website and verify that you own it — Google will walk you through this when you sign up at search.google.com/search-console.

Once you are in, look for the "Settings" menu on the left side. Click "Settings," then scroll down to find "Crawl stats." This page shows you how many pages Google crawled each day over the past 90 days, how much data it downloaded, and how long it spent crawling. You will see three numbers: requests per day, kilobytes downloaded per day, and response time in milliseconds.

If your crawl stats are flat or declining while you are adding new content, that is a sign your crawl budget is too low. If your response time is very high — over 1,000 milliseconds — your server might be slow, and Google might reduce crawl budget to avoid overloading it.

The "Coverage" report shows which pages Google found and which it indexed. Pages marked "Valid" were crawled and indexed. Pages marked "Excluded" were found but not indexed — usually because they are duplicates, have no content, or are blocked by your robots.txt file. Pages marked "Error" had problems that prevented crawling.

Ways to improve your crawl budget

The fastest way to improve crawl budget is to remove pages that do not deserve to be crawled. If you have thousands of old blog posts that get no traffic, consider deleting them or combining them into fewer, better pages. If you have test pages, staging versions, or auto-generated pages, block them from crawling using your robots.txt file or by adding a "noindex" tag to their HTML.

Fix broken links and redirect chains. Use a tool like Screaming Frog (which has a free version) to crawl your own site and find broken links. When you find them, either fix the link to point to the correct page or delete the link entirely. If you have redirects, make sure they point directly to the final page — not to another redirect.

Consolidate duplicate content. If the same page is accessible from multiple URLs, pick one as the "canonical" version and tell search engines to ignore the others. You do this by adding a canonical tag to the duplicate pages — it looks like this in the HTML: <link rel="canonical" href="https://yoursite.com/correct-page">. This tells crawlers "this page is a copy; go crawl the original instead."

Make sure your site loads quickly. If your server is slow, Google will crawl fewer pages to avoid overloading it. Use a content delivery network (CDN) like Cloudflare to speed up page delivery. Compress images and remove unnecessary code. A faster site gets a higher crawl budget.

Link to your important pages from your homepage or main navigation. Crawlers start at your homepage and follow links from there. If your best pages are only linked from old pages deep in your site, crawlers might not reach them. Make sure your most important pages are one or two clicks away from your homepage.

The difference between crawl budget and indexing

Crawl budget and indexing are not the same thing. Crawling is when a search engine visits your page and reads it. Indexing is when the search engine decides to add that page to its database so it can show up in search results. A page can be crawled but not indexed — this happens when Google reads the page but decides it is a duplicate, has no value, or should not appear in results.

You can have a high crawl budget but low indexing if many of your pages are duplicates or have no unique content. You can also have a low crawl budget but high indexing if you have few pages and they are all high-quality. The goal is to make sure every page that gets crawled is also worth indexing — that means removing low-value pages and making sure your best pages are straightforward to find.

Frequently Asked Questions

Does crawl budget affect my search rankings?

Indirectly, yes. If search engines cannot crawl your pages because your budget is too low, those pages will not be indexed and will not rank. But crawl budget itself is not a ranking factor — Google does not rank you higher because you have a big budget. The ranking factors are content quality, links, and user experience. Crawl budget just determines whether Google gets to see your content in the first place.

Can I increase my crawl budget by paying Google?

No. Crawl budget is not something you can buy. It is determined by your site's size, update frequency, and popularity. The only way to increase it is to make your site more valuable — publish new content regularly, fix technical problems, and earn links from other websites. Google will naturally crawl you more often if your site is worth crawling.

What should my crawl stats look like in Google Search Console?

There is no single "correct" number — it depends on your site's size and industry. A small business site might see 10 to 50 crawl requests per day. A large news site might see 10,000 or more. The important thing is whether the number is stable, increasing, or decreasing. If it is declining while you are adding new content, that is a sign you have a problem to fix.

If Google does not crawl a page, will it ever rank?

No. A page that is never crawled will never be indexed, and a page that is never indexed will never rank in search results. This is why checking your crawl stats and coverage report matters — if important pages are not being crawled, you need to fix the problems preventing crawlers from reaching them.

How often should I check my crawl budget?

Check Google Search Console at least once a month to make sure crawl activity is stable or increasing. If you make major changes to your site — like deleting pages, fixing redirects, or publishing a lot of new content — check again after a few weeks to see if crawl stats have changed. Most sites do not need to monitor it daily.