Crawl budget is the set of URLs on your site that a search engine like Google is able to crawl and wants to crawl. It is not a measure of ranking or quality, and it differs from indexation, which decides whether a crawled page is stored and eligible to appear in results.
For marketers running large or fast-changing sites, crawl budget decides how quickly new and updated pages get discovered. When it runs short, your freshest content sits uncrawled for days, invisible to search and to the AI answer engines that pull from the search index.
Crawl budget combines two forces that Google balances for every site: the crawl capacity limit and crawl demand. The capacity limit is how many parallel connections your server can handle and how long Google holds them open. Crawl demand is how much Google wants to crawl your URLs, based on their popularity and how stale its copies are.
Every site starts with the same conservative default capacity, which rises only when Google has more demand and your server stays healthy, and falls when Google sees latency, 5xx errors, or 429 responses. That capacity is shared across all Google crawlers, so heavy fetching for one purpose reduces what is available for others. Perceived inventory is the demand factor you control most: the more URLs Google thinks exist, including duplicates and URL parameters, the more crawl it may spread thin.
Crawl budget sits upstream of indexation and ranking. A page that never gets crawled cannot be indexed, and a page that is not indexed cannot be retrieved or cited by AI answer engines that read the search index. AirOps helps teams keep the pages that matter crawlable and current so discovery work isn't wasted.
Google decides your crawl budget continuously and updates it as your server health and content signals change. It runs as an ongoing loop, so today's crawl rate reflects recent server performance.
Set capacity: Google establishes a crawl capacity limit from how fast and reliably your server responds. Quick, error-free responses raise it; timeouts and errors lower it.
Gauge demand: Google estimates crawl demand from how many URLs it believes you have, how popular they are, and how stale its stored copies have grown.
Build the queue: Google combines capacity and demand into a prioritized list of URLs it can and wants to fetch next.
Crawl and adjust: Googlebot fetches URLs from the queue and watches response times and status codes, throttling back the moment your server strains.
Recrawl by signal: Pages that change often or carry fresh lastmod dates in your sitemap re-enter the queue sooner.
Your server logs and Search Console crawl stats show how much Google crawled and where it spent effort. They do not tell you whether a crawled page was judged good enough to index, which is a separate decision.
Whether crawl budget deserves your attention depends on your site's size and how often it changes. Google's guidance says crawl budget management is mostly relevant for large sites with 1 million or more unique pages whose content changes about weekly. It also applies to medium and larger sites with 10,000 or more unique pages that change daily. Smaller, stable sites are usually crawled efficiently without any intervention.
Faster discovery of new pages: On large sites, efficient crawling is what gets new product or content pages found in hours instead of weeks, which shortens time to organic and AI-answer visibility.
Wasted crawling starves priority pages: When Googlebot burns its budget on duplicate URLs and low-value filter parameters, your revenue pages can go days without a recrawl and miss important updates.
Upstream of AI answer visibility: Answer engines that read Google's index can only cite pages that were crawled and indexed, so crawl efficiency sets a ceiling on how often your content can appear in AI answers.
SEO managers use crawl budget analysis to find where Googlebot wastes fetches on parameter URLs and redirect chains, then reclaim that capacity for revenue pages.
Content strategists use crawl budget signals to decide which stale pages to consolidate or prune so the crawler reaches priority content faster.
Growth marketers use crawl budget monitoring to confirm that newly launched landing pages get crawled and indexed before a campaign drives traffic to them.
The maximum amount of fetching Google will do on your site before it risks degrading performance for real users, set by how quickly and reliably your server responds and recalculated continuously as that health changes.
How much Google wants to crawl your URLs at any moment, driven by how popular those pages are, how stale Google's stored copies have grown, and how large it perceives your total URL inventory to be.
The share of your crawl activity that reaches indexable, valuable URLs instead of being spent on duplicates, parameter variations, soft 404s, and dead redirect chains that signal to Google that much of your site is low value.
Speed up discovery of new and updated pages so they become eligible to rank and be cited sooner.
Reclaim wasted crawls from duplicate and parameter URLs and redirect them toward revenue pages.
Reduce server load from Googlebot by returning clean status codes and supporting HTTP 304.
Improve indexation coverage on large sites where thousands of pages compete for limited crawl attention.
Track crawl activity in Google Search Console to see exactly which URLs Googlebot fetches and how often.
Consolidate duplicate content behind canonical URLs so Google spends crawls on one version instead of many.
Block genuinely useless URLs in robots.txt, such as infinite filter combinations, so the crawler skips them entirely.
Return 404 or 410 for removed pages and fix soft 404s, because false success codes keep Google recrawling dead URLs.
Keep your XML sitemaps current with accurate lastmod dates, which tells Google which pages changed and deserve a recrawl.
Speed up your pages and server responses, since faster responses raise the crawl capacity limit Google assigns you.
Support HTTP 304 responses for unchanged pages so Google saves budget by skipping content it already has.
Avoid using noindex tags to save crawl budget. Google still has to crawl a page to see the noindex directive, so it spends the budget anyway. For URLs you genuinely never want fetched, block them in robots.txt instead, which keeps Google from spending crawl budget on them at all and significantly lowers the chance they get processed by Google's other systems.
AirOps: Builds and refreshes the priority pages on your site and keeps them current, so the crawl budget you have lands on content worth indexing and citing.
Google Search Console: Reports crawl stats, showing how many requests Googlebot made, average response times, and which URLs and status codes it hit.
Screaming Frog: Crawls your site the way a search engine would, exposing duplicate URLs, redirect chains, and parameter bloat that drain crawl budget.
Pull your crawl stats: Open the Crawl Stats report in Google Search Console and note total crawl requests, average response time, and any spikes in 5xx or 404 responses. This takes an afternoon and needs no budget approval.
Run a full site crawl: Use a crawler like Screaming Frog to map every URL a search engine can reach, flagging duplicates, redirect chains, and parameter URLs.
Classify your URLs: Sort the crawl into pages you want indexed and pages you do not, so you know where crawl budget should and should not go.
Fix the biggest leaks: Canonicalize duplicates, block useless parameters in robots.txt, and return proper 404 or 410 codes for the low-value URLs you found, starting with the ones that appear most often in your crawl stats.
Recheck and monitor: After a few weeks, compare crawl stats again to confirm Googlebot is fetching your priority pages more and your dead URLs less.
Crawl budget sets how many of your site's URLs a search engine will fetch, capacity and demand together.
You measure it through server logs and the Crawl Stats report in Google Search Console.
Server speed and reliability cap how much Google will crawl, because it slows down the moment your site strains.
The biggest risk is letting duplicate and low-value URLs absorb crawls that your revenue pages need.
The leverage is in trimming waste and speeding up your server, which frees budget for the pages you want found.
Crawl budget decides whether and how often a page gets fetched, while indexation decides whether a fetched page is stored and made eligible to appear in results. The two are sequential steps in the same pipeline. Google first has to spend crawl budget to fetch a URL. Only then does it evaluate that page for indexing, checking quality, duplication, and whether the content deserves a place in the index. A page can be crawled repeatedly and still never be indexed if Google judges it thin or duplicative. A page can also be indexed once and then rarely recrawled if it seldom changes. Confusing the two leads teams to blame crawl budget for what is really an indexing problem. If your pages are getting crawled but not indexed, the problem is usually content quality or duplication, and crawl budget is not the lever to pull. If they are not getting crawled at all, then crawl budget and site structure are worth investigating.
Check your crawl budget on a cadence that matches your site's size and change rate. For a large or fast-changing site, review the Crawl Stats report in Google Search Console monthly, and watch it closely around major launches, migrations, or redesigns when crawl patterns shift most. For a smaller, stable site, a quarterly glance is usually enough, since Google tends to crawl these efficiently on its own. Beyond the calendar, treat certain events as triggers: a spike in server errors, a jump in response time, or a sudden drop in pages crawled per day all warrant an immediate look. After you make structural fixes like canonicalization or robots.txt changes, give Google a few weeks to adjust, then compare crawl stats before and after. You are watching for the moments when crawl efficiency changes, so you can act before uncrawled pages pile up and slow your discovery.
Your crawl budget changes because both of its inputs, crawl capacity and crawl demand, move constantly. Capacity rises when your server responds quickly and reliably and falls when Google encounters slow responses, timeouts, or server errors, so a hosting problem or a traffic spike can shrink your budget within days. Demand shifts as your pages gain or lose popularity and as Google's stored copies age, so a burst of links or a wave of fresh content can pull more crawling toward your site. Seasonal patterns matter too, since Google may crawl a retail catalog harder before a shopping season. Because capacity is shared across all Google crawlers, heavy fetching for images or a large migration can temporarily reduce what is available for your standard pages. None of this is fixed, which is why two snapshots of your crawl stats taken months apart can look very different even when you changed nothing on purpose.
You cannot set your crawl budget directly, but you can influence both of the factors Google uses to calculate it. Google does not give you a direct setting for crawl budget, and it decides the number on its own. What you control is the input side. On capacity, faster and more reliable server responses let Google raise the ceiling it is willing to crawl, so performance work and clean status codes pay off. On demand, you shape what Google wants to crawl by consolidating duplicates, keeping sitemaps accurate with real lastmod dates, and earning links that raise a page's popularity. You also control waste, which is often the fastest win: blocking useless parameter URLs and fixing soft 404s stops Google from spending budget on pages that do not matter. So the honest answer is that you steer crawl budget indirectly, through server health and site hygiene, even though you cannot type in a bigger number.
There is no universal number for a good crawl budget, so judge it by efficiency instead of raw volume. A healthy pattern looks like this: Googlebot spends most of its requests on URLs you want indexed, average response times stay low and stable, and server-error rates stay near zero. Watch the ratio of crawled pages that are indexable versus those that are duplicates, redirects, or error pages; the higher the share of valuable URLs, the better your budget is working. Google's own guidance says smaller, stable sites rarely need to think about crawl budget at all, so if your site is modest and your important pages get indexed within a few days, your budget is effectively fine. For larger sites, a good sign is that new and updated priority pages get crawled within hours to a couple of days, and that no important section goes weeks without a visit. Compare your own crawl stats month over month, since your best benchmark is your own trend line.