Crawl budget
Crawl budget is how many URLs Googlebot wants and is able to crawl on your site within a given period, depending on server capacity and URL value.
Crawl budget has two components: crawl capacity (how many requests your server can handle without slowing) and crawl demand (how much Google wants to visit your URLs). For small sites it's rarely an issue; for large e-commerce, news and classifieds it's a critical optimisation lever.
Example
A shop has 200,000 URLs of which 50,000 come from faceted navigation. Google crawls there for days while important product pages are skipped. Solution: exclude facet parameters via canonical or robots.txt.
Frequently asked questions
How do I see my crawl budget usage?
In Search Console > Settings > Crawl Stats: requests per day, response codes, response times and hosts. Log-file analysis gives even deeper insight.
Related terms
Further reading
- → Our service: SEO