Crawl budget
How many URLs a search engine is able and willing to crawl on a site in a given period.
Crawl budget combines how fast a server can be crawled without strain and how much the engine wants to crawl it. Google says it is a concern mainly for very large sites or those that generate many URLs, such as faceted navigation.
Wasted crawling on duplicate, parameterised or redirecting URLs is the usual problem; server errors and slow responses reduce how much gets crawled.
Crawling & Indexing timeline · Latest crawling & indexing news · What Google said
Latest headlines mentioning it
- Googlebot Must Follow Redirects From Homepage To Favicon, Updated Doc Say
- How to find and fix crawl budget waste with log files
- How to find and fix crawl budget waste with log files
- Crawl Budget: Why It Matters on Larger Sites
- Are websites getting "fat"? Page weight, HTML size & Googlebot limits explained - transcript