Crawl Budget
The number of pages a search engine will crawl on your site within a given timeframe.
Definition
Crawl budget refers to the number of pages search engine bots will crawl and index on your website during a specific period. It is determined by factors like site health, server response time, and overall site authority. Larger sites need to optimize crawl budget to ensure important pages get indexed.
Why It Matters
For large websites with thousands of pages, crawl budget optimization ensures search engines prioritize crawling your most important content. Wasting crawl budget on duplicate, thin, or low-value pages means your key landing pages may not get indexed or updated as frequently.
Examples in Practice
An e-commerce site with 50,000 products uses robots.txt to block faceted navigation pages that create duplicate content. A news site ensures new articles are discoverable through XML sitemaps and internal links to maximize fresh content indexing.