Even Google Has a (Crawl) Budget

Search engines detect content on the web by using spiders or bots to “crawl” pages and links. Crawls can change in depth and frequency, and getting your site crawled frequently means that the engines can more quickly detect updates, improvements, and changes. Earlier this year, Google revealed details of its crawl budget for websites. But what is a crawl budget and what does it mean for site owners?

What is a Crawl Budget?

Similar to a spending budget, which determines how much you can spend, Google’s crawl budget determines how many URLs Googlebot will crawl.

According to Google, for most sites (those with fewer than several thousand URLs), crawl budget isn’t an issue.

But bigger site owners should consider two factors: crawl demand and crawl rate limit.

Crawl Demand

Crawl demand is based on supply and demand. Google wants to supply searchers with the popular and relevant URLs they’re looking for. So pages that gain more traffic are crawled more often to keep up with current search trends. Without demand for your URLs in the index, crawl activity from Googlebot will be low.

Crawl Rate Limit

We all know that user experience is hugely important to Google, so Googlebot wants to crawl sites without negatively impacting visitors. That’s where crawl rate limit comes in. This limit is the maximum rate at which Google can fetch URLs without overwhelming your server. Wait time between crawls adjusts based on your site’s health. A healthy site, one with quick load times and few site errors, produces a healthy crawl and will be crawled more frequently.

An Efficient Crawl

The more frequently your site URLs are crawled, the more chances you have to appear in search results. That makes it important to diagnose issues and improve site health with an audit.

Many factors contribute to how thoroughly and how often a site is crawled. In short, you want to do your best not to exhaust Googlebot. Below are a few things that negatively and positively impact crawl efficiency.

Things that Negatively Impact Crawl Efficiency:

  • Slow page-load times
  • Long redirect sequences
  • 404 page errors
  • Duplicate content
  • Thin or low-quality content
  • Slow page-load times

Things that Positively Impact Crawl Efficiency:

  • Meta tags
  • Structured data
  • Accelerated mobile pages

Stay Calm and Crawl Strong

All and all, understanding Google’s crawl budget restrictions is important to understanding how the engine crawls and indexes your site. Keep your site healthy, user friendly, and running well, and Google will keep crawling it as often as it can!