How are you spending your ‘crawl budget’?

Matt Jones

‘Crawl budget’ is an intriguing term. It brings to mind pictures of search engine spiders creeping across sites, hoarding little pouches of indexing money. If this image is what’s in your head, great. It’s exactly what you should be thinking about when planning your site’s structure for SEO.

Spiders may not have money pouches, but they do spend when on websites. They spend time, and for crawling and indexing, time is precious. The good news is that with SEO you can influence how search engine spiders spend their budgets on your site.

The concept of a crawl budget relates to the limited time spiders spend on low-ranking pages. The lower the rank, the less time spent. High-ranking pages, being more interesting to the search engines, get more time. SEO experts theorise that each site can expect limited crawl time, and that this time can be redirected with careful structural planning.

The method is fairly simple. If you want to boost a page’s rank, stick it next to one that’s already doing well. Link to it strongly from the home page. The association will get that page more crawl time, which means more attention to the on-page optimisation techniques you’ve applied, and eventually a better ranking.

By directing search engine attention in this way, you’re spending your crawl budget on pages that count. The attention itself won’t be sufficient to boost rankings, of course, and it’s important to have your on-page SEO jobs taken care of beforehand. It’s just a smart way to take advantage of the attention you’re given.