fbpx
Engage Web logo - horizontal-resized

ENGAGEWEB

404 error

Should you use 403 or 404 errors to limit Googlebot crawling?

404 error

Should you use 403 or 404 errors to limit Googlebot crawling?

Google has recently seen an increase in websites using 403 and 404 response codes from websites and content delivery networks, also known as CDNs. These networks help to load website files from different locations. While this can help with page speed to limit the crawling of Googlebot, should you really be doing this with your website?

What are response codes?

Response codes are sent from your website’s server to the browser or bot that is accessing your website to signal if content is available or not. For example, if a server sent a 200 response code, this tells the browser or bot that the content is available. Alternatively, if a 404 error is shown, it tells us that the content is no longer available.

What is rate limiting?

Rate limiting is used to slow down a bot or crawler, for example Googlebot, to help stop overloading a website. Therefore, sometimes it’s useful to do this so that your website isn’t affected by bots crawling your website, as you want all the content on your website crawled by bots like Googlebot.

Due to the rise in 403 or 404 response codes for limiting Googlebot, Google has given advice to tell publishers and websites to not use these response codes to do this. Instead, they should use the rate limiting feature available in Google Search Console or use a 429 response code that tells the bot that there have been too many requests sent to the server.

Google has also said that, eventually, they may drop webpages using the 404 or 403 response code, so making sure that your website is not sending these response codes to Googlebot is a good check to do.

If you’re looking to grow your business online through digital marketing such as SEO, Facebook Ads and PPC, then get in touch with our friendly team at Engage Web.

Jonathon Roberts

Get in touch

Please enable JavaScript in your browser to complete this form.

What is 7+4?

Acceptance

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

>

Book a consultation with Engage Web

Sorry to interrupt, but would you like to download our FREE Social Media Calendars?

Social Media Calendar Product Mock Up for web

 You can use them to plan your social media and content in advance, saving you time and getting better results. When you use our social media calendars, you'll always know what's trending and what to post about for your business.

FREE DOWNLOAD