Google has recently seen an increase in websites using 403 and 404 response codes from websites and content delivery networks, also known as CDNs. These networks help to load website files from different locations. While this can help with page speed to limit the crawling of Googlebot, should you really be doing this with your website?
What are response codes?
Response codes are sent from your website’s server to the browser or bot that is accessing your website to signal if content is available or not. For example, if a server sent a 200 response code, this tells the browser or bot that the content is available. Alternatively, if a 404 error is shown, it tells us that the content is no longer available.
What is rate limiting?
Rate limiting is used to slow down a bot or crawler, for example Googlebot, to help stop overloading a website. Therefore, sometimes it’s useful to do this so that your website isn’t affected by bots crawling your website, as you want all the content on your website crawled by bots like Googlebot.
Due to the rise in 403 or 404 response codes for limiting Googlebot, Google has given advice to tell publishers and websites to not use these response codes to do this. Instead, they should use the rate limiting feature available in Google Search Console or use a 429 response code that tells the bot that there have been too many requests sent to the server.
Google has also said that, eventually, they may drop webpages using the 404 or 403 response code, so making sure that your website is not sending these response codes to Googlebot is a good check to do.
If you’re looking to grow your business online through digital marketing such as SEO, Facebook Ads and PPC, then get in touch with our friendly team at Engage Web.
- Key takeaways from Google’s latest search reviews update - November 10, 2023
- Mobile-first indexing: what is it, and why has it taken so long? - November 3, 2023
- Google updates structured data understanding - October 31, 2023
