Every time you perform a Google search, the site looks through billions of pages for the optimum result in a less than a second. Its Googlebots crawl through every nook and cranny of the web, working hard to suck up all the information they need to bring you the best possible answer to your search, but what can’t they get hold of?
According to Google’s John Mueller, very little. If you move to around the 15:40 mark of the below video, you can see him taking part in a Google Hangout and responding to a question about whether Google sometimes fails to hit the mark when it comes to e-retailers, perhaps missing certain categories and subcategories due to the site’s internal linking or lack of landing pages.
Mueller says that he very rarely sees sites where he thinks the internal linking is such a problem that Google can’t find the content on it. He goes on to say that for most normally set up sites, built through a modern content management system (CMS), content will be found by Google and returned in its results.
This suggests that perhaps the main cause of being ignored by Google is using out of date tools to put your website together, and setting out content poorly so that Google doesn’t realise it’s a valid result for what the user is searching for.
An example of this could be something like a search for ‘white Nike trainers size 11. There will be plenty of sites selling exactly the products the searcher is looking for, but some will index it better than others. Some sites might cobble (pardon the pun) all their shoes together, or bracket them only by brand or style. White Nike trainers in size 11 will be there somewhere, but it can’t be instantly found due to a lack of landing pages, categories, or even simple formatting like header tags that help Google know what to prioritise. It’s likely Google will ‘find’ it – it will be somewhere among the results, but it won’t be given much prominence.
Other content Google can’t locate is that from the deep web, such as restricted sites, forums that require membership, and the likes of your own emails – thankfully!
An interesting question put forward in the comments section of an article about Mueller’s statement on SERoundTable.com is whether Google can really be sure that it isn’t missing much. After all, how can it make an accurate assessment of how much it isn’t finding? You may not see any spiders in your house for a month, but can you confidently assert that your house contains no spiders?
We have little option but to trust Google’s judgement, but as Mueller points out, there is plenty we can do to make our content more Googleable and stop it being one of those spiders lurking in your house that’s always there, but never seen.