One of the sad truths about the internet is that the vast majority of everything on it is spam. This applies to websites, content, emails and even website traffic. It’s this last one that really should concern you because not only does it mean your website isn’t receiving as many real visitors as you perhaps think, but it also means these ‘visitors’ are costing you money by using your bandwidth.
Yes, bandwidth costs money. Bandwidth is the data transfer between your website’s server and the device used to look at the website, and every megabyte of data costs you money (unless you happen to be on a web host that offers unlimited bandwidth, but they’re few and far between these days). If the homepage of your website, complete with images and CSS files, is 1MB, it means that for every single person (or automated robot) that accesses that page, your server will use 1MB of bandwidth. As a result, you could be wasting a lot of money on spam visits.
So how would you know?
The easiest, and nontechnical, way of spotting this would be to look at your Google Analytics and see if you’re getting traffic from known spam referrer websites such as buttons-for-websites.com or semalt.com. These are known as referrer spam and tend to clog up the analytics data of website owners, making it more difficult to get an accurate idea of your website’s traffic. In addition, they’ll be using your bandwidth.
To find out how much bandwidth you’ve used, you really need to use a server stats package, such as AWStats, which most websites on Apache should already have. Log into your AWStats, click on the link for ‘Full list’ under the heading ‘Hosts’ in the ‘Who’ section. This will produce a list of IP addresses that have been accessing your website over the current month. If the top IP addresses are also using a substantial amount of bandwidth each (more than a few MB) then there could well be something wrong. The below screenshot shows one website we analysed recently, where you can see several IP addresses from the same IP range have each used over 600MB. This is more than suspicious.
The top IP address is 220.127.116.11 and, between the five IP addresses, they have used 3GB of bandwidth for the month of December 2015. Is this all wasted bandwidth? How can we tell?
Well, this part is really easy. To find out, we just need to use Google, as Google knows all. By simply Googling for that top IP address, we can see that it shows up on websites such as ipgeek.net and projecthoneypot.org, with mentions of how it has been reported for spam and as a potential hacker IP address. This is not something you want accessing your website and, by blocking the whole IP range, not only can we save ourselves 3GB of bandwidth data per month, but we can also help protect our website from potential hacks.
But how can we block it?
Of course, you can block IP addresses, and whole IP ranges, from accessing your website by editing the .htaccess file, but we’re not going to go down that route as it’s a bit technical. Instead, we want to make this a bit easier for non-geeks. The simple and effective way of doing this is with a free WordPress Plugin called ‘IP Blacklist Cloud’.
As mentioned, IP Blacklist Cloud is free but there is also a paid version that offers additional features. To protect your website from spam visits, reduce your bandwidth usage and protect against hacker IPs, you do only need the free version.
Install this plugin on your website and visit the ‘Add IP to Blacklist’ section. All you need to do is paste the offending IP address into the box and press the button marked ‘Add IP’. That’s it. Simple.
You will no longer receive visits from that IP address. Of course, the offending IP addresses we saw in the screenshot were all very similar, and all started with the same three numbers: 79.171.81. This is called an IP range. It is safe to assume that every IP address in this range is also spam, and a potential threat to our website. There are 256 different IP addresses in this range, and we don’t want to block all of them one at a time, so the ‘IP Blacklist Cloud’ plugin has an option to wipe out an entire range in one simple step.
Click on the ‘IP Range’ link. Here, you can add the whole IP range. The starting IP will be 0, and the ending range will be 255. Therefore, for our IP addresses in the screenshot, the starting IP in the range would be 18.104.22.168 and the ending IP address would be 22.214.171.124. You can see that every IP address in the screenshot falls between these numbers 0-255.
To block this entire range, and stop all of these IP addresses from accessing our website, we need to add 126.96.36.199-188.8.131.52 to the box and click ‘save changes’.
The whole IP range is now blocked and we have saved ourselves 3GB of bandwidth over the next month.
Of course, that’s not all this plugin does. Instead of waiting for you to tell it what to block, this plugin actively tracks other visits to your website and, in particular, failed attempts to login to your website’s admin area.
What, you didn’t know that this happens? Yes, I’m afraid so. Websites receive anything between the odd few and the odd few thousand attempts to access them every day. The bigger the website, the more attempts that are made to hack into it. Be warned – this could be quite scary.
The ‘Failed Login’ section will track any attempts made to access the admin section, and it will track the attempted login details these hackers have tried to use. You will see usernames such as ‘admin’, ‘administrator’ and the usernames of any of the authors you have on the website. You will see passwords such as ‘password’, ‘123’, ‘admin123’ and ‘letmein’.
If any of these are your password, you need to change them.
You are now able to blacklist all of these IP addresses to stop them accessing your website and trying again. Be careful here because if anyone who is supposed to be accessing your website has tried and failed (by getting their own password wrong, for example), they will be on this list and you could end up blacklisting them as well. If this happens, you can always remove them from the blacklist later.
Don’t block yourself
To be on the safe side, there is a ‘Whitelist’ section where you can add safe IP addresses (such as your own) to the ‘Whitelist’ so they can never be blocked from accessing the website.
As a tip, once you have blacklisted all of the IP addresses, you should press the ‘Empty Failed Login Attempts’ button to delete them, as having too many stored will slow down the admin side of your WordPress websites.
Now, all of this is well and good and will help to reduce your bandwidth usage and protect against spam users, but there’s a certain kind of website visitor I would describe as a nuisance that this plugin won’t stop, and nor should it. The nuisance user is the web crawler.
You may be thinking ‘now hold on, don’t we want web crawlers accessing our website?’ Well, yes we do, within reason. We want Googlebot accessing our website, for sure. Its presence is essential to ensure our website is indexed and ranking well within Google. However, there are many, many more web spiders that aren’t quite as useful, nor are they quite as well behaved.
Stop MSNbot from using your bandwidth
For example, one of the most resource hungry, bandwidth consuming web crawlers you’ll ever get on your website is, incredibly, owned by Microsoft. MSNbot not only churns through bandwidth like Eamonn Holmes through cake, but it also doesn’t obey a robots.txt file telling it to leave your site alone.
Google listens to what you put in your robots.txt file (which is why you should be VERY careful when writing one as you can block Google from your website if you’re not careful). MSNbot doesn’t listen to it. It doesn’t care. MSNbot isn’t the only spider with a blatant disregard for your bandwidth either, there are thousands of the things and you can’t handle each one individually.
This is where we need another WordPress Plugin, this time called ‘Wordfence Security‘.
Wordfence Security does a lot of things, such as notify you via email whenever someone logs into the admin of your website, but we’re going to focus on one aspect of its features – throttling and blocking excessive users.
Once you have installed the plugin, go to the options section and scroll down to Firewall Rules. Here, you’ll see an option to set rules to block, or throttle, users such as web crawlers or humans based on their use of the website. The below screenshot shows how I have set this up for one website, with anyone or any web crawlers throttled if they exceed four page views per second. MSNbot is well known for exceeding this on occasion. If anyone exceeds this then they are going to be using a lot of bandwidth, and we don’t want that, so their use of the website is restricted.
You could go a step further and block their access. The block is for a fixed period of time, rather than being permanent, and you can configure the length of the fixed period at the bottom. It could be five minutes, it could be a few days.
You can also take action based on less intensive use, such as one page view every 60 seconds. I wouldn’t advise this though.
By using these two plugins on your WordPress website in the manner described, you will find your bandwidth use considerably reduced. This, in turn, will cut your expenses on your web hosting, as bandwidth costs money.
It should be noted that these two plugins also do a lot of other things, and both have paid versions with additional features. For example, the paid version of Wordfence allows you to completely block entire countries from your website, which is very useful if your website is only aimed at one country (such as the UK or USA) and you’re getting a lot of traffic from China, Russia or Ukraine (where most spam attacks originate).
The free versions are more than worth trying out however, and will be great additions to your website’s overall protection.