Here at GreenAsh, we consider it a top priority to monitor the number of visitors that we get on each page. We do this so that we can gauge what the most visited pages on the site are, and display these top pages in a list, so that you - our visitors - can quickly and conveniently find our most popular content.
At 12:30pm today, we took the drastic step of resetting all page counters to 1. We did this because the figures that were being recorded by the counters were way out of proportion to the actual number of people that were visiting each page. In some cases, pages had received less than 10% of the visits that the counters claimed they had received. But unlike humans, computers do not lie when they report figures. So how did the counter figures become so massively disproportionate?
The answer lies with search engine bots. This is the name given to computer systems that are programmed to "surf the web", much like humans do: except that unlike us, these systems do it for the sole purpose of indexing every page that they can find, in order to make more and more of the web accessible to search engines. This article on web crawlers explains the phenomenon in greater detail. Googlebot is the most active of all search engine bots: it made over 2,000 visits to pages on this site (and hence caused over 2,000 misleading hit counter increments) in the past month alone.
The existence of search engine bots is not a bad thing: in fact, quite the opposite! Every time Googlebot visits a page on the Internet, it indexes that page so that it is accessible to someone who searches for it using Google. So we are more than happy for these bots to make thousands of visits each month. It's just that as a byproduct of all this indexing, the bots inflate our site's statistics, flood our access logs, and (to some extent) slow down performance for everyone else.
That's why we've introduced a new system, which will allow the bots to continue indexing our site, but also to keep out of our statistical info. The bots can now visit GreenAsh pages, just as they always could, only now we have a list of the most common bots stored in our system. Every time a page is visited, our system will check to see if the visit was made by a bot - and if it was, then that visit will not cause our hit counters to increment. So hopefully, from now on you should all be seeing much more accurate figures when you view content on GreenAsh.