« Trying things in Python | Main | The kind spirit of Melksham »
June 19, 2007
What proportion of our web traffic is robots?
We welcome search engines to our site - to index our content and point their visitors back to us where appropriate, but such search engines are a means to an end and not an end in themselves. How much traffic to our web site is true visitor traffic, and how much is automata?
Our Current Visitors page allows us to take a snapshot of the HTML and PHP pages that use our standard template that have been called up, and from where, in the last five minutes. The database records which browser was in use, the referring page and the country of origin and makes a fascinating read.
Our Most Popular Pages display looks back at yesterday's log files and tells us where people have visited - and a bias towards a particular page will tend to indicate heave real traffic as automata tend to spider evenly.
Pie charts for the last 4 weeks give us a variety of other statistics, and we have many additional reports from this source and extra information we can call up (password protected or from our own URLs only) on the earlier reports.
And here is another interesting pointer - a graph of log file size for the last four weeks
We noticed a distinct weekly "cyclic" flow in this graph (it will automatically update as this blog archives, so you may not see it if you come back here in 2008 or later!) and as automata run 24 x 7 for the most part, we're pretty sure that the peaks and troughs are caused by real visitors.
I started with the question "What proportion of our web traffic is robots? when perhaps I should have asked "What proportion of our web traffic is generated by real human beings browsing at the time?". From the current peaks and troughs on the graph, and the other evidence, I'm guestimating that the figure is somewhere between 40% and 65%, with a very high proportion of that being business rather than home users.
Posted by gje at June 19, 2007 06:44 AM