Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
 
Python, Lua, Tcl, C and C++ training - public course schedule [here]
Private courses on your site - see [here]
Please ask about maintenance training for Perl, PHP, Java, Ruby, MySQL and Linux / Tomcat systems
 
Server logs - drawing a graph of gathered data

Server Logging Graph - last 5 daysYou can log pretty much any server data on a Linux or Unix system at regular intervals using a crontab job that adds a line onto the end of a file. That crontab job could be as simple as a shell command, or it could be a bash or perl script (the two most natural languages to choose for such a task) or something else. As a part of last week's work, I wrote such a logging script (source code [here]) and on yesterday's PHP course, I picked up the data files that I've generated over the last few days and produced this graph of server useage. If I were to show you the raw data file, you would find the pattern that the graph shows almost impossible to spot.

What patterns *can* we spot? Well - there's a distinct event of some sort at around 4 a.m., and there was a bigger one around 5 a.m. one day. I can 'account' for these - we schedule some nightly admin / tidy up work on our server, include a couple of quite slow analyses of our site and logs which result in summary reports of the previous dey being emailed to us. And, twice a week, the system backs itself up; that's the big bump in the yellow curve. Otherwise, I'm going to say "time will tell" with these graphs. I'm tempted to let traffic be logged for several months and add the results up to look for further patterns - we already know from other traffic graphing that the pattern varies significantly based on the day of the week, so I can't just combine all the days I have logged to even out glitches!


When I post this article, the graph shown here will look almost identical to the one above - but as time passes it will vary. For this second display is the "live" one! It's using PHP's gd module - which is shipped with PHP these days, although you need to configure it in. The application behind the graphic looks for the latest five files of log data, and reads them in. Rather than printing out the results of its analysis, it makes calls to functions like ImageFilledRectangle and ImageTTFText and ImageLine which draw into a canvas or pixel map held on the server, and once it's completed that it calls ImageJpeg on that canvas to convert it to a .jpg image which is then sent to the browser for display.

The complete graphic script is available [here]; I cover graphics in PHP very briefly on our PHP Programming course, but if it's something you'll be seriously using you should also look at our PHP Techniques Workshop which is a 2 day intermediate / advanced level course.





Notes:

1. I had a "management choice" as I wrote this example - whether to use a graphing package such as phplot - see [example] or whether to stick with gd. I elected to stick with gd which comes as a standard part of the PHP distribution these days (though it has to be configured in) rather than creating a dependency on an external piece of code if anyone wants to grab a copy of this and try it for themselves.

2. The sample graph I captured at the very top of this article includes the day that the clocks went back ... so there's 25 hours of data showing on one of the lines; there's a potentially long discussion about how the time change should be handled in the application, but in the overall run of things it's not going to make a significant difference, so I have simply let the curve extend. There might also be other glitches if we have to adjust the server clock, or if the server goes down, and we may loose a sample if it's rebooted.

3. Images shown on this page are scaled down. Full size, current, image [here]
(written 2010-11-03, updated 2014-09-29)

 
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
A606 - Web Application Deployment - Apache httpd - log files and log tools
  [4491] Web Server Admin - some of those things that happen, and solutions - (2015-05-10)
  [4404] Which (virtual) host was visited? Tuning Apache log files, and Python analysis - (2015-01-23)
  [4307] Identifying and clearing denial of service attacks on your Apache server - (2014-09-27)
  [3984] 20 minutes in to our 15 minutes of fame - (2013-01-20)
  [3974] TV show appearance - how does it effect your web site? - (2013-01-13)
  [3670] Reading Google Analytics results, based on the relative populations of countries - (2012-03-24)
  [3554] Learning more about our web site - and learning how to learn about yours - (2011-12-17)
  [3491] Who is knocking at your web site door? Are you well set up to deal with allcomers? - (2011-10-21)
  [3447] Needle in a haystack - finding the web server overload - (2011-09-18)
  [3443] Getting more log information from the Apache http web server - (2011-09-16)
  [3087] Making the most of critical emails - reading behind the scene - (2010-12-16)
  [3019] Apache httpd Server Status - monitoring your server - (2010-10-28)
  [3015] Logging the performance of the Apache httpd web server - (2010-10-25)
  [1796] libwww-perl and Indy Library in your server logs? - (2008-09-13)
  [1780] Server overloading - turns out to be feof in PHP - (2008-09-01)
  [1761] Logging Cookies with the Apache httpd web server - (2008-08-20)
  [1656] Be careful of misreading server statistics - (2008-05-28)
  [1598] Every link has two ends - fixing 404s at the recipient - (2008-04-02)
  [1503] Web page (http) error status 405 - (2008-01-12)
  [1237] What proportion of our web traffic is robots? - (2007-06-19)
  [376] What brings people to my web site? - (2005-07-13)

A692 - Web Application Deployment - Monitoring and load testing your server
  [2272] Monitoring and loading tools for testing Apache Tomcat - (2009-07-07)
  [2082] Jmeter - a first test case - (2009-03-14)
  [2080] Using ApacheBench and jconsole to test and monitor Tomcat - (2009-03-14)
  [2079] Java - Memory Allocation and garbage collection - (2009-03-14)
  [1718] Increasing Java Virtual Machine memory for Tomcat - (2008-07-24)

H309 - PHP - Maps, Graphics and Geographics
  [4655] Image indexer / thumbnail display scripts in PHP - (2016-02-25)
  [4437] Adding a PHP build option, rotating an image based on camera data, and a new look at thumbnails in PHP - (2015-02-22)
  [4365] The changing face of Christmas - (2014-12-26)
  [4178] Where are you? How to write a geosensitive application - (2013-09-18)
  [3817] Fpdf - generating .pdf documents easily from your PHP program - (2012-07-24)
  [3734] QR codes with marketing logos embedded - (2012-05-16)
  [3584] QR codes - graphics images that provide quick phone links - (2012-01-18)
  [3536] UK Mapping Data - and more to come - under government Open Data measures - (2011-12-03)
  [3211] Computer Graphics in PHP - World (incoming data) to Pixel (screen) conversion - (2011-03-24)
  [3197] Finding and diverting image requests from rogue domains - (2011-03-08)
  [3133] An image from a website that occasionally comes out as hyroglyphics - (2011-01-14)
  [2939] Protecting your images from use out of context - (2010-08-29)
  [2729] Uploading a document or image to its own URL via a browser - (2010-04-18)
  [2675] Redirecting to your main domain for correct security keys - (2010-03-13)
  [2583] Reducing image size on digital photos - PHP - (2010-01-17)
  [2390] Dynamic / changing images on your web page - (2009-09-01)
  [2361] Geocoding - converting address to latitude / Longitude with PHP via Google - (2009-08-14)
  [2343] World Flags in your PHP pages - (2009-08-10)
  [1956] Images for Christmas - (2008-12-21)
  [1923] Making it all worthwhile - (2008-12-04)
  [1756] Ever had One of THOSE mornings? - (2008-08-16)
  [1752] Dynamic maps / geographics in PHP - (2008-08-13)
  [1734] All around the world? - (2008-08-03)
  [1724] addslashes v mysql_real_escape_string in PHP - (2008-07-27)
  [1628] Gant charts - drawing them with a PHP script - (2008-05-03)
  [1391] Ordnance Survey Grid Reference to Latitude / Longitude - (2007-10-14)
  [1390] Converting from postal address to latitude / longitude - (2007-10-13)
  [1389] Controlling and labelling Google maps via PHP - (2007-10-13)
  [1194] Drawing hands on a clock face - PHP - (2007-05-19)
  [1104] Drawing dynamic graphs in PHP - (2007-03-09)
  [937] Display an image from a MySQL database in a web page via PHP - (2006-11-22)
  [665] PHP Image viewing application - (2006-04-01)
  [563] Merging pictures using PHP and GD - (2006-01-13)
  [320] Ordnance Survey - using a 'Get a map' - (2005-05-22)


Back to
Coding efficiency - do not repeat yourself!
Previous and next
or
Horse's mouth home
Forward to
VAT rise in January - it is really up 14% not just 2.5%
Some other Articles
Get all your ducks in a row ... and they may turn into swans
Liquorice allsorts and Dolly Mixtures
PHP data sources - other web servers, large data flows, and the client (browser)
VAT rise in January - it is really up 14% not just 2.5%
Server logs - drawing a graph of gathered data
Coding efficiency - do not repeat yourself!
Learning to Program ... in PHP. Course examples.
Testing the new pavement ready for Christmas
Autumn walk from Bowerhill
Retaining web site visitors - reducing the one page wonders
4755 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96 at 50 posts per page


This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2017: 404 The Spa • Melksham, Wiltshire • United Kingdom • SN12 6QL
PH: 01144 1225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/mouth/3027_Ser ... -data.html • PAGE BUILT: Sat May 27 16:49:10 2017 • BUILD SYSTEM: WomanWithCat