Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
 
For 2021 - online Python 3 training - see ((here)).

Our plans were to retire in summer 2020 and see the world, but Coronavirus has lead us into a lot of lockdown programming in Python 3 and PHP 7.
We can now offer tailored online training - small groups, real tutors - works really well for groups of 4 to 14 delegates. Anywhere in the world; course language English.

Please ask about private 'maintenance' training for Python 2, Tcl, Perl, PHP, Lua, etc.
Server logs - drawing a graph of gathered data

You can log pretty much any server data on a Linux or Unix system at regular intervals using a crontab job that adds a line onto the end of a file. That crontab job could be as simple as a shell command, or it could be a bash or perl script (the two most natural languages to choose for such a task) or something else. As a part of last week's work, I wrote such a logging script (source code [here]) and on yesterday's PHP course, I picked up the data files that I've generated over the last few days and produced this graph of server useage. If I were to show you the raw data file, you would find the pattern that the graph shows almost impossible to spot.

What patterns *can* we spot? Well - there's a distinct event of some sort at around 4 a.m., and there was a bigger one around 5 a.m. one day. I can 'account' for these - we schedule some nightly admin / tidy up work on our server, include a couple of quite slow analyses of our site and logs which result in summary reports of the previous dey being emailed to us. And, twice a week, the system backs itself up; that's the big bump in the yellow curve. Otherwise, I'm going to say "time will tell" with these graphs. I'm tempted to let traffic be logged for several months and add the results up to look for further patterns - we already know from other traffic graphing that the pattern varies significantly based on the day of the week, so I can't just combine all the days I have logged to even out glitches!


When I post this article, the graph shown here will look almost identical to the one above - but as time passes it will vary. For this second display is the "live" one! It's using PHP's gd module - which is shipped with PHP these days, although you need to configure it in. The application behind the graphic looks for the latest five files of log data, and reads them in. Rather than printing out the results of its analysis, it makes calls to functions like ImageFilledRectangle and ImageTTFText and ImageLine which draw into a canvas or pixel map held on the server, and once it's completed that it calls ImageJpeg on that canvas to convert it to a .jpg image which is then sent to the browser for display.

The complete graphic script is available [here]; I cover graphics in PHP very briefly on our PHP Programming course, but if it's something you'll be seriously using you should also look at our PHP Techniques Workshop which is a 2 day intermediate / advanced level course.





Notes:

1. I had a "management choice" as I wrote this example - whether to use a graphing package such as phplot - see [example] or whether to stick with gd. I elected to stick with gd which comes as a standard part of the PHP distribution these days (though it has to be configured in) rather than creating a dependency on an external piece of code if anyone wants to grab a copy of this and try it for themselves.

2. The sample graph I captured at the very top of this article includes the day that the clocks went back ... so there's 25 hours of data showing on one of the lines; there's a potentially long discussion about how the time change should be handled in the application, but in the overall run of things it's not going to make a significant difference, so I have simply let the curve extend. There might also be other glitches if we have to adjust the server clock, or if the server goes down, and we may loose a sample if it's rebooted.

3. Images shown on this page are scaled down. Full size, current, image [here]
(written 2010-11-03, updated 2014-09-29)

 
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
H309 - PHP - Maps, Graphics and Geographics
  [320] Ordnance Survey - using a 'Get a map' - (2005-05-22)
  [563] Merging pictures using PHP and GD - (2006-01-13)
  [665] PHP Image viewing application - (2006-04-01)
  [937] Display an image from a MySQL database in a web page via PHP - (2006-11-22)
  [1104] Drawing dynamic graphs in PHP - (2007-03-09)
  [1194] Drawing hands on a clock face - PHP - (2007-05-19)
  [1389] Controlling and labelling Google maps via PHP - (2007-10-13)
  [1390] Converting from postal address to latitude / longitude - (2007-10-13)
  [1391] Ordnance Survey Grid Reference to Latitude / Longitude - (2007-10-14)
  [1628] Gant charts - drawing them with a PHP script - (2008-05-03)
  [1724] addslashes v mysql_real_escape_string in PHP - (2008-07-27)
  [1734] All around the world? - (2008-08-03)
  [1752] Dynamic maps / geographics in PHP - (2008-08-13)
  [1756] Ever had One of THOSE mornings? - (2008-08-16)
  [1923] Making it all worthwhile - (2008-12-04)
  [1956] Images for Christmas - (2008-12-21)
  [2343] World Flags in your PHP pages - (2009-08-10)
  [2361] Geocoding - converting address to latitude / Longitude with PHP via Google - (2009-08-14)
  [2390] Dynamic / changing images on your web page - (2009-09-01)
  [2583] Reducing image size on digital photos - PHP - (2010-01-17)
  [2675] Redirecting to your main domain for correct security keys - (2010-03-13)
  [2729] Uploading a document or image to its own URL via a browser - (2010-04-18)
  [2939] Protecting your images from use out of context - (2010-08-29)
  [3133] An image from a website that occasionally comes out as hyroglyphics - (2011-01-14)
  [3197] Finding and diverting image requests from rogue domains - (2011-03-08)
  [3211] Computer Graphics in PHP - World (incoming data) to Pixel (screen) conversion - (2011-03-24)
  [3447] Needle in a haystack - finding the web server overload - (2011-09-18)
  [3536] UK Mapping Data - and more to come - under government Open Data measures - (2011-12-03)
  [3584] QR codes - graphics images that provide quick phone links - (2012-01-18)
  [3734] QR codes with marketing logos embedded - (2012-05-16)
  [3817] Fpdf - generating .pdf documents easily from your PHP program - (2012-07-24)
  [4178] Where are you? How to write a geosensitive application - (2013-09-18)
  [4365] The changing face of Christmas - (2014-12-26)
  [4437] Adding a PHP build option, rotating an image based on camera data, and a new look at thumbnails in PHP - (2015-02-22)
  [4655] Image indexer / thumbnail display scripts in PHP - (2016-02-25)

A692 - Web Application Deployment - Monitoring and load testing your server
  [1718] Increasing Java Virtual Machine memory for Tomcat - (2008-07-24)
  [2079] Java - Memory Allocation and garbage collection - (2009-03-14)
  [2080] Using ApacheBench and jconsole to test and monitor Tomcat - (2009-03-14)
  [2082] Jmeter - a first test case - (2009-03-14)
  [2272] Monitoring and loading tools for testing Apache Tomcat - (2009-07-07)
  [3015] Logging the performance of the Apache httpd web server - (2010-10-25)
  [3019] Apache httpd Server Status - monitoring your server - (2010-10-28)

A606 - Web Application Deployment - Apache httpd - log files and log tools
  [376] What brings people to my web site? - (2005-07-13)
  [1237] What proportion of our web traffic is robots? - (2007-06-19)
  [1503] Web page (http) error status 405 - (2008-01-12)
  [1598] Every link has two ends - fixing 404s at the recipient - (2008-04-02)
  [1656] Be careful of misreading server statistics - (2008-05-28)
  [1761] Logging Cookies with the Apache httpd web server - (2008-08-20)
  [1780] Server overloading - turns out to be feof in PHP - (2008-09-01)
  [1796] libwww-perl and Indy Library in your server logs? - (2008-09-13)
  [3087] Making the most of critical emails - reading behind the scene - (2010-12-16)
  [3443] Getting more log information from the Apache http web server - (2011-09-16)
  [3491] Who is knocking at your web site door? Are you well set up to deal with allcomers? - (2011-10-21)
  [3554] Learning more about our web site - and learning how to learn about yours - (2011-12-17)
  [3670] Reading Google Analytics results, based on the relative populations of countries - (2012-03-24)
  [3974] TV show appearance - how does it effect your web site? - (2013-01-13)
  [3984] 20 minutes in to our 15 minutes of fame - (2013-01-20)
  [4307] Identifying and clearing denial of service attacks on your Apache server - (2014-09-27)
  [4404] Which (virtual) host was visited? Tuning Apache log files, and Python analysis - (2015-01-23)
  [4491] Web Server Admin - some of those things that happen, and solutions - (2015-05-10)


Back to
Coding efficiency - do not repeat yourself!
Previous and next
or
Horse's mouth home
Forward to
VAT rise in January - it is really up 14% not just 2.5%
Some other Articles
Get all your ducks in a row ... and they may turn into swans
Liquorice allsorts and Dolly Mixtures
PHP data sources - other web servers, large data flows, and the client (browser)
VAT rise in January - it is really up 14% not just 2.5%
Server logs - drawing a graph of gathered data
Coding efficiency - do not repeat yourself!
Learning to Program ... in PHP. Course examples.
Testing the new pavement ready for Christmas
Autumn walk from Bowerhill
Retaining web site visitors - reducing the one page wonders
4759 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96 at 50 posts per page


This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2021: 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01144 1225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/mouth/3027_Ser ... -data.html • PAGE BUILT: Sun Oct 11 16:07:41 2020 • BUILD SYSTEM: JelliaJamb