Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
Perl - still a very effective language indeed for extracting and reporting

Perl remains a marvellous programming language for a number of applications, including quick / one off scripts which may be needed to manipulate data is ways that vary from day to day and week to week - research type work, if you like.

We manage / look after our own web server, hosting our IT training business, our hotel the Great Western Coffee Shop Forum and a whole series of other lower volume sites such as Melksham Chamber of Commerce and Industry and the TransWilts Community Rail Partnership. Various monitoring and logging scripts run on the system - from an external heartbeat which sends us an email if the server's not running within predefined metrics (or not running at all - this is an external script) through a script which takes a snapshot of vital parameters every few minutes to our web server logs which record details of each and every completed web server access.

With all this data ... when we want to find what's the reason behind a changing pattern, we're looking for a needle in a haystack - over 200,000 access records / over 50 Mbytes of logging datain some 24 hour periods. We need the "Practcial Extraction and Reporting Language".

Here are a couple of examples of Perl code that I've used so far this month, during which time we've had a couple of pattern changes on the server that I wished to investigate:

  while (<>) {
    ($h,$m,$s) = /:(\d+):(\d+):(\d+)/;
    $sid = $h * 3600 + $m * 60 + $s;
    print if ($sid - $oldsid < -450) ;
    $oldsid = $oldsid > $sid ? $oldsid : $sid;
    }


This identify slow running requests to the server - ones that take over 450 seconds to complete; this isn't quite as simple as it sounds, as only the start time of each request is recorded, but the recording is in the order of completion so the Perl job is looking back at previous record times ...

  while (<>) {
    @n = split;
    $c{$n[0]}++;
    }
  for $k (sort {$c{$a} - $c{$b}} keys %c) {
    printf "%6d %s\n",$c{$k},$k;
    }


And that piece of code counts accesses from each different visitor to the server, printing them out with the most frequent visitor listed last.

Each of these scripts was a vital step in identifying issues that were effecting server performance - one of our housekeeping scripts was being run rather more often than it should have been by visiting search engine crawlers (so we modified that script to ensure it's not run more than once every 24 hours), and we were getting up to 300 visits in 5 seconds from a system in the Ukraine; I don't know exactly what it was, but it was up to no good with accesses from the same computer claiming to be from Chrome, Firefox and Internet Explorer browsers ... when I suspect it was an automaton that was either our of control or malicious. A quick fix there is to have our web server simply send back a "forbidden" (403 code) response which it can do efficiently without any noticable effect on performance!

I have just given you the bare bones scripts above. We strongly recommend that it you're using things like this on a regular basis, you comment your scripts well, you use better variable names, you add flexibility by allowing different variables to be fed in, and you refrain from using constant values where you might want a change in the future. Such techniques are taught on both our Learning to program in Perl and Perl Programming courses, which run a number of times each year at our Melksham, Wiltshire training centre or can be run in private if you've got a group of delegates.
(written 2014-09-20, updated 2014-09-21)

 
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
P050 - Perl - General
  [4296] Polishing the Perl courses - updated training - (2014-09-17)
  [3911] How well do you know Perl and / or Python? - (2012-11-04)
  [3902] Shell - Grep - Sed - Awk - Perl - Python - which to use when? - (2012-10-22)
  [3823] Know Python or PHP? Want to learn Perl too? - (2012-07-31)
  [3407] Perl - a quick reminder and revision. Test yourself! - (2011-08-26)
  [3332] DNA to Amino Acid - a sample Perl script - (2011-06-24)
  [3322] How much has Perl (and other languages) changed? - (2011-06-10)
  [3093] How many toilet rolls - hotel inventory and useage - (2010-12-18)
  [2971] Should the public sector compete with businesses? and other deep questions - (2010-09-26)
  [2825] Perl course - is it tailored to Linux, or Microsoft Windows? - (2010-06-25)
  [2783] The Perl Survey - (2010-05-27)
  [2736] Perl Course FAQ - (2010-04-23)
  [2504] Learning to program in ... - (2009-11-15)
  [2374] Lead characters on Perl variable names - (2009-08-24)
  [2242] So what is this thing called Perl that I keep harping on about? - (2009-06-15)
  [2228] Where do I start when writing a program? - (2009-06-11)
  [1897] Keeping on an even keel - (2008-11-21)
  [1750] Glorious (?) 12th August - what a Pe(a)rl! - (2008-08-12)
  [743] How to debug a Perl program - (2006-06-04)
  [400] New in the shops - (2005-08-01)
  [116] The next generation of programmer - (2004-11-13)

P210 - Perl - Topicalization and Special Variables
  [4700] Obfurscated code - it might work, but is it maintainable? - (2016-07-02)
  [4682] One line scripts - Awk, Perl and Ruby - (2016-05-20)
  [4395] Preparing data through a little bit of Perl - (2015-01-15)
  [3449] Apache Internal Dummy Connection - what is it and what should I do with it? - (2011-09-19)
  [2972] Some more advanced Perl examples from a recent course - (2010-09-27)
  [2876] Different perl examples - some corners I rarely explore - (2010-07-18)
  [2833] Fresh Perl Teaching Examples - part 2 of 3 - (2010-06-27)
  [1922] Flurinci knows Raby Lae PHP and Jeve - (2008-12-04)
  [1860] Seven new intermediate Perl examples - (2008-10-30)
  [1829] Dont bother to write a Perl program - (2008-10-10)
  [1728] A short Perl example - (2008-07-30)
  [1705] Environment variables in Perl / use Env - (2008-07-11)
  [1704] Finding operating system settings in Perl - (2008-07-10)
  [1508] How not to write Perl? - (2008-01-15)
  [1444] Using English can slow you right down! - (2007-11-25)
  [1289] Pure Perl - (2007-08-03)
  [1232] Bathtub example - (2007-06-14)
  [1221] Bathtubs and pecking birds - (2007-06-07)
  [1136] Buffering output - why it is done and issues raised in Tcl, Perl, Python and PHP - (2007-04-06)
  [969] Perl - $_ and @_ - (2006-12-07)
  [639] Progress bars and other dynamic reports - (2006-03-09)
  [493] Running a Perl script within a PHP page - (2005-11-12)


Back to
Public courses - Autumn 2014 and 2015
Previous and next
or
Horse's mouth home
Forward to
Sunday is never quiet at Well House Manor
Some other Articles
Learning to program in Java - yes, we can help.
Please do not ask me to be the chair!
Libre Office - unable to get past REOPEN WINDOWS? question
Sunday is never quiet at Well House Manor
Perl - still a very effective language indeed for extracting and reporting
Public courses - Autumn 2014 and 2015
The Best Bed and Breakfast in Melksham?
Python - an interesting application
Lua courses - an update
4750 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95 at 50 posts per page


This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2017: 404 The Spa • Melksham, Wiltshire • United Kingdom • SN12 6QL
PH: 01144 1225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/mouth/4301_Per ... rting.html • PAGE BUILT: Sat May 27 16:49:10 2017 • BUILD SYSTEM: WomanWithCat