Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
 
For 2023 (and 2024 ...) - we are now fully retired from IT training.
We have made many, many friends over 25 years of teaching about Python, Tcl, Perl, PHP, Lua, Java, C and C++ - and MySQL, Linux and Solaris/SunOS too. Our training notes are now very much out of date, but due to upward compatability most of our examples remain operational and even relevant ad you are welcome to make us if them "as seen" and at your own risk.

Lisa and I (Graham) now live in what was our training centre in Melksham - happy to meet with former delegates here - but do check ahead before coming round. We are far from inactive - rather, enjoying the times that we are retired but still healthy enough in mind and body to be active!

I am also active in many other area and still look after a lot of web sites - you can find an index ((here))
Perl - still a very effective language indeed for extracting and reporting

Perl remains a marvellous programming language for a number of applications, including quick / one off scripts which may be needed to manipulate data is ways that vary from day to day and week to week - research type work, if you like.

We manage / look after our own web server, hosting our IT training business, our hotel the Great Western Coffee Shop Forum and a whole series of other lower volume sites such as Melksham Chamber of Commerce and Industry and the TransWilts Community Rail Partnership. Various monitoring and logging scripts run on the system - from an external heartbeat which sends us an email if the server's not running within predefined metrics (or not running at all - this is an external script) through a script which takes a snapshot of vital parameters every few minutes to our web server logs which record details of each and every completed web server access.

With all this data ... when we want to find what's the reason behind a changing pattern, we're looking for a needle in a haystack - over 200,000 access records / over 50 Mbytes of logging datain some 24 hour periods. We need the "Practcial Extraction and Reporting Language".

Here are a couple of examples of Perl code that I've used so far this month, during which time we've had a couple of pattern changes on the server that I wished to investigate:

  while (<>) {
    ($h,$m,$s) = /:(\d+):(\d+):(\d+)/;
    $sid = $h * 3600 + $m * 60 + $s;
    print if ($sid - $oldsid < -450) ;
    $oldsid = $oldsid > $sid ? $oldsid : $sid;
    }


This identify slow running requests to the server - ones that take over 450 seconds to complete; this isn't quite as simple as it sounds, as only the start time of each request is recorded, but the recording is in the order of completion so the Perl job is looking back at previous record times ...

  while (<>) {
    @n = split;
    $c{$n[0]}++;
    }
  for $k (sort {$c{$a} - $c{$b}} keys %c) {
    printf "%6d %s\n",$c{$k},$k;
    }


And that piece of code counts accesses from each different visitor to the server, printing them out with the most frequent visitor listed last.

Each of these scripts was a vital step in identifying issues that were effecting server performance - one of our housekeeping scripts was being run rather more often than it should have been by visiting search engine crawlers (so we modified that script to ensure it's not run more than once every 24 hours), and we were getting up to 300 visits in 5 seconds from a system in the Ukraine; I don't know exactly what it was, but it was up to no good with accesses from the same computer claiming to be from Chrome, Firefox and Internet Explorer browsers ... when I suspect it was an automaton that was either our of control or malicious. A quick fix there is to have our web server simply send back a "forbidden" (403 code) response which it can do efficiently without any noticable effect on performance!

I have just given you the bare bones scripts above. We strongly recommend that it you're using things like this on a regular basis, you comment your scripts well, you use better variable names, you add flexibility by allowing different variables to be fed in, and you refrain from using constant values where you might want a change in the future. Such techniques are taught on both our Learning to program in Perl and Perl Programming courses, which run a number of times each year at our Melksham, Wiltshire training centre or can be run in private if you've got a group of delegates.
(written 2014-09-20, updated 2014-09-21)

 
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
P210 - Perl - Topicalization and Special Variables
  [493] Running a Perl script within a PHP page - (2005-11-12)
  [639] Progress bars and other dynamic reports - (2006-03-09)
  [969] Perl - $_ and @_ - (2006-12-07)
  [1136] Buffering output - why it is done and issues raised in Tcl, Perl, Python and PHP - (2007-04-06)
  [1221] Bathtubs and pecking birds - (2007-06-07)
  [1232] Bathtub example - (2007-06-14)
  [1289] Pure Perl - (2007-08-03)
  [1444] Using English can slow you right down! - (2007-11-25)
  [1508] How not to write Perl? - (2008-01-15)
  [1704] Finding operating system settings in Perl - (2008-07-10)
  [1705] Environment variables in Perl / use Env - (2008-07-11)
  [1728] A short Perl example - (2008-07-30)
  [1829] Dont bother to write a Perl program - (2008-10-10)
  [1860] Seven new intermediate Perl examples - (2008-10-30)
  [1922] Flurinci knows Raby Lae PHP and Jeve - (2008-12-04)
  [2833] Fresh Perl Teaching Examples - part 2 of 3 - (2010-06-27)
  [2876] Different perl examples - some corners I rarely explore - (2010-07-18)
  [2972] Some more advanced Perl examples from a recent course - (2010-09-27)
  [3449] Apache Internal Dummy Connection - what is it and what should I do with it? - (2011-09-19)
  [4395] Preparing data through a little bit of Perl - (2015-01-15)
  [4682] One line scripts - Awk, Perl and Ruby - (2016-05-20)
  [4700] Obfurscated code - it might work, but is it maintainable? - (2016-07-02)

P050 - Perl - General
  [116] The next generation of programmer - (2004-11-13)
  [400] New in the shops - (2005-08-01)
  [743] How to debug a Perl program - (2006-06-04)
  [1750] Glorious (?) 12th August - what a Pe(a)rl! - (2008-08-12)
  [1897] Keeping on an even keel - (2008-11-21)
  [2228] Where do I start when writing a program? - (2009-06-11)
  [2242] So what is this thing called Perl that I keep harping on about? - (2009-06-15)
  [2374] Lead characters on Perl variable names - (2009-08-24)
  [2504] Learning to program in ... - (2009-11-15)
  [2736] Perl Course FAQ - (2010-04-23)
  [2783] The Perl Survey - (2010-05-27)
  [2825] Perl course - is it tailored to Linux, or Microsoft Windows? - (2010-06-25)
  [2971] Should the public sector compete with businesses? and other deep questions - (2010-09-26)
  [3093] How many toilet rolls - hotel inventory and useage - (2010-12-18)
  [3322] How much has Perl (and other languages) changed? - (2011-06-10)
  [3332] DNA to Amino Acid - a sample Perl script - (2011-06-24)
  [3407] Perl - a quick reminder and revision. Test yourself! - (2011-08-26)
  [3823] Know Python or PHP? Want to learn Perl too? - (2012-07-31)
  [3902] Shell - Grep - Sed - Awk - Perl - Python - which to use when? - (2012-10-22)
  [3911] How well do you know Perl and / or Python? - (2012-11-04)
  [4296] Polishing the Perl courses - updated training - (2014-09-17)


Back to
Public courses - Autumn 2014 and 2015
Previous and next
or
Horse's mouth home
Forward to
Sunday is never quiet at Well House Manor
Some other Articles
Learning to program in Java - yes, we can help.
Please do not ask me to be the chair!
Libre Office - unable to get past REOPEN WINDOWS? question
Sunday is never quiet at Well House Manor
Perl - still a very effective language indeed for extracting and reporting
Public courses - Autumn 2014 and 2015
The Best Bed and Breakfast in Melksham?
Python - an interesting application
Lua courses - an update
4759 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96 at 50 posts per page


This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2024: 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01144 1225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/mouth/4301_Per ... rting.html • PAGE BUILT: Sun Oct 11 16:07:41 2020 • BUILD SYSTEM: JelliaJamb