When I'm programming a log file analysis in Perl, I'll often "slurp" the whole file into a list which I can then traverse efficiently as many times as I need. If I need to look backwards from some interesting event to see what happened in the immediate lead up to it, I can do so simply by looking at a few previous list elements.
This technique fails if my log file can be so massive that it won't fit into my computer memory, or if doing so produces severe swapping. Fortunately, there are other ways of dealing with this in Perl.
* Using a queue, each item that's read from the file and is a potential
prior event is pushed (on unshift-ed) onto a list
* When an event that triggers a report is found, the prior events in the queue can be examimed and reported on
* The queue of prior events is kept trimmed to an appropriate length.
I've applied this technique to quotations for private courses in our log files ... when I find a quote request, I want to know the background.
Code to add all .html and .php requests to a queue just in case the user calls up a quote:
next unless (m!T \S+\.(php|html)!);
unshift @queue,$_; # Add to queue
$#queue = 20; # Trim queue
Code to report when a trigger event is found
if (m!GET /net/quote\.html!) { # is it a quote?
($host,$page) = (split)[0,6];
print "Host: $host\nQuote: $page\n";
foreach (@queue) { # If so, report on previous requests
($ohost,$opage) = (split)[0,6];
print "From: $opage\n" if ($host eq $ohost);
}
print "\n";
The code to keep the queue trimmed back is part of the "add to queue" code just above.
Here's an example of the results that I get from running my program ...
Host: cache2.uk.Ichangedthis.com
Quote: /net/quote.html?where=bs32+4tr
From: /course/otc.html
and this tells me that the person who was looking for a quotation for training in BS32 had previously visited the page of general information about on site training ...
Full source code of the example that I'm discussing here is available in our
handling huge data resource, and is taught on our
Perl for larger projects course.
(written 2006-06-15)
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
P667 - Perl - Handling Huge Data [639] Progress bars and other dynamic reports - (2006-03-09)
[975] Answering ALL the delegate's Perl questions - (2006-12-09)
[1397] Perl - progress bar, supressing ^C and coping with huge data flows - (2007-10-20)
[1920] Progress Bar Techniques - Perl - (2008-12-03)
[1924] Preventing ^C stopping / killing a program - Perl - (2008-12-05)
[2376] Long job - progress bar techniques (Perl) - (2009-08-26)
[2805] How are you getting on? - (2010-06-13)
[2806] Macho matching - do not do it! - (2010-06-13)
[2834] Teaching examples in Perl - third and final part - (2010-06-27)
[3374] Speeding up your Perl code - (2011-07-30)
[3375] How to interact with a Perl program while it is processing data - (2011-07-31)
P208 - Perl - Lists [28] Perl for breakfast - (2004-08-25)
[140] Comparison Chart for Perl programmers - list functions - (2004-12-04)
[230] Course sizes - beware of marketing statistics - (2005-02-27)
[240] Conventional restraints removed - (2005-03-09)
[355] Context in Perl - (2005-06-22)
[463] Splitting the difference - (2005-10-13)
[560] The fencepost problem - (2006-01-10)
[622] Queues and barrel rolls in Perl - (2006-02-24)
[773] Breaking bread - (2006-06-22)
[928] C++ and Perl - why did they do it THAT way? - (2006-11-16)
[968] Perl - a list or a hash? - (2006-12-06)
[1304] Last elements in a Perl or Python list - (2007-08-16)
[1316] Filtering and altering Perl lists with grep and map - (2007-08-23)
[1703] Perl ... adding to a list - end, middle, start - (2008-07-09)
[1828] Perl - map to process every member of a list (array) - (2008-10-09)
[1917] Out of memory during array extend - Perl - (2008-12-02)
[1918] Perl Socket Programming Examples - (2008-12-02)
[2067] Perl - lists do so much more than arrays - (2009-03-05)
[2226] Revision / Summary of lists - Perl - (2009-06-10)
[2295] The dog is not in trouble - (2009-07-17)
[2484] Finding text and what surrounds it - contextual grep - (2009-10-30)
[2813] Iterating over a Perl list and changing all items - (2010-06-15)
[2833] Fresh Perl Teaching Examples - part 2 of 3 - (2010-06-27)
[2996] Copying - duplicating data, or just adding a name? Perl and Python compared - (2010-10-12)
[3400] $ is atomic and % and @ are molecular - Perl - (2011-08-20)
[3548] Dark mornings, dog update, and Python and Lua courses before Christmas - (2011-12-10)
[3669] Stepping through a list (or an array) in reverse order - (2012-03-23)
[3870] Writing more maintainable Perl - naming fields from your data records - (2012-09-25)
[3906] Taking the lead, not the dog, for a walk. - (2012-10-28)
[3939] Lots of ways of doing the same thing in Perl - list iteration - (2012-12-03)
[4609] Mapping an array / list without a loop - how to do it in Perl 6 - (2016-01-03)
Some other Articles
Discounts and approved supplier listsPerl - turning seconds into days, hours, minutes and secondsJourney planning - Xephos v Transport DirectMuch more that the world cupHuge data files - what happened earlier?Great new inventionsSelf help in PerlWatch your Google profileVery good for woodliceHorse and Python training