When I'm programming a log file analysis in Perl, I'll often "slurp" the whole file into a list which I can then traverse efficiently as many times as I need. If I need to look backwards from some interesting event to see what happened in the immediate lead up to it, I can do so simply by looking at a few previous list elements.

This technique fails if my log file can be so massive that it won't fit into my computer memory, or if doing so produces severe swapping. Fortunately, there are other ways of dealing with this in Perl.

* Using a queue, each item that's read from the file and is a potential prior event is pushed (on unshift-ed) onto a list
* When an event that triggers a report is found, the prior events in the queue can be examimed and reported on
* The queue of prior events is kept trimmed to an appropriate length.

I've applied this technique to quotations for private courses in our log files ... when I find a quote request, I want to know the background.

Code to add all .html and .php requests to a queue just in case the user calls up a quote:

next unless (m!T \S+\.(php|html)!);
unshift @queue,$_; # Add to queue
$#queue = 20; # Trim queue

Code to report when a trigger event is found

if (m!GET /net/quote\.html!) { # is it a quote?
($host,$page) = (split)[0,6];
print "Host: $host\nQuote: $page\n";
foreach (@queue) { # If so, report on previous requests
($ohost,$opage) = (split)[0,6];
print "From: $opage\n" if ($host eq $ohost);
}
print "\n";

The code to keep the queue trimmed back is part of the "add to queue" code just above.

Here's an example of the results that I get from running my program ...

Host: cache2.uk.Ichangedthis.com
Quote: /net/quote.html?where=bs32+4tr
From: /course/otc.html
and this tells me that the person who was looking for a quotation for training in BS32 had previously visited the page of general information about on site training ...

Full source code of the example that I'm discussing here is available in our handling huge data resource, and is taught on our Perl for larger projects course.
(written 2006-06-15)

 P208 - Perl - Lists
  [3548] Dark mornings, dog update, and Python and Lua courses before Christmas - (2011-12-10)
  [3400] $ is atomic and % and @ are molecular - Perl - (2011-08-20)
  [2996] Copying - duplicating data, or just adding a name? Perl and Python compared - (2010-10-12)
  [2833] Fresh Perl Teaching Examples - part 2 of 3 - (2010-06-27)
  [2813] Iterating over a Perl list and changing all items - (2010-06-15)
  [2484] Finding text and what surrounds it - contextual grep - (2009-10-30)
  [2295] The dog is not in trouble - (2009-07-17)
  [2226] Revision / Summary of lists - Perl - (2009-06-10)
  [2067] Perl - lists do so much more than arrays - (2009-03-05)
  [1918] Perl Socket Programming Examples - (2008-12-02)
  [1917] Out of memory during array extend - Perl - (2008-12-02)
  [1828] Perl - map to process every member of a list (array) - (2008-10-09)
  [1703] Perl ... adding to a list - end, middle, start - (2008-07-09)
  [1316] Filtering and altering Perl lists with grep and map - (2007-08-23)
  [1304] Last elements in a Perl or Python list - (2007-08-16)
  [968] Perl - a list or a hash? - (2006-12-06)
  [928] C++ and Perl - why did they do it THAT way? - (2006-11-16)
  [773] Breaking bread - (2006-06-22)
  [622] Queues and barrel rolls in Perl - (2006-02-24)
  [560] The fencepost problem - (2006-01-10)
  [463] Splitting the difference - (2005-10-13)
  [355] Context in Perl - (2005-06-22)
  [240] Conventional restraints removed - (2005-03-09)
  [230] Course sizes - beware of marketing statistics - (2005-02-27)
  [140] Comparison Chart for Perl programmers - list functions - (2004-12-04)
  [28] Perl for breakfast - (2004-08-25)

P667 - Perl - Handling Huge Data
  [3375] How to interact with a Perl program while it is processing data - (2011-07-31)
  [3374] Speeding up your Perl code - (2011-07-30)
  [2834] Teaching examples in Perl - third and final part - (2010-06-27)
  [2806] Macho matching - do not do it! - (2010-06-13)
  [2805] How are you getting on? - (2010-06-13)
  [2376] Long job - progress bar techniques (Perl) - (2009-08-26)
  [1924] Preventing ^C stopping / killing a program - Perl - (2008-12-05)
  [1920] Progress Bar Techniques - Perl - (2008-12-03)
  [1397] Perl - progress bar, supressing ^C and coping with huge data flows - (2007-10-20)
  [975] Answering ALL the delegate's Perl questions - (2006-12-09)
  [639] Progress bars and other dynamic reports - (2006-03-09)

Back to Great new inventions

Previous and next or Horse's mouth home

Forward to Much more that the world cup

Discounts and approved supplier lists
Perl - turning seconds into days, hours, minutes and seconds
Journey planning - Xephos v Transport Direct
Much more that the world cup
Huge data files - what happened earlier?
Great new inventions
Self help in Perl
Watch your Google profile
Very good for woodlice
Horse and Python training