Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
 
Python and Tcl - public course schedule [here]
Private courses on your site - see [here]
Please ask about maintenance training for Perl, PHP, Lua, etc
 
Huge data files - what happened earlier?

When I'm programming a log file analysis in Perl, I'll often "slurp" the whole file into a list which I can then traverse efficiently as many times as I need. If I need to look backwards from some interesting event to see what happened in the immediate lead up to it, I can do so simply by looking at a few previous list elements.

This technique fails if my log file can be so massive that it won't fit into my computer memory, or if doing so produces severe swapping. Fortunately, there are other ways of dealing with this in Perl.

* Using a queue, each item that's read from the file and is a potential prior event is pushed (on unshift-ed) onto a list
* When an event that triggers a report is found, the prior events in the queue can be examimed and reported on
* The queue of prior events is kept trimmed to an appropriate length.

I've applied this technique to quotations for private courses in our log files ... when I find a quote request, I want to know the background.

Code to add all .html and .php requests to a queue just in case the user calls up a quote:

next unless (m!T \S+\.(php|html)!);
unshift @queue,$_; # Add to queue
$#queue = 20; # Trim queue


Code to report when a trigger event is found

if (m!GET /net/quote\.html!) { # is it a quote?
($host,$page) = (split)[0,6];
print "Host: $host\nQuote: $page\n";
foreach (@queue) { # If so, report on previous requests
($ohost,$opage) = (split)[0,6];
print "From: $opage\n" if ($host eq $ohost);
}
print "\n";


The code to keep the queue trimmed back is part of the "add to queue" code just above.

Here's an example of the results that I get from running my program ...

Host: cache2.uk.Ichangedthis.com
Quote: /net/quote.html?where=bs32+4tr
From: /course/otc.html

and this tells me that the person who was looking for a quotation for training in BS32 had previously visited the page of general information about on site training ...

Full source code of the example that I'm discussing here is available in our handling huge data resource, and is taught on our Perl for larger projects course.
(written 2006-06-15)

 
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
P208 - Perl - Lists
  [4609] Mapping an array / list without a loop - how to do it in Perl 6 - (2016-01-03)
  [3939] Lots of ways of doing the same thing in Perl - list iteration - (2012-12-03)
  [3906] Taking the lead, not the dog, for a walk. - (2012-10-28)
  [3870] Writing more maintainable Perl - naming fields from your data records - (2012-09-25)
  [3669] Stepping through a list (or an array) in reverse order - (2012-03-23)
  [3548] Dark mornings, dog update, and Python and Lua courses before Christmas - (2011-12-10)
  [3400] $ is atomic and % and @ are molecular - Perl - (2011-08-20)
  [2996] Copying - duplicating data, or just adding a name? Perl and Python compared - (2010-10-12)
  [2833] Fresh Perl Teaching Examples - part 2 of 3 - (2010-06-27)
  [2813] Iterating over a Perl list and changing all items - (2010-06-15)
  [2484] Finding text and what surrounds it - contextual grep - (2009-10-30)
  [2295] The dog is not in trouble - (2009-07-17)
  [2226] Revision / Summary of lists - Perl - (2009-06-10)
  [2067] Perl - lists do so much more than arrays - (2009-03-05)
  [1918] Perl Socket Programming Examples - (2008-12-02)
  [1917] Out of memory during array extend - Perl - (2008-12-02)
  [1828] Perl - map to process every member of a list (array) - (2008-10-09)
  [1703] Perl ... adding to a list - end, middle, start - (2008-07-09)
  [1316] Filtering and altering Perl lists with grep and map - (2007-08-23)
  [1304] Last elements in a Perl or Python list - (2007-08-16)
  [968] Perl - a list or a hash? - (2006-12-06)
  [928] C++ and Perl - why did they do it THAT way? - (2006-11-16)
  [773] Breaking bread - (2006-06-22)
  [622] Queues and barrel rolls in Perl - (2006-02-24)
  [560] The fencepost problem - (2006-01-10)
  [463] Splitting the difference - (2005-10-13)
  [355] Context in Perl - (2005-06-22)
  [240] Conventional restraints removed - (2005-03-09)
  [230] Course sizes - beware of marketing statistics - (2005-02-27)
  [140] Comparison Chart for Perl programmers - list functions - (2004-12-04)
  [28] Perl for breakfast - (2004-08-25)

P667 - Perl - Handling Huge Data
  [3375] How to interact with a Perl program while it is processing data - (2011-07-31)
  [3374] Speeding up your Perl code - (2011-07-30)
  [2834] Teaching examples in Perl - third and final part - (2010-06-27)
  [2806] Macho matching - do not do it! - (2010-06-13)
  [2805] How are you getting on? - (2010-06-13)
  [2376] Long job - progress bar techniques (Perl) - (2009-08-26)
  [1924] Preventing ^C stopping / killing a program - Perl - (2008-12-05)
  [1920] Progress Bar Techniques - Perl - (2008-12-03)
  [1397] Perl - progress bar, supressing ^C and coping with huge data flows - (2007-10-20)
  [975] Answering ALL the delegate's Perl questions - (2006-12-09)
  [639] Progress bars and other dynamic reports - (2006-03-09)


Back to
Great new inventions
Previous and next
or
Horse's mouth home
Forward to
Much more that the world cup
Some other Articles
Discounts and approved supplier lists
Perl - turning seconds into days, hours, minutes and seconds
Journey planning - Xephos v Transport Direct
Much more that the world cup
Huge data files - what happened earlier?
Great new inventions
Self help in Perl
Watch your Google profile
Very good for woodlice
Horse and Python training
4759 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96 at 50 posts per page


This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2019: 404 The Spa • Melksham, Wiltshire • United Kingdom • SN12 6QL
PH: 01225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/mouth/762_Huge ... lier-.html • PAGE BUILT: Sat May 27 16:49:10 2017 • BUILD SYSTEM: WomanWithCat