Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
 
For 2021 - online Python 3 training - see ((here)).

Our plans were to retire in summer 2020 and see the world, but Coronavirus has lead us into a lot of lockdown programming in Python 3 and PHP 7.
We can now offer tailored online training - small groups, real tutors - works really well for groups of 4 to 14 delegates. Anywhere in the world; course language English.

Please ask about private 'maintenance' training for Python 2, Tcl, Perl, PHP, Lua, etc.
Huge data files - what happened earlier?

When I'm programming a log file analysis in Perl, I'll often "slurp" the whole file into a list which I can then traverse efficiently as many times as I need. If I need to look backwards from some interesting event to see what happened in the immediate lead up to it, I can do so simply by looking at a few previous list elements.

This technique fails if my log file can be so massive that it won't fit into my computer memory, or if doing so produces severe swapping. Fortunately, there are other ways of dealing with this in Perl.

* Using a queue, each item that's read from the file and is a potential prior event is pushed (on unshift-ed) onto a list
* When an event that triggers a report is found, the prior events in the queue can be examimed and reported on
* The queue of prior events is kept trimmed to an appropriate length.

I've applied this technique to quotations for private courses in our log files ... when I find a quote request, I want to know the background.

Code to add all .html and .php requests to a queue just in case the user calls up a quote:

next unless (m!T \S+\.(php|html)!);
unshift @queue,$_; # Add to queue
$#queue = 20; # Trim queue


Code to report when a trigger event is found

if (m!GET /net/quote\.html!) { # is it a quote?
($host,$page) = (split)[0,6];
print "Host: $host\nQuote: $page\n";
foreach (@queue) { # If so, report on previous requests
($ohost,$opage) = (split)[0,6];
print "From: $opage\n" if ($host eq $ohost);
}
print "\n";


The code to keep the queue trimmed back is part of the "add to queue" code just above.

Here's an example of the results that I get from running my program ...

Host: cache2.uk.Ichangedthis.com
Quote: /net/quote.html?where=bs32+4tr
From: /course/otc.html

and this tells me that the person who was looking for a quotation for training in BS32 had previously visited the page of general information about on site training ...

Full source code of the example that I'm discussing here is available in our handling huge data resource, and is taught on our Perl for larger projects course.
(written 2006-06-15)

 
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
P667 - Perl - Handling Huge Data
  [639] Progress bars and other dynamic reports - (2006-03-09)
  [975] Answering ALL the delegate's Perl questions - (2006-12-09)
  [1397] Perl - progress bar, supressing ^C and coping with huge data flows - (2007-10-20)
  [1920] Progress Bar Techniques - Perl - (2008-12-03)
  [1924] Preventing ^C stopping / killing a program - Perl - (2008-12-05)
  [2376] Long job - progress bar techniques (Perl) - (2009-08-26)
  [2805] How are you getting on? - (2010-06-13)
  [2806] Macho matching - do not do it! - (2010-06-13)
  [2834] Teaching examples in Perl - third and final part - (2010-06-27)
  [3374] Speeding up your Perl code - (2011-07-30)
  [3375] How to interact with a Perl program while it is processing data - (2011-07-31)

P208 - Perl - Lists
  [28] Perl for breakfast - (2004-08-25)
  [140] Comparison Chart for Perl programmers - list functions - (2004-12-04)
  [230] Course sizes - beware of marketing statistics - (2005-02-27)
  [240] Conventional restraints removed - (2005-03-09)
  [355] Context in Perl - (2005-06-22)
  [463] Splitting the difference - (2005-10-13)
  [560] The fencepost problem - (2006-01-10)
  [622] Queues and barrel rolls in Perl - (2006-02-24)
  [773] Breaking bread - (2006-06-22)
  [928] C++ and Perl - why did they do it THAT way? - (2006-11-16)
  [968] Perl - a list or a hash? - (2006-12-06)
  [1304] Last elements in a Perl or Python list - (2007-08-16)
  [1316] Filtering and altering Perl lists with grep and map - (2007-08-23)
  [1703] Perl ... adding to a list - end, middle, start - (2008-07-09)
  [1828] Perl - map to process every member of a list (array) - (2008-10-09)
  [1917] Out of memory during array extend - Perl - (2008-12-02)
  [1918] Perl Socket Programming Examples - (2008-12-02)
  [2067] Perl - lists do so much more than arrays - (2009-03-05)
  [2226] Revision / Summary of lists - Perl - (2009-06-10)
  [2295] The dog is not in trouble - (2009-07-17)
  [2484] Finding text and what surrounds it - contextual grep - (2009-10-30)
  [2813] Iterating over a Perl list and changing all items - (2010-06-15)
  [2833] Fresh Perl Teaching Examples - part 2 of 3 - (2010-06-27)
  [2996] Copying - duplicating data, or just adding a name? Perl and Python compared - (2010-10-12)
  [3400] $ is atomic and % and @ are molecular - Perl - (2011-08-20)
  [3548] Dark mornings, dog update, and Python and Lua courses before Christmas - (2011-12-10)
  [3669] Stepping through a list (or an array) in reverse order - (2012-03-23)
  [3870] Writing more maintainable Perl - naming fields from your data records - (2012-09-25)
  [3906] Taking the lead, not the dog, for a walk. - (2012-10-28)
  [3939] Lots of ways of doing the same thing in Perl - list iteration - (2012-12-03)
  [4609] Mapping an array / list without a loop - how to do it in Perl 6 - (2016-01-03)


Back to
Great new inventions
Previous and next
or
Horse's mouth home
Forward to
Much more that the world cup
Some other Articles
Discounts and approved supplier lists
Perl - turning seconds into days, hours, minutes and seconds
Journey planning - Xephos v Transport Direct
Much more that the world cup
Huge data files - what happened earlier?
Great new inventions
Self help in Perl
Watch your Google profile
Very good for woodlice
Horse and Python training
4759 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96 at 50 posts per page


This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2021: 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01144 1225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/mouth/762_Huge ... lier-.html • PAGE BUILT: Sun Oct 11 16:07:41 2020 • BUILD SYSTEM: JelliaJamb