looking behind in huge data files

Handling Huge Data example from a Well House Consultants training course
More on Handling Huge Data [link]

This example is described in the following article(s):
• Huge data files - what happened earlier? - [link]

Source code: behind Module: P667

#!/usr/bin/env perl

# What lead up to a quote request?

# See documentation after the Perl source

while (<>) {
   next if (/googlebot/); # Eliminate quotes to spiders
   next if (/crawl/);
   next if (/spider/);
   next if (/inktomi/);
   if (m!GET /net/quote\.html!) { # is it a quote?
      ($host,$page) = (split)[0,6];
      print "Host: $host\nQuote: $page\n";
      foreach (@queue) { # If so, report on previous requests
         ($ohost,$opage) = (split)[0,6];
         print "From: $opage\n" if ($host eq $ohost);
         }
      print "\n";
   } else {
      next unless (m!T \S+\.(php|html)!);
      unshift @queue,$_; # Add to queue
      $#queue = 20; # Trim queue
   }
}

__END__

=head1

When you're processing large data files in Perl, you
can simply 'slurp' them into a list in memory, and
iterate forward and look back through that list as
many times as you like. But when your data volume
increases from being large to being so huge that it
won't fit in memory, that's another issue.

"Perl makes the difficule easy, and the impossible
becomes possible" according to Larry Wall. And such
is the case with looking back in huge data.

In this sample program, I'm reading a potenially
massive web access log file that's named on the
command line, and looking for requests for quotations.
I then look back a few pages to discover (and report
on) web pages viewed by the same user just prior to
them asking for the quote.

The mechanism is to use a Perl list as a queue onto
which I add (via unshift) each new .html page as I
travese the whole huge web log, and which I keeped
trimmed back to a threshold of 20 elements as my
lookback buffer.

=cut

Learn about this subject

This module and example are covered on our public Perl for Larger Projects course. If you have a group of three or more trainees who need to learn the subject, we can also arrange a private or on site course for you.

Books covering this topic

Yes. We have over 700 books in our library. Books covering Perl are listed here and when you've selected a relevant book we'll link you on to Amazon to order.

Other Examples

This example comes from our "Handling Huge Data" training module. You'll find a description of the topic and some other closely related examples on the "Handling Huge Data" module index page.

Full description of the source code

You can learn more about this example on the training courses listed on this page, on which you'll be given a full set of training notes.

Many other training modules are available for download (for limited use) from our download centre under an Open Training Notes License.

Other resources

• Our Solutions centre provides a number of longer technical articles.
• Our Opentalk forum archive provides a question and answer centre.
• The Horse's mouth provides a daily tip or thought.
• Further resources are available via the resources centre.
• All of these resources can be searched through through our search engine
• And there's a global index here.

Purpose of this website

This is a sample program, class demonstration or answer from a training course. It's main purpose is to provide an after-course service to customers who have attended our public private or on site courses, but the examples are made generally available under conditions described below.

Web site author

This web site is written and maintained by Well House Consultants.

Conditions of use

Past attendees on our training courses are welcome to use individual examples in the course of their programming, but must check the examples they use to ensure that they are suitable for their job. Remember that some of our examples show you how not to do things - check in your notes. Well House Consultants take no responsibility for the suitability of these example programs to customer's needs.

This program is copyright Well House Consultants Ltd. You are forbidden from using it for running your own training courses without our prior written permission. See our page on courseware provision for more details.

Any of our images within this code may NOT be reused on a public URL without our prior permission. For Bona Fide personal use, we will often grant you permission provided that you provide a link back. Commercial use on a website will incur a license fee for each image used - details on request.