Log File Analysis - Perl example

this example from a Well House Consultants training course
More on this [link]

This example is described in the following article(s):
• Converting to Perl - the sort of programs you will write - [link]

This example references the following resources:
http://forum.doctissimo.fr/sante/cancers/file-la-tentation-sujet_158041_2.htm

Source code: lll1 Module: P255

=head1 Traffic Pattern Analysis

This is a serious (but very specific) piece of 'emergency' code written to investigate
some strange traffic patterns on our web site. Our log files are huge, so we've
processed them line by line ...

We're particularly interested in the number and status of accesses to each of our pages,
and we want to see the figures as a percentage of our traffic too, rather than just as
a raw byte count.

=cut

sub bycount { $amount{$a} <=> $amount{$b}; }

print `clear`;
print "Log Analysis\n";
open (FH,"log") or die;

# Process the file ... and report all "buffet-lunch" records

while (<FH>) {
        # Eliminate Apache 2.2.x's internal dummy connections!
        if (/^83\.170\.93\.77/) {
                $ninjas++;
                next; }

        # Gather the various statistics we're interested in
        $real++;
        ($url,$sta,$bytes) = (split(/\s+/))[6,8,9];
        $counter{$url}++;
        $status{$url} = $sta;
        $amount{$url} += $bytes;
        $yumyum += $bytes;

        # Report on the buffet lunch records
        if ($url eq "/slideshow/buffet-lunch.jpg") {
                print ;
                }
        }

print "Real $real and internal $ninjas\n";

# Wait for the user to press [enter]
# (give him / her a chance to read the results so far)
$paws = <STDIN>;

@bunch = sort bycount (keys %amount);
foreach $item (@bunch) {
        $flag = " ";
        $flag = "*" if ($status{$item} != 200) ;
        $pc = $amount{$item} / $yumyum * 100;
        printf "%1s %3d %5d %9d %4.2f%% %s\n", $flag,
                $status{$item},$counter{$item},
                $amount{$item},$pc,$item;
        }

__END__

Sample output - earlier section ...

75.73.46.227 - - [07/Mar/2009:01:27:28 +0000] "GET /slideshow/buffet-lunch.jpg HTTP/1.1" 200 921434 "http://1scotland16ecologists.wordpress.com/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_5; en-us) AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.2 Safari/525.20.1"
81.185.152.103 - - [07/Mar/2009:01:37:37 +0000] "GET /slideshow/buffet-lunch.jpg HTTP/1.1" 200 921434 "http://forum.doctissimo.fr/sante/cancers/file-la-tentation-sujet_158041_2.htm" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7"
Real 132408 and internal 19201

Later report ....

  200 856 17774172 0.61% /demo/train_graphic_fgw.jpg
  200 358 20139046 0.69% /pix/lhs9_123568klnoqrstuvwxyz.jpg
  200 11 22946825 0.79% /sitemap.xml
  200 9786 24648353 0.85% /wellimg/mine132.jpg
  200 67 25291467 0.87% /resources/smap.php?adder=/mouth
  200 629 25651671 0.88% /pix/tsw_0123456789abc.jpg
  200 2692 35158734 1.21% /commonimages/phpandmysqlcourses.jpg
  200 768 35792035 1.23% /mouth/158_MySQL-LEFT-JOIN-and-RIGHT-JOIN-INNER-JOIN-and-OUTER-JOIN.html
  200 44 40536100 1.40% /slideshow/buffet-lunch.jpg
  200 3932 43851878 1.51% /bing/images/WHClogoBW.jpg
  200 12931 75820274 2.61% /bing/C3.css
  200 757 115952491 3.99% /net/recents.html

Learn about this subject

This module and example are covered as required on private courses. Should you wish to cover this example and associated subjects, and you're attending a public course to cover other topics with us, please see our extra topic program.

Books covering this topic

Yes. We have over 700 books in our library. Books covering Perl are listed here and when you've selected a relevant book we'll link you on to Amazon to order.

Other Examples

This example comes from our "this" training module. You'll find a description of the topic and some other closely related examples on the "this" module index page.

Full description of the source code

You can learn more about this example on the training courses listed on this page, on which you'll be given a full set of training notes.

Many other training modules are available for download (for limited use) from our download centre under an Open Training Notes License.

Other resources

• Our Solutions centre provides a number of longer technical articles.
• Our Opentalk forum archive provides a question and answer centre.
• The Horse's mouth provides a daily tip or thought.
• Further resources are available via the resources centre.
• All of these resources can be searched through through our search engine
• And there's a global index here.

Purpose of this website

This is a sample program, class demonstration or answer from a training course. It's main purpose is to provide an after-course service to customers who have attended our public private or on site courses, but the examples are made generally available under conditions described below.

Web site author

This web site is written and maintained by Well House Consultants.

Conditions of use

Past attendees on our training courses are welcome to use individual examples in the course of their programming, but must check the examples they use to ensure that they are suitable for their job. Remember that some of our examples show you how not to do things - check in your notes. Well House Consultants take no responsibility for the suitability of these example programs to customer's needs.

This program is copyright Well House Consultants Ltd. You are forbidden from using it for running your own training courses without our prior written permission. See our page on courseware provision for more details.

Any of our images within this code may NOT be reused on a public URL without our prior permission. For Bona Fide personal use, we will often grant you permission provided that you provide a link back. Commercial use on a website will incur a license fee for each image used - details on request.