Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
 
Python and Tcl - public course schedule [here]
Private courses on your site - see [here]
Please ask about maintenance training for Perl, PHP, Lua, etc
 
Asking a long running application for intermediate reports
Handling Huge Data example from a Well House Consultants training course
More on Handling Huge Data [link]

This example is described in the following article(s):
   • Progress bars and other dynamic reports - [link]
   • How are you getting on? - [link]

Source code: huge3 Module: P667
#!/usr/bin/env perl

# Huge 3 - a demonstration of better user feedback
#
# in this example I have added a signal handler so that
# the use can "kill" the process from another window and
# get the results so far reported on the screen in summary (USR1)
# or to a file (USR2)

use site;

@lookfor = @ARGV;

# Set up the signal handler, tell the user the process ID so that
# he doesn't have to use the ps command to find out.

$SIG{"USR1"} = "setsummary"; # setsummary in main package
$SIG{"USR2"} = "filedump";
print STDERR "use the command\n";
print STDERR " kill -USR1 $$\nto summarise so far to screen\n";
print STDERR " kill -USR2 $$\nto report results so far to out.txt\n";
$summary = 0; $dump = 0;
# Also produce a file containing the process ID (so that other
# programs can easily find our ID, or the user can alias a kill
# command: kill -USR1 `cat huge3.pid`
open (FH,">$0.pid");
print FH "$$\n";
close FH;

# Nicities - trap signals that may be issued accidentally.
# using the \& notation means "in current package" for the subroutine.
# We could have also used the word IGNORE to just ignore a signal.
$SIG{"TERM"} = \&nodie; # kill command with no signal name given
$SIG{"INT"} = \&nodie; # ^C from keyboard
$tell = 0;

init site("content.rdf"); # run "init" sub in "site" module.

# Following code extra initialisation for for report line

$datalength = -s "content.rdf";
select STDERR;
$|=1;
select STDOUT;
$started = time();
$st = localtime($started);
($stb) = ($st=~/(\d\d:\d\d)/);

# loop through each available site!
# remember those that look interesting!

while ($current = new site()) { # run "new" sub in "site" module.
        $mtot = 1;
        foreach $keyword(@lookfor) {
                $mtot *= $current -> getmatch($keyword);
                }
        push @got,$current if ($mtot > 1);
        $nsites ++;

        # Generate report lines

        unless ($nsites%5000) {
                # report on how we're doing
                $percent = $site::bytesread/$datalength * 100;
                printf STDERR "\rsites %-8d matches %-6d percent %3d%%",
                        $nsites,@got+0,$percent;

                # and / or report on estimated timing.
                $alternate++;
                unless ($alternate %2) {
                        $now = time();
                        $est_ended = $started + ($now-$started)
                                / ($site::bytesread/$datalength);
                        $no = localtime($now);
                        ($nob) = ($no=~/(\d\d:\d\d)/);
                        $en = localtime($est_ended);
                        ($enb) = ($en=~/(\d\d:\d\d)/);
                        printf STDERR "\rStarted %5s; now %5s Est. end %5s",
                                $stb,$nob,$enb;
                }
                }

        # Generate summary line is signal set

        if ($summary) {
                $nprinted = 0;
                print STDERR "\n";
                if (@got) {
                foreach $good(@got) {
                $url = $good -> getURL();
                printf STDERR "%35s ",$url;
                print STDERR "\n" if ($good eq $got[-1] or $nprinted++%2);
                }
                } else {
                print STDERR "No hits found yet\n";
                }
                $summary = 0;
                }

        if ($tell) {
                print STDERR "\nCan't kill this way\n";
                $tell = 0 ;
        }

        dumpout(0) if ($dump); # USR2 received
}

dumpout(1); # dump to file on completion. Nice touch as it means that
# if the results scroll off the screen after hours of processing,
# the user can get back to them!

# All sites now gotten! Those which did not match have been eliminated
# from $current and so the memory within "site" was released.

foreach $good(@got) {
        $url = $good -> getURL();
        $tit = $good -> getTitle();
        $des = $good -> getDescription();
        print "Have a look at: $url\n$tit\n$des\n\n";
        }
###############################################################
sub dumpout {
my ($final) = @_;
open (FH,">out.txt");
print FH "Searching for @lookfor\n";
print FH "These are ",$final?"complete":"interim"," results\n";
print FH "Searched through $nsites sites, matched ",@got+0,
                " started at $st\n\n";
foreach $good(@got) {
        $url = $good -> getURL();
        $tit = $good -> getTitle();
        $des = $good -> getDescription();
        print FH "url: $url\ntitle: $tit\ndesc: $des\n\n";
        }
close FH; # Vital to flush buffers on interim report!
$dump = 0;
}
################################################################
# When a signal is receive, just note it. We don't act on it
# here - it could be received at any point in the running of the
# program, including in the middle of updating an object. We'll
# act on the request at the end of handling the current site.

# Beware - anything substantial in placed into these asyncronous
# subroutines can cause core dumps when you run - even a print!

sub setsummary {
        $summary = 1;
}
sub nodie {
        $tell = 1;
}
sub filedump {
        $dump = 1;
}

Learn about this subject
This module and example are covered on our public Perl for Larger Projects course. If you have a group of three or more trainees who need to learn the subject, we can also arrange a private or on site course for you.

Books covering this topic
Yes. We have over 700 books in our library. Books covering Perl are listed here and when you've selected a relevant book we'll link you on to Amazon to order.

Other Examples
This example comes from our "Handling Huge Data" training module. You'll find a description of the topic and some other closely related examples on the "Handling Huge Data" module index page.

Full description of the source code
You can learn more about this example on the training courses listed on this page, on which you'll be given a full set of training notes.

Many other training modules are available for download (for limited use) from our download centre under an Open Training Notes License.

Other resources
• Our Solutions centre provides a number of longer technical articles.
• Our Opentalk forum archive provides a question and answer centre.
The Horse's mouth provides a daily tip or thought.
• Further resources are available via the resources centre.
• All of these resources can be searched through through our search engine
• And there's a global index here.

Web site author
This web site is written and maintained by Well House Consultants.

Purpose of this website
This is a sample program, class demonstration or answer from a training course. It's main purpose is to provide an after-course service to customers who have attended our public private or on site courses, but the examples are made generally available under conditions described below.

Conditions of use
Past attendees on our training courses are welcome to use individual examples in the course of their programming, but must check the examples they use to ensure that they are suitable for their job. Remember that some of our examples show you how not to do things - check in your notes. Well House Consultants take no responsibility for the suitability of these example programs to customer's needs.

This program is copyright Well House Consultants Ltd. You are forbidden from using it for running your own training courses without our prior written permission. See our page on courseware provision for more details.

Any of our images within this code may NOT be reused on a public URL without our prior permission. For Bona Fide personal use, we will often grant you permission provided that you provide a link back. Commercial use on a website will incur a license fee for each image used - details on request.

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2020: Well House Manor • 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01225 708225 • FAX: 01225 793803 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/resources/ex.php4 • PAGE BUILT: Sun Jan 31 13:04:01 2016 • BUILD SYSTEM: WomanWithCat