Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
For 2023 (and 2024 ...) - we are now fully retired from IT training.
We have made many, many friends over 25 years of teaching about Python, Tcl, Perl, PHP, Lua, Java, C and C++ - and MySQL, Linux and Solaris/SunOS too. Our training notes are now very much out of date, but due to upward compatability most of our examples remain operational and even relevant ad you are welcome to make us if them "as seen" and at your own risk.

Lisa and I (Graham) now live in what was our training centre in Melksham - happy to meet with former delegates here - but do check ahead before coming round. We are far from inactive - rather, enjoying the times that we are retired but still healthy enough in mind and body to be active!

I am also active in many other area and still look after a lot of web sites - you can find an index ((here))
Long job - progress bar techniques (Perl)

Here's a "Perl for Larger Projects" example --- for use in illustrating the "advanced file and directory handling" and "handling huge data set" modules.

Scenario ... I want to go through all the files and directories on a big drive, and find the largest file(s). It will take a while, so I want progress reports, and I want to be able to suppress and ^C (control C) inputs by the user so that the program can't be stopped without very specific actions.

Let's look at the code section by section.

Firstly, set up the %SIG hash so that the ^C signal runs a function called jose and the USR1 signal sets a flag to indicate that it's been received (you rarely do very much in a signal handler, as you don't know exactly when it will be run - it's usual just to get a flag).

sub jose {
  print "No way!\n";
  die ("\nInsistent little toad!\n") if ($jose_counter > 2) ;
$SIG{"INT"} = \&jose;
$jose_counter = 0;
sub sofar {
  $sof = 1;
$SIG{"USR1"} = \&sofar;
$sof = 0;
print "My ID is $$\n";

Set up a queue to contain a list of all the directories to be traversed. We'll push new directories we find onto it, and we'll shift ones that we've already covered off it ... so that when the queue is empty, we know the whole traversal has been completed. A queue like this is far easier to implement than a circular buffer with two pointers, which is the 'conventional' way of doing the job.

@queue = ("/Users/grahamellis");
$biggest = 0;
$occasional = 0;

By default, Perl buffers output. We want it to flush its buffers every time it write to STDOUT, so that our progress reports come out in a single stream and not spurts of 4k characters

$| = 1;

Get a directory off the list, and read all the things (symbols) it contains:

while ($current = shift (@queue)) {
  opendir (DH,$current);
  @things = readdir DH;

Loop through the contents ... eliminate the current directory and the parent directory (this is done to avoid an infinite loop of directories) and then check if the item is a directory. If it IS, add it to the queue to be handled later.

  foreach $item (@things) {
    next if ($item =~ /^\.{1,2}$/);
    $filename = "$current/$item";
    if (-d $filename) {
      push @queue,$filename;

Not a directory, so look at the (file) size. Because we just did a -d operation on the file, we can now just use an underscore with the -s; the _ instructs Perl to reuse the results of its last enquiry from the file system ... it will have buffered up all the stat information.

If the file is larger than any already checked, make a note of the name and size:

    if (-s _ > $biggest) {
      $biggest = -s _;
      $bf = $filename;

Do we need to update the status line? We have chosen to do so once every 50 directories parsed, reporting the name of the latest directory and queue length. We have very strictly ensured that the report will always be the same number of characters long, so there is no chance of "droppings" being left on the end of a report line, and we have used \r to give a carriage return without a line feed.

  $occasional = ($occasional + 1) % 50;
  if (! $occasional) {
    $cur = $current;
    length($cur) > 25 and
      $cur = substr($cur,length($cur)-25);
    printf "Q len %4d - %30.30s\r",@queue+0,$cur;

If the USR1 signal has been received during the latest cycle through the loop, report now on the largest file so far, and turn the reporting back off (the signal wants to say "report once" rather than "report every cycle from now on" after all!)

  if ($sof) {
    print "\nSo far - $biggest, $bf\n";
    $sof = 0;

And at the VERY end, print the results!

print "\nbiggest is $bf, sized $biggest\n";

Full source code [here]

Update - added 26th August I have received an email overnight from a gentleman who points out that my Perl code (above) fails to "use strict", makes poor use of file handles, uses variable names that aren't very good, and so on. He's right - it wasn't claimed to be an example of excellent practise.

The code was written during a course earlier this week, illustrating points as I wrote it, and showing the mechanisms involved without overburdening the source with extras. As usual, some of my delegates have existing code that they have to maintain, and some of the less commonly used structures in my example are there to illustrate those structures ... not because it's necessarily a good idea to use them in production code. If you follow the link to the full source code, you'll find further details as to what this example is (and is not) intended to be.
(written 2009-08-26)

Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
P667 - Perl - Handling Huge Data
  [639] Progress bars and other dynamic reports - (2006-03-09)
  [762] Huge data files - what happened earlier? - (2006-06-15)
  [975] Answering ALL the delegate's Perl questions - (2006-12-09)
  [1397] Perl - progress bar, supressing ^C and coping with huge data flows - (2007-10-20)
  [1920] Progress Bar Techniques - Perl - (2008-12-03)
  [1924] Preventing ^C stopping / killing a program - Perl - (2008-12-05)
  [2805] How are you getting on? - (2010-06-13)
  [2806] Macho matching - do not do it! - (2010-06-13)
  [2834] Teaching examples in Perl - third and final part - (2010-06-27)
  [3374] Speeding up your Perl code - (2011-07-30)
  [3375] How to interact with a Perl program while it is processing data - (2011-07-31)

Back to
Designing your data structures for a robust Perl application
Previous and next
Horse's mouth home
Forward to
Wiltshire / Melksham Weddings - guest accommodation
Some other Articles
Object Oriented programming - a practical design example
Making variables persistant, pretending a database is a variable and other Perl tricks
Handling XML in Perl - introduction and early examples
Wiltshire / Melksham Weddings - guest accommodation
Long job - progress bar techniques (Perl)
Designing your data structures for a robust Perl application
Lead characters on Perl variable names
Translation from Ghanaian to English
Public Transport from (and to) Melksham on Sundays
Quiet summer days? I think not!
4759 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96 at 50 posts per page

This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2024: 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01144 1225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/mouth/2376_Lon ... Perl-.html • PAGE BUILT: Sun Oct 11 16:07:41 2020 • BUILD SYSTEM: JelliaJamb