Perl - progress bar, supressing ^C and coping with huge data flows

If you're handling a huge amount of data (gigabytes!) in a Perl program, memory won't allow you to slurp it all into a list and you'll traverse the data with a loop from file or from database. And because of the sheer volume of data, it may take a while to process. During such proessing, you may wish to:
• See a progress bar on the screen
• Be able to ask for intermediate results
• Disable ^C (Control-C) interrupts to avoid accidental termination

A progress bar can be printed out as text, using \r rather than \n at the end of each line. If you do that, though, you need to turn autoflush mode on so that yo actually see the reports rather than having them buffered, and you should use a formatted print that always generates the same length line to avoid "droppings" being left on the end when a short line overwrites a long line. If you're likely to send your output to a file, the progress bar should go to STDERR.

You can replace your ^C handler by changing the signal handler in $SIG{INT} to call a sub of your choice. As such a sub can be called at any stage in your loop it's good practise to simply note that ^C has been entered by setting a flag, and then checking at the end of each loop iteration whether or not there's a need to report. And don't forget to turn off that need to report once you have done the output - otherwise you'll report at EVERY iteratin thereafter!

Having used ^C to generate a report rather than kill your process, how can you actually kill it? There are other ways via the OS, but better to allow the user to enter ^C twice in succession, and on the second entry in (say) 3 seconds give an interactive "Quit?" option.

Net result of applying these techniques? A program that you can really control, and a great deal of time saved as you no longer have to wait for it to run to completion every time. Psychologically much better too - you know how it's getting on rather than having to wait and see.

On yesterdays's Perl for Larger Projects course, I wrote a complete example showing almost all of the above in a single program that analyses a massive web log file. I've commented it, so if you're already something of a Perl expert or have access to our Handling Huge Data course notes it wll already make sense to you. Source code Link. If this is something you need to learn, we would welcome you on the course

Here is some sample output to complete the example.

[trainee@campion huge]$ perl hugehunter

we have done 160000 lines ( 10.60%)

^C

Intermediate report

6 crawl-66-249-73-81.googlebot.com

1 crawler4067.ask.com

2 host86-148-48-188.range86-148.btcentralplus.com

1 livebot-65-55-209-97.search.live.com

1 lj511086.crawl.yahoo.net

1 lj511737.crawl.yahoo.net

2 lj511971.crawl.yahoo.net

1 lj512128.crawl.yahoo.net

1 mail.bezanilla.cl

[etc]

Quit? y

[trainee@campion huge]$

Note - example used logs to STDOUT not STDERR. You'll need to select STDERR before you set autoflush if you switch it over as recommended.
(written 2007-10-20, updated 2007-10-23) 5d5d

Associated topics are indexed under

P667 - Perl - Handling Huge Data
  [3375] How to interact with a Perl program while it is processing data - (2011-07-31)
  [3374] Speeding up your Perl code - (2011-07-30)
  [2834] Teaching examples in Perl - third and final part - (2010-06-27)
  [2806] Macho matching - do not do it! - (2010-06-13)
  [2805] How are you getting on? - (2010-06-13)
  [2376] Long job - progress bar techniques (Perl) - (2009-08-26)
  [1924] Preventing ^C stopping / killing a program - Perl - (2008-12-05)
  [1920] Progress Bar Techniques - Perl - (2008-12-03)
  [975] Answering ALL the delegate's Perl questions - (2006-12-09)
  [762] Huge data files - what happened earlier? - (2006-06-15)
  [639] Progress bars and other dynamic reports - (2006-03-09)

Back to
Using PHP to upload images / Store on MySQL database - security questions

Previous and next
or
Horse's mouth home

Forward to
Someone else's wedding

Some other Articles

Tcl - using [] or {} for conditions in an if (and while)
Dark Dawn
Pictures Framed
Someone else's wedding
Perl - progress bar, supressing ^C and coping with huge data flows
Using PHP to upload images / Store on MySQL database - security questions
Dont just convert to Perl - re-engineer!
Business to Business Networking - North and West Wilts / Melksham
First Alternative / what has happened there?
Autumnal lighting on a London trip

4086 posts, page by page

Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82 at 50 posts per page

This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

Public Training Courses

Running regularly at our UK training Centre.
[Schedule] - [About] - [Book]

Like this? ??