Home Accessibility Courses Diary The Mouth Forum Resources Site Map About Us Contact
Controlling multiple asynchronous processes in Perl

Running processes in parallel

Perl is a great glueware language which includes a number of facilities that allow you to run other processes from your Perl script - for example, you can open a file using a | character in the file handle, and you can run another process by calling it up in backquotes.

Examples

open (PRHI,"du -sk * |"); # Report disc use to me
open (PRHO,"| mail ... "); # mail the output on PRHO
$weblet = `wget http://www.wellho.net/index.html`;
           # Read a web page

In all these cases, though, you start off writing synchronised code in which you need to be aware of when to write to, or read from, the other process. You're also running just running your single Perl process so you've only got a single flow.

FORKING

The fork() function allows you to divide your Perl process into two - each identical except for the value that's returned by the fork call itself. Once you've forked, your (now) two processes can each go off down their own route in the logic.

When you fork, one of the processes retains the process id that the original pre-fork process used, and the fork() function returns to it the new process id that's associated with the other process. This process is known as the parent.

The other process, know as the child process, gets a new process id assigned to it and a zero return value from the fork. If it need to know its parent's process if for later on, that id should be saved PRIOR TO the fork. Thus:

$parent = $$;
unless ($pid = fork()) {
                # Child process goes here
  # $parent is parent and $$ is child
                exit();
                }
# Parent process goes here
# $pid is child, $$ is parent

Forking is only half of the story, though. Once you've got two or more processes running, under most circumstances they'll need to be able to communicate with one another. Typically a child process that's running a task will need to communicate with a parent to return results, or to let the parent know that the child task has completed. And you need to select your mechanism for this communication very carefully - you certainly don't want to put any of the processes into a tight checking loop that would run to the severe detriment of other processes running on the same c.p.u,

COMMUNICATING VIA PIPES

If you open a PIPE before you fork your process, you're setting up a read and a write file handle with the write linked to the read. When you fork, the write file handle of the parent is attached to the read file handle of the child, and the read file handle of the parent to the write file handle of the child. In other words, the parent can write a message to the pipe that the child will be able to read, and the child can write a message to the pipe that the parent will be able to read.

For a parent forking just a single child, simple file handles can be used and the scheme is easy to work. For a parents that's forking multiple children, you really need the parent to have a list of file handles and unfortunately this isn't possible - you have to use a "typeglob".

Here's an example of a parent process that's forking a whole list of children:

@waitlist = (10,20,16,12,8,22);
foreach $item (@waitlist) {
        pipe *{$item},RETHAND;
        unless ($pid = fork()) {
                # Child process
                sleep $item;
                print RETHAND "Completed $item\n";
                exit();
                }
        }

Each of the children is going to run a sleep command - that's an example of the processing that you would do in there - and then prints its results back to the parent.

Using the code above, each of the children will be asleep at the same time, for anywhere between 8 and 22 seconds.

Let's then wait for the children to wake up:

foreach $item (@waitlist) {
        $response = <$item>;
        $now = localtime();
        print "$now - $response";
        }

That works well enough - after the longest sleep time of 22 seconds, all the child processes will have completed and the parent process will then exit.

fire:~/feb06 grahamellis$ perl pingalong
Started Sun Feb 26 15:52:10 2006
Forked by Sun Feb 26 15:52:10 2006
Sun Feb 26 15:52:20 2006 - Completed 10
Sun Feb 26 15:52:30 2006 - Completed 20
Sun Feb 26 15:52:30 2006 - Completed 16
Sun Feb 26 15:52:30 2006 - Completed 12
Sun Feb 26 15:52:30 2006 - Completed 8
Sun Feb 26 15:52:32 2006 - Completed 22
Completed Sun Feb 26 15:52:32 2006
fire:~/feb06 grahamellis$

Complete Source of this example
That's OK, but not brilliant. If you look at the timing of each of the completed processes, you'll see that the parent waited for the children in the order they were forked - so the parent is waiting for some of the slower children to complete while the faster but younger ones are waiting for attention.

ALERTING A PROCESS VIA A SIGNAL

If a process wants to communicate with another asynchronously (i.e. unexpectedly), it can do so via a signal.

The receiving process must first be set up to allow it to receive signals; this is done by setting an appropriate member of the %SIG hash. There are around 30 different signals available, but most of them already have other uses and we suggest you use the elements called USR1 or USR2. Thus:

$SIG{USR1} = "doneit";

sub doneit {
        $gotone = 1;
        }

in the source code of the parent / prior to the fork. This means that any process receiving a USR1 signal will simply set the $gotone variable to 1 so that the code in that receiving process can handle it in due course. It is important to limit the code in the signal handler function to almost nothing, since the signal can be received at any time and more complex code can lead to very nasty synchronisation and locking problems.

Also in the parent code, then, you'll check for the signal being received; since our parent isn't doing any processing in its own right, we'll put it into a slow sleep loop:

while (! $gotone) {
                sleep 1;
                }

Now all the child need to do is to use a kill command (yes, killing is not fatal to either child or parent) to send the signal:

kill "USR1",$parent;

and the parent will be alerted.

How does the parent know which child process has contacted it, though, if all it receives is a nondescript tug on the sleeve? It has to look and find out.

The first way you do this, which is somewhat clunky and inefficient if there's a lot of signalling going on, is through a temporary file. This mechanism, though, is perfectly adequate for a series of processes where there isn't much communication - for example, a setup where a Perl script is monitoring 20 or 30 long-running tests.

The child need to write a message to the parent BEFORE it signals:

open (FH,">zz$item.tmp");
print FH "Completed $item\n";
close FH;
kill "USR1",$parent;

and the parent then looks for, reads, deletes any signal files:

@temps = <zz*.tmp>;
foreach $rfile (@temps) {
        open (RF,$rfile);
        $response = <RF>;
        close RF;
        unlink $rfile;
}

If you run our previous example, with a whole series of faster and slower children spawned, under this new scheme, there's great news. The children that only sleep for a few seconds alert the parent and are dealt with while the others continue to slumber - even if the short-sleep children were started after the slower ones:

fire:~/feb06 grahamellis$ perl pinga2
Started Sun Feb 26 16:22:39 2006
Forked by Sun Feb 26 16:22:39 2006
Sun Feb 26 16:22:47 2006 - Completed 8
Sun Feb 26 16:22:49 2006 - Completed 10
Sun Feb 26 16:22:51 2006 - Completed 12
Sun Feb 26 16:22:55 2006 - Completed 16
Sun Feb 26 16:22:59 2006 - Completed 20
Sun Feb 26 16:23:01 2006 - Completed 22
Completed Sun Feb 26 16:23:01 2006
fire:~/feb06 grahamellis$

Complete Source of this example
READING BACK ASYNCHRONOUSLY VIA SIGNALS AND PIPES

Temporary files aren't a great way of signalling on a busy system - it's much easier to write information to pipes in the children and have the parent check the pipe. However, any attempt to read from the pipe without appropriate causation would result in the reader waiting until data is available - the very problem we had in the first example and managed to alleviate in the second.

If we use a combination of signals and pipes, and Perl's select command, we can have each child process write back on a much more efficient pipe, and then signal the parent to say that it's done so. The parent can then look around with the select and see where the message is from.

Here's the code to reap all the sleeping children from that last example:

while ($kids > 0) {
        while (! $gotone) {
                sleep 1;
                }
        $gotone = 0;
        foreach $item (@waitlist) {
                $rin = $win = "";
                vec($rin, fileno(*{$item}), 1) = 1;
                $ein = $rin ;
                if (select($rin,$win,$ein,0)) {
                        sysread($item,$response,40);
                        $now = localtime();
                        print "$now - $response";
                        close $item;
                        $kids--;
                        }
                }
        }

A little more complex - and note the use of sysread rather than <> or read to overcome any buffering issues. It runs quickly, though, and all the children are dealt with within a second of them finishing, even if the finish out of any forecastable order.

fire:~/feb06 grahamellis$ perl pinga3
Started Sun Feb 26 16:31:01 2006
Forked by Sun Feb 26 16:31:01 2006
Sun Feb 26 16:31:09 2006 - Completed 8
Sun Feb 26 16:31:11 2006 - Completed 10
Sun Feb 26 16:31:13 2006 - Completed 12
Sun Feb 26 16:31:17 2006 - Completed 16
Sun Feb 26 16:31:21 2006 - Completed 20
Sun Feb 26 16:31:23 2006 - Completed 22
Completed Sun Feb 26 16:31:23 2006
fire:~/feb06 grahamellis$

Complete Source of this example

See also Perl for Larger Projects course

Please note that articles in this section of our web site were current and correct to the best of our ability when published, but by the nature of our business may go out of date quite quickly. The quoting of a price, contract term or any other information in this area of our website is NOT an offer to supply now on those terms - please check back via our main web site

Related Material

Perl - Interprocess Communication
  [3940] Run other processes from within your Perl program - (2012-12-03)
  [3412] Handling binary data in Perl is easy! - (2011-08-30)
  [3011] What are .pid files? - (2010-10-23)
  [3010] Children, zombies, and reaping processes - (2010-10-23)
  [2970] Perl - doing several things at the same time - (2010-09-25)
  [2694] Multiple processes (forking) in Python - (2010-03-25)
  [1918] Perl Socket Programming Examples - (2008-12-02)
  [604] Perl - multiprocess applications - (2006-02-13)

resource index - Perl
Solutions centre home page

You'll find shorter technical items at The Horse's Mouth and delegate's questions answered at the Opentalk forum.

At Well House Consultants, we provide training courses on subjects such as Ruby, Lua, Perl, Python, Linux, C, C++, Tcl/Tk, Tomcat, PHP and MySQL. We're asked (and answer) many questions, and answers to those which are of general interest are published in this area of our site.

1 unpublished comment pending on this page

edit your own (not yet published) comments

© WELL HOUSE CONSULTANTS LTD., 2014: Well House Manor • 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01144 1225 708225 • FAX: 01144 1225 899360 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/solutions/perl-con ... -perl.html • PAGE BUILT: Wed Mar 28 07:47:11 2012 • BUILD SYSTEM: wizard