Training, Open Source computer languages
PerlPHPPythonMySQLApache / TomcatTclRubyJavaC and C++LinuxCSS 
Search for:
Home Accessibility Courses Diary The Mouth Forum Resources Site Map About Us Contact
Net::FTP and forks, Win32

Posted by darrenward (darrenward), 18 July 2006
Hello All,

I'm having a problem with Net::FTP and Parallel::ForkManager.  What I am trying to do is download multiple files from an FTP server simultaneously, using forks.  The forks are handled by Parallel::ForkManager.

When I run my code it downloads the files but gives a Windows error, perl.exe memory could not be "written"\memory could not be "read".  It would appear that the Net::FTP objects are being destroyed before the file has finished downloading\fork completed, and so this error is being produced.  I have tried adding more tasks to the fork so that it would complete them after the get and then call the fork finish but it doesn't make any difference.  It does not appear to be a Parallel::ForkManager problem as I could not get normal forking to work either, same error messages.  

It's probably something really simple, or it is just not possible to use forks with Net::FTP on Win32.

Does anyone have any ideas, any help would be much appreciated?

Part of my test.pl
I have made another module, which uses Parallel::ForkManager, which this passes callbacks to.

Code:
use ForkManager;

#create NET::FTP objects
for (1..$number) {
   my $ftp = Net::FTP->new(___FTPSERVER___) or die "Can't connect to server: $@";
   #$ftp->debug(1);
   $ftp->login($username, $password) or warn "Could not login";
   push @ftpconnections, $ftp;
}

#Get list of files
my @lines = $ftpconnections[1]->ls('darren');


for (1..3) {
   my $forkingsub = sub {
       my $file = $lines[$_];
       print time . "\tFile: $$file\n";
       $ftpconnections[$_]->get($file, $file) or warn "problem getting file: $!";
       $ftpconnections[$_]->command('NOOP');
       print time . "\tFinished downloading the file $$file\n";
   };
   
   $fm->Fork($forkingsub);
}

#Had wait_for_all_children here but it made no difference, doesn't seem to wait.


ForkManager.pm  --> This is the one I created

Code:
package ForkManager;

use strict;
use Parallel::ForkManager;
use Data::Dumper;

$|++;

sub new {
   my $pkg = shift;
   
   my $self = {};
   bless $self, $pkg;  
   
   my $number = shift;  
   $self->{parallell} = Parallel::ForkManager->new($number);

   return $self;
};

sub Fork {
   my $self = shift;
   my $stufftodo = shift;
   
   if (my $pid = $self->{parallell}->start($_)) { #and next
       print time . "\tFork $pid is $_ " . Dumper $self->{parallell};
      # return;
   }
   else {  
       &$stufftodo;
   }
   print time . "\tFinished Child process $$ $_\n";
   print time . "\tCalling finish on the Child $$ $_\n";
   $self->{parallell}->finish($_);
}

sub waitforall {
   my $self = shift;
   print time . "\tWAIT FOR ALL\n";
   
   $self->{parallell}->wait_all_children;
}

1;




Posted by admin (Graham Ellis), 19 July 2006
Only 2 out of about 30 CPAN test results reported for Parallel::ForkManager is for a Windows based system ... the remainder are for *nix operating systems.

As I understand it, forking is one of those areas that's very much different in how it can be done between *nix and *dose, so perhaps there's issues there. But I do also wonder about my variables going out of scope in the parallel processing context, and I would suggest as a next stage writing a very simple test case to fork out two parallel processes without using transient variables and seeing if the problem persists.

P.S.  Any way you can test your code out on a Unix / Linux box?




This page is a thread posted to the opentalk forum at www.opentalk.org.uk and archived here for reference. To jump to the archive index please follow this link.

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2014: Well House Manor • 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01144 1225 708225 • FAX: 01144 1225 899360 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho