Perl fork does not work inside a loop read loop from a file
I run a while loop, reading each line in the file, and then process the line data up to the child. After N lines, I want to wait for the child processes to finish and continue on the next N lines, etc.
It looks something like this:
while ($w=<INP>) {
# ignore file header
if ($w=~m/^\D/) { next;}
# get data from line
chomp $w;
@ws = split(/\s/,$w);
$m = int($ws[0]);
$d = int($ws[1]);
$h = int($ws[2]);
# only for some days in the year
if (($m==3)and($d==15) or ($m==4)and($d==21) or ($m==7)and($d==18)) {
die "could not fork" unless defined (my $pid = fork);
unless ($pid) {
some instructions here using $m, $d, $h ...
}
push @qpid,$pid;
# when all processors are busy, wait for child processes
if ($#qpid==($procs-1)) {
for my $pid (@qpid) {
waitpid $pid, 0;
}
reset 'q';
}
}
}
close INP;
This does not work. After the first round of processes, I get a PID of 0, the @qpid array gets mixed up, and the file starts reading in (seemingly) random locations, jumping back and forth. The end result is that most of the lines in the file are read two or three times. Any ideas?
Thank you very much in advance,
S.
source to share
I am concerned that your algorithm is not very efficient:
Let the basic fork process process 1 to N.
If processes 2 through N finished before process 1, then no new processes will be started until process 1 ends.
Instead of trying to get your implementation details right, use Parallel :: ForkManager to get working code easily.
use strict;
use warnings;
use Parallel::ForkManager;
my $pm = Parallel::ForkManager->new($MAX_PROCESSES);
while( my $w=<INP> ) {
next if $w=~m/^\D/; # ignore file header
chomp $w;
next unless match_dates( $w,
{ m => 3, d => 15 },
{ m => 7, d => 18 },
{ y => 2008 }, # Added this to show match_dates() capability.
);
my $pid = $pm->start and next;
.. Do stuff in child here ..
$pm->finish; # Terminate child
}
close INP;
# Returns true if ANY of the supplied date templates matches for ALL keys defined in that template.
sub match_dates {
my $string = shift;
my %target;
@target{qw( m d y )} = split(/\s/,$string);
DATE:
for my $date ( @_ ) {
my @k = keys %$match;
my $count = 0;
for( @k ) {
next DATE unless $date{$_} == $target{$_};
$count++;
}
return 1 if $count == @k; # All keys match
}
return;
}
source to share