Data in linux FIFO seems to be lost
I have a bash script that wants to do some work in parallel, I did this by putting each job in a subshell that runs in the background. While the number of jobs running concurrently should be limited, I achieve this by first putting some lines in a FIFO, and then, before expanding the subshell, the parent script needs to read a line from that FIFO. Only after it receives the string can it unlock the subshell. Everything is working fine so far. But when I tried to read a line from a FIFO in a subshell, it seems that only one subshell can get a line, even if there are more lines in the FIFO. So I wonder why the other subshell (s) cannot read the line, even if there are more lines in the FIFO.
My test code looks something like this:
#!/bin/sh
fifo_path="/tmp/fy_u_test2.fifo"
mkfifo $fifo_path
#open fifo for r/w at fd 6
exec 6<> $fifo_path
process_num=5
#put $process_num lines in the FIFO
for ((i=0; i<${process_num}; i++)); do
echo "$i"
done >&6
delay_some(){
local index="$1"
echo "This is what u can see. $index \n"
sleep 20;
}
#In each iteration, try to read 2 lines from FIFO, one from this shell,
#the other from the subshell
for i in 1 2
do
date >>/tmp/fy_date
#If a line can be read from FIFO, run a subshell in bk, otherwise, block.
read -u6
echo " $$ Read --- $REPLY --- from 6 \n" >> /tmp/fy_date
{
delay_some $i
#Try to read a line from FIFO, __ only one subshell succeeds the following line. __
read -u6
echo " $$ This is in child # $i, read --- $REPLY --- from 6 \n" >> /tmp/fy_date
} &
done
And the output file / tmp / fy_date has the content:
Mon Apr 26 16:02:18 CST 2010
32561 Read --- 0 --- from 6 \n
Mon Apr 26 16:02:18 CST 2010
32561 Read --- 1 --- from 6 \n
32561 This is in child # 1, read --- 2 --- from 6 \n
There I expect a line like this:
32561 This is in child # 2, read --- 3 --- from 6 \n
But it never shows up and the child's process # 2 is blocked there until I issue:
echo something> /tmp/fy_u_test2.fifo
a source to share
Be aware that a FIFO on POSIX systems is essentially a named pipe. To move data through the pipe, one side needs a reader and the other side needs a writer, and when someone is shut down, the other loses its usefulness.
In other words, you cannot cat
on the fifo after another other reader exits, because the contents of the FIFO will disappear.
You might want to familiarize yourself with a regular file (and use file locking to make sure you sync your access to that normal file) or use a directory with multiple files in it, or even use shared memory or something similar (perhaps not in a shell script though). It all depends on what your ultimate goal is, in fact, what is the best way to do it.
a source to share
It seems to have something to do with the shell call read -u6. If my STDIN shell is closed when "read -u6" is issued, it tries to read 128 bytes from fd 6. But if STDIN is left untouched when "read -u6" is issued, it reads the bytes one by one until will meet "\ n". I discovered this weird action from "strace" where in the first case the call to "read -u6" caused the following syscall:
read(6, "0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n"..., 128) = 50
and in the latter case, the call to "read -u6" invoked the following syscall:
30371 16:27:15 read(6, "0", 1) = 1
30371 16:27:15 read(6, "\n", 1) = 1
The following is the test code:
#!/bin/bash
fifo_path="/tmp/fy_u_test2.fifo"
mkfifo $fifo_path
#open fifo for r/w at fd 6
exec 6<> $fifo_path
#comment or decomment the following line makes difference
exec 0>&-
process_num=20
#put $process_num lines in the FIFO
for ((i=0;i<${process_num};i++));do
echo "$i"
done >&6
delay_some(){
local index="$1"
echo "This is what u can see. $index \n"
sleep 10;
}
#In each iteration, try to read 2 lines from FIFO, one from this shell,
#the other from the subshell
for i in 1 2 3
do
date >>/tmp/fy_date
#If a line can be read from FIFO, run a subshell in bk, otherwise, block.
read -u6
echo " $$ Read --- $REPLY --- from 6 \n" >> /tmp/fy_date
{
delay_some $i
#Try to read a line from FIFO
# read -u6
echo " $$ This is in child # $i, read --- $REPLY --- from 6 \n" >> /tmp/fy_date
echo " $$ Again this is in child # $i, read --- $REPLY --- from 6 \n" >> /tmp/fy_date
echo "$i xx" >&6
# echo xx >&6
} &
done
#sleep 13
#wait
#read -u6
echo "$$ After fork, in parent, read --- $REPLY --- from 6 \n" >> /tmp/fy_date
a source to share
It could be a concurrency issue, with both subshells trying to read from the same fifo at the same time. Does this happen all the time?
You can try adding an instruction flock -x 6
or changing the delay for two subshells and see what happens.
BTW, I can confirm that with bash 3.2 and kernel 2.6.28, your code is working fine.
a source to share
I found that the data was left unread in the FIFO when the parent shell exited, being lost when the parent exited.
If I have the following code:
#!/bin/sh
fifo_path="/tmp/fy_u_test2.fifo"
mkfifo $fifo_path
#open fifo for r/w at fd 6
exec 6<> $fifo_path
process_num=9
#put $process_num lines in the FIFO
for ((i=0;i<${process_num};i++));do
echo "$i"
done >&6
for i in 1 2 3;
do
read -u6
done
After this code the command 'cat / tmp / fy_u_test2.fifo' gives nothing.
BUT if I have the following code.
#!/bin/sh
fifo_path="/tmp/fy_u_test2.fifo"
mkfifo $fifo_path
#open fifo for r/w at fd 6
exec 6<> $fifo_path
process_num=9
#put $process_num lines in the FIFO
for ((i=0;i<${process_num};i++));do
echo "$i"
done >&6
for i in 1 2 3;
do
read -u6
done
#__ notice this line __
sleep 60
After executing this code to run while it sleeps 60 seconds, the command 'cat / tmp / fy_u_test2.fifo' gives the following output:
$ cat /tmp/fy_u_test2.fifo 3 4 five 6 7 8
a source to share
For reasons explained in other answers, here you don't want to use a pipe, unless you can read and write from a pipe at the same time.
Therefore, it is advisable to use a different IPC facility or restructure the use of fifos so that the asynchronous process fills the pipe while the main process creates worker processes (or vice versa).
Here's how you can get what you want using a simple file as a queue:
#!/usr/bin/env bash
stack=/tmp/stack
> "$stack"
# Create an initial 5 spots on the stack
for i in {1..5}; do
echo >> "$stack"
done
for i in {1..10}; do
# Wait for a spot on the stack.
until read; do sleep 1; done
{
echo "Starting process #$i"
sleep $((5 + $i)) # Do something productive
echo "Ending process #$i"
# We're done, free our spot on the stack.
echo >> "$stack"
} &
done < "$stack"
Sidenote: This method is not ideal for unlimited work as it adds bytes to the stack file for every process that starts, which means the stack file grows slowly.
a source to share