Should I use multiple threads in this situation? [Ruby]
I open multiple files and process them one line at a time. Files contain tokens that share data, so sometimes processing one file may wait while others catch the same token.
I did this at first with just one stream and with an array indicating with true / false if the file should be read in the current iteration or if it should wait for some of the others to catch up.
Will using streams make it easier? More efficient? Does Ruby have a mechanism for this?
a source to share
I'm not sure if using streams in ruby is beneficial. I recently wrote and tested an application that was supposed to do parallel computations, but I didn't get what I expected even on a quad core processor, it did computations sequentially, one thread after another. Read this article article , she discusses thread scheduling, it may be that things have not changed, at least for the original ruby.
a source to share
First, threads never make anything easier. Threading is only applicable for accelerating applications. Threading introduces many new complications, it may seem convenient to describe multiple threads of execution, but it always makes life more difficult.
Second, premature optimization is the root of all evil. Do not try to speed up file processing unless you know what the bottleneck is. Do the simplest thing that could work (but not easier).
Third, streams can help if the file reading process is independent so that the stream can process the file without worrying about what other streams are doing. It doesn't appear to be the case in your case. Since different threads need to communicate with each other, you are unlikely to see a speed advantage when using threads.
Fourth, I do not know Ruby and therefore cannot comment on what mechanisms it has.
a source to share