Python: Huge file read with linecache. Regular files can be accessed ()
I got into a situation where multiple threads are viewing the same huge file with multiple file pointers to the same file. The file will contain at least 1 million lines. The length of each line varies from 500 characters to 1500 characters. No operations will be recorded in the file. Each thread will start reading the same file from different lines. What's the efficient way ...? Using Python Linecache or regular readline () or is there some other efficient way?
+2
a source to share
1 answer
Have a look at the module mmap
: http://docs.python.org/library/mmap.html
This will allow you to use the file as an array while the OS handles the actual reading and buffering.
+2
a source to share