Improving I / O Performance in C ++ Programs [External Merge Sort]
I am currently working on a project related to external merge sort using replace-replace and k-way merge. I have implemented a project in C ++ [works on Linux]. Its very simple and right now only deals with fixed record sizes.
For reading and writing, I am using the (i / o) fstream classes. After running the program for several iterations, I noticed that
- I / O reader blocks for requests larger than 4KB (typical block size). Maximum buffer size exceeding 4K will result in performance degradation.
- Output operations do not seem to need buffering, linux seems to take care of buffering. So I take the write (write) instead of maintaining a dedicated write buffer and then flush them out right away using write (records []).
But the application performance seems to be slow. How can I improve performance? Do I have to support special I / O streams to take care of reading blocks, or are there existing C ++ classes that already use this abstraction? (Something like BufferedInputStream in java)
a source to share
This high-performance I / O is easiest to accomplish with mmap
. This gives the kernel much more freedom to perform I / O operations and to schedule CPU time for your application. For example, when you read in 1MB using ifstream, the kernel can only return when reading all the data. But with mmap (), data can be returned incrementally as it becomes available.
However, you must understand how this happens. Just because the data is in RAM doesn't mean you should treat it as random access. Don't serve it on std::sort
. This will touch random parts of the mmap area, causing the page to crash right and right. As a result, you will be forced to search for your hard drive to eliminate random page faults. Instead, mmap()
two inputs and combine them. Since the mmap command has told the kernel what data you will need in the future, the kernel will pass the data to you as quickly as possible, and your merge sort will result in a page error (i.e., Halt) when it is temporarily missing.
a source to share
Have a look at using the C low level I / O library. Http://www.linuxtopia.org/online_books/programming_books/gnu_libc_guide/Low_002dLevel-I_002fO.html or ftp://ftp2.developpez.be/developps/linux/alp/alp -apB-low-level-io.pdf
On Windows, I got a few VIs a long time ago to run 10x faster using open low level IO than using fopen.
You might not get the same performance advantage, I know it will be something.
a source to share
Threads are known for their performance issues when compared to the regular CI core. In fact, they act as "easy to use and suitable for different situations", but lack of performance. What I would do in your situation is go to C-style I / O, profiling, and then an action based on the profiling results.
a source to share