Can I use less memory when using DeflateStream?
My application needs to decompress files that contain many compressed Deflate blocks (as well as other types of compression and encryption). Memory profiling shows that the stream constructor is responsible for allocating most of the application's memory over its lifetime (54.19%, followed by DeflateStream.read at 12.96%, and less than 2% for the rest).
To put this in perspective, each file block is typically 4KiB (unpacked), and the DeflateStream constructor allocates slightly over 32KiB (presumably for a sliding window). The garbage collector has a field day, since all these gouging threads go on for almost no time (everyone leaves before the next one appears)! Goodbye cache efficiency.
I can keep using DeflateStream, but I'm wondering if there is a better alternative. Maybe a way to reset the thread and use it again?
a source to share
Two comments without any actual measurements to support this:
- I think you will find that the amount of time spent allocating (and zeroing) these temporary buffers is marginally close to the time spent actually decompressing.
- The fact that these buffers are very transient means that while it may be 50% of the memory for an application's lifetime, none of them exist concurrently. Note that this shouldn't hurt cache efficiency too much either ... I would guess that most of these buffers don't expect to be used in cache memory because the pages will expire very quickly.
In short, unless you have a measurable deflation flow problem (either in speed or in absolute memory), I would just keep using it ... it is better to use a solution you know than introduce another that can have a whole host of problems that are difficult to deal with.
a source to share
Do you have any performance issues or are you just worried about memory usage?
Most objects are short-lived, so memory management and the garbage collector are designed to efficiently handle short-lived objects. Many classes within a project are designed to be used once and then thrown away at a shorter time frame.
If you try to hang on objects, they are more likely to survive the garbage collection, which means they will be moved from one heap generation to the next. Heap generations are not just logical division of an object, but an object is actually moved from one area of memory to another. The garbage collector usually works with the principle of moving all living objects on the heap to the next generation and then simply emptying the heap, so long-lived objects are expensive rather than short objects.
Due to this design, it is normal for memory bandwidth to be high while actual memory usage remains low.
a source to share
There's a DeflateStream in DotNetZip , effectively replacing the built-in DeflateStream in the .NET BCL. Ionic.Zlib.DeflateStream has a configurable buffer size. I don't know if this will lead to better memory efficiency in your scenario, but it might be worth trying. Here's the doc .
I have not tested decompression, but rather compression. In my tests, I found limited returns on expanding the buffer size beyond 4k as the amount of data is compressed is compressed. On the other hand, you still get accurate and correct compression, although it is less efficient even if the buffer is 1024 bytes. I suppose you will see similar results in decompression.
In any case, the window size is not directly set from the open interface. But this is open source and you should be able to easily resize the default Wwindow if needed. Also, if you find it valuable, I can take the view window size request as a custom parameter in DeflateStream. I didn't expose it because nobody asked for it. But still?
You said you have a different compression. If you are doing Zlib or GZip, there is also ZlibStream and GZipStream in the DotNetZip package.
If you want to make Zip files, you need the complete DotNetZip library (Ionic.Zip.dll, ~ 400k). If you are just doing {Deflate, Zlib, GZip} Stream then there is Ionic.Zlib.dll which is about 90k.
DotNetZip is free, but donations are encouraged .
a source to share