.NET DB Query Without Allocations?
I was tasked with rewriting some libraries written in C # so that there are no assignments after the download is complete.
I just got one project that makes some DB requests via OdbcConnection every 30 seconds. I've always used .ExecuteReader (), which creates the OdbcDataReader. Is there some pattern (like the SocketAsyncEventArgs socket pattern) that allows the native OdbcDataReader to be reused? Or some other clever way to avoid highlighting?
I didn't bother to learn LINQ as all the dbs at work are Oracle based and the last one I checked was not an official Linq To Oracle vendor. But if there is a way to do this in Linq, I can use one of the third parties.
Update:
I don't think I have clearly stated the reasons for the no-allocation requirement. We have one critical thread, and it is very important that it does not hang. This is for a real-time trading application and we are seeing up to 100ms freeze for some Gen 2 collections (I also heard games are written the same in C #). There is one background thread that does some kind of consistency checking and runs every 30 seconds. He is making a db request right now. The request is quite slow (about 500ms to return with all data), but that's ok because it doesn't interfere with the critical thread. Unless a worker thread is allocating memory, this will cause the GCs to freeze all threads.
I was told that all libraries (including this one) cannot allocate memory after startup. Whether I agree with this or not, what is the demand from the people who sign the checks :).
Now there are obviously ways that I could get data into this process without allocation. I could set up another process and connect it to this using a socket. The new .NET 3.5 sockets have been specifically optimized to not allocate them at all using the new SocketAsyncEventArgs pattern. (Actually, we use them to connect to multiple systems and never see any GC of them.) Then you have a preallocated byte array that reads from the socket and walks through the data without allocating any lines along the way. (I am not familiar with other forms of IPC in .NET, so I am not sure if memory-allocated and named pipes are allocated or not).
But if there is a faster way to get that no-alloc request without doing all these problems, I would prefer that.
source to share
You cannot reuse IDataReader
(or OdbcDataReader
or SqlDataReader
or any equivalent class). They are intended to be used with only one request. These objects encapsulate a single recordset, so once you've got it and replayed it, it doesn't make sense anymore.
Making a device to read data is incredibly cheap anyway, vanishingly small compared to the cost of actually executing the request. I see no logical reason for this "no allocation" requirement.
I would go so far as to say that it is almost impossible to rewrite the library without allocating memory. Even something as simple as boxing an integer or using a string variable will allocate some memory. Even if it were possible to reuse the reader (which is not, as I explained), it would still have to send the query to the database again, which would require allocating memory in the form of preparing the query, sending it over the network, getting the results again, etc.
Avoiding memory allocation is simply not a practical goal. It is best to avoid certain types of memory allocation if and when you determine that a particular operation is using too much memory.
source to share
For such a requirement, are you sure a high level language like C # is your choice?
You cannot tell if the .NET library functions you are using are internally allocating memory or not. The standard does not guarantee this, so if they do not use distributions in the current version of the .NET framework, they may start doing so later.
source to share