The MTU Discovery Path
I am developing an application that processes (video processing, etc.) and sends large files (up to tens of gigabytes) over a network. I am sending files using FTP. To improve the performance / memory consumption of the application, I would like to optimize the buffers so that I do not send too large and fragmented packets. The problem is that I don't have a lot of RAM to store the file data during upload. Basically, I read enough bytes from disk, process it, and send it straight to its destination. I am currently looking to implement MTU path discovery.
I am familiar with the basic concept of how to do this. I would like to know if there is any .NET API on Windows that tracks MTU to destination?
I assume there is no such thing, but a friend of mine told me that Windows Vista is tracking.
I am developing this application for Windows XP, but I would like to know if Windows has such a network tracking API.
a source to share
winsock does not support reporting MTU detected, although other TCP / IP stacks (such as AIX via the IP_GETPMTU socket option). Since winsock cannot communicate this, .NET cannot provide an API (which should be on top of winsock).
I recommend sending data in 64kiB chunks. This is the maximum IP packet size and is probably larger than the MTU, so the stack will send multiple full segments. The last chunk may be smaller, but then the system may delay sending it (because it still needs to get acknowledgments for earlier data), so if you quickly follow the next 64kiB send, the system will merge the chunks into packets again using the mtu path ...
a source to share
If you are using UDP you might want fragmentation, but if so why are you using your own protocol to transfer large files? Use TCP instead and don't worry about it. If you are using your own protocol over UDP with congestion control and all the necessary stuff, then set the DF bit on the packets and process the responses.
If you are using TCP you don't have to worry about fragmentation at all, the stack does it for you.
a source to share
Use FtpWebRequest
and FtpWebResponse
. The point is that these classes do not use huge buffers, but streams (which are implemented using the largest buffer size). Memory usage will be minimal.
The packet size is not under your control, but under the network driver.
If you need maximum speed then implement data transfer / receive via class TcpClient
on the server and client side.
UPD: Here is an example of how files should be uploaded: http://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest.aspx See class AsynchronousFtpUpLoader
.
One more thing. I experimented with MTU a while ago. All my changes reduced the data transfer rate. Windows network driver is smart enough to use the best MTU.
a source to share
PMTU Discovery is used by your TCP stack only to optimize packet size when sending. At the level, you only see the stream (i.e. Before batching, even happens, let alone fragmentation). What you are really the problem is, it looks like you are trying to send data as best you can, however you are connecting to TCP / IP slower so this causes the space it takes to create the ram. Since you are using FTP, I would expect the FTP implementation you are using already supports and understands this ??? Since you are requesting the buffers, it looks like a roll you are using socket based or something. If you are using synchronous dispatch methods when the socket buffer is full, it should block the send until it has buffer space.
Can you provide more details on which API you are using to send the file, Builtin vs. roll you have? Whether it's a socket, network stream, etc.
a source to share