Why don't some servers use CRLF after the last block length?
I am working with an HTTP request tool (similar to cURL) and have a server response issue. Either that or my understanding of the RFC for HTTP 1.1 and chunked data.
What I see is the chunked data should be in this format:
4\r\n
Wiki\r\n
5\r\n
pedia\r\n
e\r\n
in\r\n\r\nchunks.\r\n
0\r\n
\r\n
what I actually see is the following:
4\r\n
Wiki\r\n
5\r\n
pedia\r\n
e\r\n
in\r\n\r\nchunks.\r\n
0
In other words, multiple servers I've tested sent more data after 0 .. not CRLF, much less CRLFCRLF.
How are we supposed to know that this is the end of fragmented data without the correct format for fragmented tags? Timeouts happen with CRLF lookups after 0 and that's not enough.
Use Content-Length, definitely when I know it; for file uploads, checking the file size is negligible in terms of resources. For packet transfers, we do not look at the message body for the CRLF pair. It first reads the specified number of bytes and then reads two more bytes to confirm that they are CR and LF. If it is not, the body of the message is badly formed and either the size was incorrect or the data was corrupted.
For more information read RCF which says
A server using channel-to-send encoding in its response MUST NOT use a trailer for any header fields if at least one of the following is true:
a) the request included a TE header field that indicates that "trailers" are acceptable in transmitting the encoding of the response, as described in section 14.39; or,
b) the server is the originating server for the response, the trailer fields are composed entirely of optional metadata, and the recipient can use the message (in an order acceptable to the origin server) without receiving this metadata. In other words, the origin server is willing to accept the possibility that trailer fields can be silently discarded along the way to the client.
Method for determining the length of the message body:
If the header has Transfer-Encoding and the packet transfer is final encoding, then the message body length is determined by reading and decoding the fragmented data until the transfer encoding indicates that the data is complete.
If the header has Transfer-Encoding, and the chunk transfer is not the final encoding, then the message body length is determined by reading the connection until it is closed by the server.
If the header has transfer coding in the request, and the channel transfer is final coding, then the length of the message body cannot be reliably determined; the server MUST respond with a 400 (Bad Request) status code and then close the connection.
If a message is received with both the Transfer-Encoding header field and Content-Length, Transfer-Encoding overrides Content-Length. Such a message may indicate an attempt to broadcast a request response and should be treated as an error. The sender MUST remove the received Content-Length before forwarding such a message downstream.