Re: [htdig3-dev] size of dynamic pages

Subject: Re: [htdig3-dev] size of dynamic pages
From: Gabriele Bartolini (
Date: Wed Feb 09 2000 - 23:31:50 PST

>>At the time I suggested that, it wasn't clear to me that there would
>>never be a content-length header when reading chunked input. I see
>>now that this is the case. However, the _content_length field should
>>reflect the total size of the original document, regardless of whether
>>htdig truncated it to max_doc_size, while _document_length should
>>reflect the size actually read it, not exceeding max_doc_size.

OK. That's absolutely clear :-)

>>If _content_length is less than _document_length, then _content_length
>>is surely incorrect, or wasn't given, as it should always be greater than
>>or equal to _document_length. So, I think the logic above still makes
>>sense. Certainly for non-chunked input, that's the way it should be
>>done. For chunked input, you may want to do it differently, but you
>>do have to allow for the possibility that _content_length will be larger
>>than _document_length (see below).

Of course ...

>>> Well, as far as ReadChunkedBody() is, there's another problem. We don't
>>> max_document_size attribute, and I have no idea on how to use it (but
>>> closing the connection as the size has been reached -- in this way I'll
>>> never know how much the content-length is). That sucks ;-(

Gabriele, why don't you think sometimes? :-D

>>Yes, it occurred to me late yesterday that this was a problem. It seems
>>to me that the only option is to read the entire chunked input, to get
>>the correct length, but only append a maximum of max_doc_size bytes to
>>the _contents string. Then, _document_length will be set to the length
>>of this string (i.e. it will not exceed max_doc_size), but _content_length
>>will be set to the total length of all chunks.

OK. That's easy to implement. But, for example, shouldn't we give the
possibility to the user to "stop" the reading from the connection stream
once we get to the max_document_size. Just an idea ... Imagine you set
max_document_size to 60000 Bytes have to get a 500000 Bytes document.
Probably you'd better close the connection. But in this case, for chunked
bodies, we will never know anything about the content-length. And with
persistent connections it becomes less efficient. I don't think it's a good
idea to close the connection ...

I get along with the simplest solution, that is to say empty the buffer of
the chunked body, so retrieve the content-length. And append only at
maximum ma_doc_size bytes to the response content.

>>That really clarified things. Thanks for the research.

Hey, you are welcome ... How many times have I been asking you about
things, and you always kindly and greatly answered. It's my pleasure!

>>Yes, that was our goal, so obviously if we want to support HTTP/1.1,
>>we must accept chunked input - which we do - but we must also make sure
>>max_doc_size is respected, even if that means reading and discarding
>>any extra output from the server.


>>It's not clear to me what these "entity-headers" in the trailer are, and
>>what we should do with them. Are they to be treated just like headers
>>received before the data? If so, I guess we should parse them. In any
>>case, it would seem the code should be there to deal with the trailer,
>>if any server puts one out, even if we don't have any examples right
>>now of servers that do.

You find me not ready on this ... I haven't understood anything by reading
this piece of article. Sorry ! :-(

Ciao Ciao


Gabriele Bartolini
Computer Programmer - supposed to be ;-)
U.O. Rete Civica - Comune di Prato
Prato - Italia - Europa


Luna Rossa Prada Challenge - America One 5-4 ;-D

To unsubscribe from the htdig3-dev mailing list, send a message to
You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Wed Feb 09 2000 - 23:35:00 PST