Re: [htdig3-dev] size of dynamic pages

Subject: Re: [htdig3-dev] size of dynamic pages
From: Gilles Detillieux (
Date: Mon Feb 07 2000 - 13:38:17 PST

According to Geoff Hutchison:
> At 7:37 PM +0200 2/6/00, Valdas Andrulis wrote:
> >GH> At 11:22 PM +0200 2/5/00, Valdas Andrulis wrote:
> >GH> >When indexing dynamicaly generated documents it shows document size
> >GH> >is 0. I undestand that the size is unknow at the begingining, but after
> >GH> >retrieval of document it should be known or i am wrong?
> >GH>
> >GH> Which version is doing this? 3.2.0b1?
> >
> >Yes version 3.2.0b1.
> >
> >I will send output log for 3.2.0b1 and for 3.1.4 versions in
> >another letter.
> I took about ten seconds to look at this, but I think the problem is
> that the server doesn't send a Content-Length: header and in this
> case, I think the new HtHTTP code has a bug--if the header is not
> sent, it doesn't set the document size after the transfer. I would
> assume the code should set the content-length to the amount in the
> buffer.
> (I seem to remember we ran into something similar in 3.1.0b1.)

Actually, it was the opposite problem, when going from 3.1.0b4 to
the final 3.1.0 release. Older versions didn't even recognise the
Content-Length header, and so with some servers htdig didn't handle the
end of the input stream correctly because it was trying to read more
than was available. It also reported the file size as max_doc_size if
this was smaller than the content-length.

The new code obviously assumes the Content-Length header will always be
given - a very wrong assumption. The code in ReadBody seems to work as
the old code did, where it will ignore a negative content_length setting
(which meant it didn't see the header). However, the HtHTTP_Response
constructor doesn't initialise it to -1. I think there will be a few
other tests for this that will need to be reinserted as well.

This and the robots.txt problem are good examples of why this release
is not recommended for production use yet.

Gilles R. Detillieux              E-mail: <>
Spinal Cord Research Centre       WWW:
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

------------------------------------ To unsubscribe from the htdig3-dev mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Mon Feb 07 2000 - 15:48:59 PST