Re: htdig: Digging throug HTTP and wait between documents

Geoff Hutchison (
Tue, 8 Dec 1998 13:23:53 -0500

At 12:35 PM -0500 12/8/98, Gilles Detillieux wrote:
>A quick fix would be to add a sleep() call just before c.connect()
>in Document::RetrieveHTTP() (file htdig/ That would slow
>the whole dig down, whether it's accessing the same server repeatedly,
>or interleaving its requests.

This "quick fix" will actually be in 3.1.0b3.

>A proper fix would involve keeping track of the time each host was
>accessed last, and before any access to a host, if the last access was
>more recent than the number of seconds in some new config parameter,
>then it would sleep for the difference in time. By recording the time
>at each c.close(), and checking it before c.connect(), it would ensure
>a minimum idle time between each connection.

This is, of course, preferred. Since I'd like to introduce as little code
as possible into the tree before releasing 3.1.0b3 (hopefully by the end of
the week), the "quick fix" will have to do.

-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:29:49 PST