Re: [htdig] slow update

Geoff Hutchison (
Sun, 18 Jul 1999 20:19:04 -0400 (EDT)

On Sun, 18 Jul 1999, Frank Guangxin Liu wrote:

> the files). I would suggest to modify "htdig" so that
> once it finds a server is down it should "remember" this
> in the current digging session and skip digging this server
> instead of having numerous timeouts.

This is a good point.

> Even for an initial htdig, it should be better for htdig
> to "remember" if a server is down (either the server machine
> itself is down, or the httpd process on that machine is down).

Actually, it's just not very smart about it. If a server is down when it
tries to get the robots.txt file, it concludes the server is dead and
ignores it. It should also keep track of whether the server dies midway

-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Sun Jul 18 1999 - 16:36:10 PDT