[htdig] slow update

Frank Guangxin Liu (frank@ctcqnx4.ctc.cummins.com)
Sun, 18 Jul 1999 16:58:39 -0500 (EST)

Normally the update htdig is pretty fast but this time
it took several days. A further research shows that a
major web server was down during the update dig. It appears
to me that htdig tries to update earch url to that server
and timeout on earch url. Since that is a major server and
has tens of thousands files, it took a long time for htdig
to check through (update) all of them (and then timeout each of
the files). I would suggest to modify "htdig" so that
once it finds a server is down it should "remember" this
in the current digging session and skip digging this server
instead of having numerous timeouts.

Even for an initial htdig, it should be better for htdig
to "remember" if a server is down (either the server machine
itself is down, or the httpd process on that machine is down).


To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word "unsubscribe" in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Sun Jul 18 1999 - 14:18:28 PDT