htdig: no all URLS included

Panagiotis Kappos (
Mon, 13 Apr 1998 16:57:02 +0300

We face the following problem with HTDig.
We want to index about 500 different web servers and we define the max
digging depth to 2. We have never managed to see the HTDig finish all sites
due to the fact that it stop indefinetely waiting response from a server.
Is the any way HTDig to timeout and continue to the next URL in the raw?
Also can separate the URLS in groups of 50, and then merge the databases?

Any insight on the above problem would be greatly appreciated.

Best Regards,
Panagiotis Kappos
To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:26:01 PST