Doing updates and Digging with HTDIG

Jason Bennett (
Tue, 25 Nov 1997 16:55:03 -0330 (NST)

Hi there. I've been having some trouble using the htdig function of your
search engine and was wondering if you could assist. I find that when making
the initial database, I get about 45,000 URL's and it often stops (using
verbose mode) for a couple of hours, then continues digging. Is there a reason
for this?

Here are the last 2 entries from the verbose listing. It's been stuck there
for about 2 hours now. It will probably continue, but to give you some idea of
time here, I started the rundig session yesterday in the morning and it is now
late evening the next day. Is this normal? The sites I have entered into the
htdig.conf file as start url's are all within 800 km's of my location (All on
an island in the North Atlantic). Internet connectivity is A.O.K. as I've
connected many times to these sites.

 size = 1635


Also, is there an update mode that I can run in a cron tab for htdig that will simply go out and check for functioning links? New sites that I add?

If I wish to remove a site, but it is still an active site, is there any way I can do this?

Thanks for all your assistance with this Jason


============================================================================ =[ Jason Bennett Few things are harder to put up with =[ Network Administrator than the annoyance of a good example. =[ Compusult Limited - Mark Twain (1835-1910) =[ Mount Pearl,NF =[ 709-745-7914

=[ ============================================================================

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:25:13 PST