Re: [htdig] Indexing a list of sites -- catching failures


Subject: Re: [htdig] Indexing a list of sites -- catching failures
From: D.J.Adams@soton.ac.uk
Date: Tue Aug 29 2000 - 04:24:48 PDT


>
> I have set up htdig so that every night, it indexes a long list of small
> web sites. In general, this works very well, but I've found that I have to
> be very careful when adding new sites to the end of the list.
>
> Some sites seem to cause htdig to fail. When this happens, htdig doesn't
> continue on with the rest of the list -- it simply skips to the next step
> in rundig. This means that I have to do some careful adding and substracting
> to the list of sites before I figure out what caused the index to fail.
>
> What I would like to do is to somehow index each site separately and have
> some kind of error log if htdig hits a site that it fails on (for whatever
> reason). Then, I would like for it to procede to the next site in the list,
> whether or not it failed to index the previous site.
>
> Thanks for any help,
> Todd Wallace

I do a weekly index of over 900 servers and have never had this happen.
What version of htdig are you using?

Also, I would recommend using the http_proxy attribute in the config
file if you possibly can.

-- 
 
David J Adams
<D.J.Adams@soton.ac.uk>
Computing Services
University of Southampton

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig-unsubscribe@htdig.org You will receive a message to confirm this. List archives: <http://www.htdig.org/mail/menu.html> FAQ: <http://www.htdig.org/FAQ.html>



This archive was generated by hypermail 2b28 : Tue Aug 29 2000 - 10:02:03 PDT