Subject: Re: [htdig] Indexing a list of sites -- catching failures
Date: Tue Aug 29 2000 - 04:24:48 PDT
> I have set up htdig so that every night, it indexes a long list of small
> web sites. In general, this works very well, but I've found that I have to
> be very careful when adding new sites to the end of the list.
> Some sites seem to cause htdig to fail. When this happens, htdig doesn't
> continue on with the rest of the list -- it simply skips to the next step
> in rundig. This means that I have to do some careful adding and substracting
> to the list of sites before I figure out what caused the index to fail.
> What I would like to do is to somehow index each site separately and have
> some kind of error log if htdig hits a site that it fails on (for whatever
> reason). Then, I would like for it to procede to the next site in the list,
> whether or not it failed to index the previous site.
> Thanks for any help,
> Todd Wallace
I do a weekly index of over 900 servers and have never had this happen.
What version of htdig are you using?
Also, I would recommend using the http_proxy attribute in the config
file if you possibly can.
-- David J Adams <D.J.Adams@soton.ac.uk> Computing Services University of Southampton
------------------------------------ To unsubscribe from the htdig mailing list, send a message to firstname.lastname@example.org You will receive a message to confirm this. List archives: <http://www.htdig.org/mail/menu.html> FAQ: <http://www.htdig.org/FAQ.html>
This archive was generated by hypermail 2b28 : Tue Aug 29 2000 - 10:02:03 PDT