Re: [htdig] remove_bad_urls

Torsten Neuer (
Tue, 26 Oct 1999 08:25:47 +0200

Maria Cervantes wrote:
> Hi, I'm using htdig-3.1.2 and I have to index 66 servers, but sometimes
> some of them can't be reached because they are down, etc. So I never get
> the whole list of servers indexed.
> However, I wish to keep in the database of the htdig, the documents of
> the servers that coudn't be reach, but were indexed in a other time.
> I put in the config file remove_bad_urls: false
> I run htdig three days ago and I got:
> htdig: Run complete
> htdig: 66 servers seen:
> htdig: 162 documents
> I run htdig yesterday and I got:
> htdig: Run complete
> htdig: 65 servers seen:
> htdig: 0 documents
> I searched for words I know are in this url and I didn't get any
> result.
> Do you have any idea??
> Did I misunderstand the usage of remove_bad_urls??

Do you "update" using an initial dig ("-i" command line option)?
In this case "remove_bad_urls" won't come into effect.


InWise - Wirtschaftlich-Wissenschaftlicher Internet Service GmbH
Waldhofstraße 14                            Tel: +49-4101-403605
D-25474 Ellerbek                            Fax: +49-4101-403606
E-Mail:            Internet:

------------------------------------ To unsubscribe from the htdig mailing list, send a message to containing the single word unsubscribe in the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Mon Oct 25 1999 - 23:35:06 PDT