Re: [htdig] One solution for slow dig on Linux.

Subject: Re: [htdig] One solution for slow dig on Linux.
From: Geoff Hutchison (
Date: Tue Dec 21 1999 - 06:18:44 PST

On Tue, 21 Dec 1999, Sean Pecor wrote:

> <grin>. I had a fairly large development project that required a search
> engine capable of handling approximately 150,000 internal pages and 10,000+
> pages on thousands of external web servers. Despite being warned by the FAQ
> that htdig wasn't built for this task, I couldn't resist giving it a go;

Our of curiousity, where does it say this? I see:

1.2. Can I index the internet with ht://Dig?

No, as above, ht://Dig is not meant as an internet-wide search engine.
While there is theoretically nothing to stop you from indexing as much as
you wish, practical considerations (e.g. time, disk space, memory, etc.)
will limit this.

(BTW, we have lots of people indexing well above 150,000 URLs.)

-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Tue Dec 21 1999 - 06:33:38 PST