Re: [htdig] Capacities/Cababilities of htdig

Doug (
Wed, 03 Feb 1999 12:22:26 -0800

Aviram Carmi wrote:
> Hi all,
> First posting to the list, so please excuse any mistakes.
> I would like to know what are the limitations/capacity of htdig.
> I know that most of these would depend on the hardware platform and
> bandwidth, so could you kindly let me know what hardware platform and what
> bandwidth do you have.

        We use sun sparcs.

> - How many sites/pages can it index in a reasonable amount of time?

        This is entirely dependant on your definition of "reasonable." :) On
our system it takes approximately 2 seconds for htdig to index 3 pages
of html. Add more time for long and/or complex pages, and/or slow
network connections between you and the remote site.

> - How many user queries can it handle?

        We're averaging 1,000 searches per day with no visible load on the

> - What is the largest number of sites/pages that you index using htdig?

        How much disk space do you have? :) And that's not a frivolous
answer... theoretically there is no practical limit. Whether it blows up
at 284 million pages or not, no one knows... yet.

        As for response time, the more you add to your indexes the faster the
search, but then you're back to the disk space thing again.

Hope this helps,

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Wed Feb 10 1999 - 17:09:05 PST