RE: [htdig] Quick Question

Geoff Hutchison (
Thu, 13 May 1999 11:28:58 -0400 (EDT)

On Thu, 13 May 1999, Brandon LaBonte wrote:

> We think about 250 servers, and probably 100,000 unqiue URL's....Does this
> exceed the reasonable capacity of htdig?

No, but it will take some time for an initial index. Each server slows
things down a bit with the added connections.

> Also, we are running this on a P2 with only 64 megs of Ram, and IDE Drives.
> We are trying to get the ram to 128 (maybe 256), and the drives to SCSI.
> Will this make a marked improvement in performance, or is this more an
> application issue?

You will almost definitely need more RAM. You can get by with virtual
memory, but you're going to be swapping your data a lot. I index around
70,000-75,000 URLs from one server using a P2 with 128MB RAM and a SCSI
drive. I get fairly good performance, but obviously updating the database
takes much less time than the first indexing.

-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Thu May 13 1999 - 08:39:00 PDT