RE: [htdig] suggestions for large multi-server indexing?

Subject: RE: [htdig] suggestions for large multi-server indexing?
From: Harold Sinclair (
Date: Fri Jun 09 2000 - 10:40:14 PDT


On Fri, 9 Jun 2000, Budd, S wrote:

> We index using 3.1.3
> htmerge: Total documents: 177428
> htmerge: Total doc db size (in K): 1676650
> from 181 servers. It takes about 28 hours
> with one initial dig and another update dig
> two hours later.

How are you doing the dig -- each server individually with its own
configuration file and then merging all into one big database at
the end? Or somehow else..?

> -----Original Message-----
> From: Albert Lunde []
> Sent: Friday, June 09, 2000 5:11 PM
> To:
> Subject: [htdig] suggestions for large multi-server indexing?
> (1) What are the are the pros and cons of doing a single big index
> (giving it starting URLs across all servers) vs. doing a number of
> small indexes and merging them?

I'd like to hear your comments on this!


To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Fri Jun 09 2000 - 08:31:13 PDT