RE: [htdig] suggestions for large multi-server indexing?


Subject: RE: [htdig] suggestions for large multi-server indexing?
From: Harold Sinclair (hsinclai@linux1091.dn.net)
Date: Fri Jun 09 2000 - 10:40:14 PDT


Hi,

On Fri, 9 Jun 2000, Budd, S wrote:

> We index using 3.1.3
>
> htmerge: Total documents: 177428
> htmerge: Total doc db size (in K): 1676650
> from 181 servers. It takes about 28 hours
> with one initial dig and another update dig
> two hours later.

How are you doing the dig -- each server individually with its own
configuration file and then merging all into one big database at
the end? Or somehow else..?

> -----Original Message-----
> From: Albert Lunde [mailto:Albert-Lunde@nwu.edu]
> Sent: Friday, June 09, 2000 5:11 PM
> To: htdig@htdig.org
> Subject: [htdig] suggestions for large multi-server indexing?
 
> (1) What are the are the pros and cons of doing a single big index
> (giving it starting URLs across all servers) vs. doing a number of
> small indexes and merging them?

I'd like to hear your comments on this!

-Harold

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-unsubscribe@htdig.org
You will receive a message to confirm this.



This archive was generated by hypermail 2b28 : Fri Jun 09 2000 - 08:31:13 PDT