Re: [htdig] multiple digging

Subject: Re: [htdig] multiple digging
From: Torsten Neuer (
Date: Tue Dec 28 1999 - 13:24:28 PST

"Kempen, Geert" wrote:
> Anyone knows if this has changed in 3.1.4b
> (couldn't find in the release notes)
> >> On Thu, 23 Jul 1998, Steffen Wieschalla wrote:
> >> Is it possible to index a very big site with more than one process and
> >> merge the output to one big database?
> > Not at the moment. This is on the TODO list for a future release.

It is not directly supported by Ht://Dig, but it is possible ;-)

The only thing you have to care about, is that you really need to
know all about the site. Then you are able to split up the search
process for this site using different start_url and limit_urls_to
directives in the configuration file for the indexer processes, run
them concurrently and merge them into a single database for the
site search facility. Should be no problem, if you are the webmaster
of the site...


InWise - Wirtschaftlich-Wissenschaftlicher Internet Service GmbH
Waldhofstraße 14                            Tel: +49-4101-403605
D-25474 Ellerbek                            Fax: +49-4101-403606
E-Mail:            Internet:

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Tue Dec 28 1999 - 12:13:19 PST