Re: [htdig] Indexing huge websites (really Linux 2G max file size

Geoff Hutchison (
Wed, 04 Aug 1999 14:24:05 -0400 wrote:
> Does anyone have any suggestions on how I can continue to offer a
> search of the full website?

There are a few options. You can try the bigfiles patch for Linux, get
an OS that doesn't have a 2GB restriction, or try a transparent
filesystem compression package.

> Perhaps someone has done work to split ht:/Dig's index over multiple
> files?

You can, but you cannot search the separate databases. See the previous
thread about distributed searching (e.g. Beowulf). If you have logical
ways of separating the data, this isn't a problem. For example, split
the databases based on mailing list.

-Geoff Hutchison
Williams Students Online

------------------------------------ To unsubscribe from the htdig mailing list, send a message to containing the single word unsubscribe in the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Wed Aug 04 1999 - 11:24:56 PDT