Re: [htdig] Indexing huge websites (really Linux 2G max file size


Geoff Hutchison (ghutchis@wso.williams.edu)
Wed, 04 Aug 1999 14:24:05 -0400


marc@nozell.com wrote:
> Does anyone have any suggestions on how I can continue to offer a
> search of the full website?

There are a few options. You can try the bigfiles patch for Linux, get
an OS that doesn't have a 2GB restriction, or try a transparent
filesystem compression package.

> Perhaps someone has done work to split ht:/Dig's index over multiple
> files?

You can, but you cannot search the separate databases. See the previous
thread about distributed searching (e.g. Beowulf). If you have logical
ways of separating the data, this isn't a problem. For example, split
the databases based on mailing list.

-- 
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig@htdig.org containing the single word unsubscribe in the SUBJECT of the message.



This archive was generated by hypermail 2.0b3 on Wed Aug 04 1999 - 11:24:56 PDT