[htdig] Indexing huge websites (really Linux 2G max file size problem)

Wed, 4 Aug 1999 13:50:45 -0400


I maintain a very large archive (archiver.rootsweb.com) of almost
9,000 mailing lists with more being added every day. Currently we
have 900,000 messages (7.5G) which is also growing rapidly.

The problem I've run into is ht://Dig's index has become larger
Linux's 2G maximum file size.

Does anyone have any suggestions on how I can continue to offer a
search of the full website?

Perhaps someone has done work to split ht:/Dig's index over multiple

Any suggestions for other open source search engines that can split
over multiple index files?


Marc Nozell <marc @ nozell.com>                  http://www.nozell.com

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig@htdig.org containing the single word unsubscribe in the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Wed Aug 04 1999 - 10:54:47 PDT