htdig: Database Upgrade Help
Thu, 5 Nov 1998 13:20:26 -0800

We are running a *very* modified version of htdig-3.0.8b1, the code has
been heavily modified to do many extra things but these modifications mean
that we cannot easily upgrade to htdig-3.1.

We have hit what we are told are gdbm size limitations of 2GB. Running
htmerge fails in the last part of the merge (where it is counting up the
documents it has seen) with "Virtual memory exceeded in `new' ". We are not
out of memory as we have 5GB of swap and vmstat shows that we are only
using about 2.1GB of it.

htmerge: Sorting...
htmerge: Removing doc #0
htmerge: Merging...
htmerge: 312800:

htmerge: Total word count: 312867
Virtual memory exceeded in `new'

We are running under Solaris 2.6 on a Sun Enterprise 450/2250. We are
indexing about 550,000 documents, our file sizes (at the time the merge
failed) are:

957969723 Oct 27 19:34 db.docdb
67426370 Nov 4 21:48
953222215 Nov 4 17:43 db.wordlist
532695286 Nov 4 17:43 db.words.gdbm

To fix our current problem (and to get better performance) we would like to
change from gdbm to the Berkeley db but we cannot do it by applying the
patches as they were made against a standard system and ours is anything
but that.

Could someone give us some pointers as to which parts of the code we would
need to change in order to use the Berkeley db. Also any help with the
current problem as a short term fix would be much appreciated.

Thanks in advance,

Paul Lucas

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:28:45 PST