Re: [htdig] reference count overflow

Subject: Re: [htdig] reference count overflow
From: John Caldwell (
Date: Thu Jun 15 2000 - 16:20:54 PDT

On Thu, 15 Jun 2000, Geoff Hutchison wrote:

> At 11:54 AM -0700 6/15/00, John Caldwell wrote:
> >I'm using htdig to index about 140,000 documents. The dig went fine, and
> >the merge went fine too... but now when I try to execute the search, I get
> >a "reference count overflow" error:
> >[snip]
> >Is this just too many docs for htdig to handle? I searched the archives
> >and found someone else with the same problem, but I didn't see any
> >replies.
> <sigh>
> FIrst off, it would really help if you mentioned what version of
> ht://Dig, what OS, where you got ht://Dig (i.e. source or package),
> compiler, etc.


SunOS 5.7 Generic_106541-10 sun4u sparc SUNW,UltraSPARC-IIi-cEngine

gcc 2.95.2

> Secondly, I'll mention what I said in the FAQ. There is
> no limit whatsoever to what ht://Dig can handle if you have proper
> resources.

Yes, I read that in the FAQ. This is on a Sun Netra T1 with 1gb ram, 1gb
swap, and about 50gb of disk. The docdb file is only 184mb, and the index
files are in the range of 15-20mb. The run of "htdig" completed just
fine, and htmerge never bombed either.

> I don't know why you didn't see any replies in the list archives...

There's one message in the archive, with _no_ replies:

> Nevertheless, normally when we get bug reports of this type, they
> seem completely transient. When we try to get debugging information
> from people, their response is typically "oh, it doesn't happen
> anymore, I'll let you know if it comes back." (NOTE: If you've sent a
> database report before and it's still happening, speak up!)

My initial assumption was that it was some sort of configuration issue. I
was not approaching this as a "bug report". I'm not trying to say "your
software sucks, please fix it", I'm trying to say "I'm getting unepxected
behavior with a large db". I have 3 other search indexes on the machine
that work perfectly fine, but each of them is for 1000-2000 documents. I
only saw this problem with the larger DB.

> So my first suggestion is to try reindexing from scratch. Do you
> still see it? If so, are you willing to apply a patch, compile in
> debugging code and run htsearch through gdb?

I'm willing to reindex from scratch, but how many times am I going to have
to index 140,000 documents to get it to work? To prevent a complete and
total DOS on the site I have the wait time set to 1 second.. at that rate
it takes about 2 days to index all the documents. Kinda makes debugging

To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Thu Jun 15 2000 - 14:11:23 PDT