Re: [htdig] reference count overflow

Subject: Re: [htdig] reference count overflow
From: Geoff Hutchison (
Date: Thu Jun 15 2000 - 16:34:59 PDT

At 4:20 PM -0700 6/15/00, John Caldwell wrote:

Upgrade. It doesn't fix anything database-related, but I wouldn't
want to keep insecure software up.

See <>

>Yes, I read that in the FAQ. This is on a Sun Netra T1 with 1gb ram, 1gb
>swap, and about 50gb of disk. The docdb file is only 184mb, and the index
>files are in the range of 15-20mb. The run of "htdig" completed just
>fine, and htmerge never bombed either.

In short, you don't worry so much about resources. :-)

>There's one message in the archive, with _no_ replies:

Funny. I seem to remember replying with a message similar to the one
I sent you. The list archives have sometimes sent messages to the
great void, unfortunately. (Especially since we've gone through a few
different MTAs...)

>My initial assumption was that it was some sort of configuration issue. I
>was not approaching this as a "bug report". I'm not trying to say "your
>software sucks, please fix it", I'm trying to say "I'm getting unepxected
>behavior with a large db". I have 3 other search indexes on the machine
>that work perfectly fine, but each of them is for 1000-2000 documents. I
>only saw this problem with the larger DB.

No, it's not a configuration issue. I also wouldn't call it "a large
db." Maybe by your standards, but 1000-2000 is the smallest size I
use for tests. I generally aim to be sure it can handle 80,000 or so
before I'm willing to release.

But *I'm* saying "if you're willing to help, I'd like to fix the problems.

>I'm willing to reindex from scratch, but how many times am I going to have
>to index 140,000 documents to get it to work? To prevent a complete and
>total DOS on the site I have the wait time set to 1 second.. at that rate
>it takes about 2 days to index all the documents. Kinda makes debugging

Do you need to go through HTTP? There's a lot less load when you use

But it's also slow to do debugging when I can't get to your machine
and try things out either. :-( Something that would help in the
meantime would be to apply the database diagnostic patch at
<> and try
running htsearch under gdb as mentioned in the comments at the
beginning of the patch. If the patch doesn't apply correctly, let me
know--I think it was garbled in the e-mail.

-Geoff Hutchison
Williams Students Online

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Thu Jun 15 2000 - 14:28:22 PDT