Re: [htdig] Re: 3.1.5 strange freeze problem (fwd)

Subject: Re: [htdig] Re: 3.1.5 strange freeze problem (fwd)
From: Peter L. Peres (
Date: Tue Oct 10 2000 - 13:19:12 PDT

On Mon, 9 Oct 2000, Andrew Scherpbier wrote:

>"Peter L. Peres" wrote:
>> >swapping
>> No, it was not swapping much just then but it is swapping a LOT when it
>> runs. I have 32 MB RAM and 150 MB swap (on several disks). The swap space
>> gets used up to 80-90 MB at times (when running X11 concurrently, with
>> surprising performance under the circumstances, and load >2.4 ;-). Of
>> course I was not doing that at the time.
>Does the htdig process get very big?

Size ~= 60 MB. What is 'very big' on 32 bit Unix ? 1 GB ? ;-)

>It would probably help to run htdig with a single '-v' option so that you can
>see what page it is working on when it stops. (Make sure you either redirect
>the output or run htdig from within 'script' because there will be a single
>line per web page and it sounds like you have a lot of pages...)

It runs with -iasv from a nohup script. I don't care about the size, I
have a lot of disk space now. As I said, NO data (html input etc) file was
open or being read by htdig when it froze. i.e. this happened after
practically the whole indexing took place. lsof would not lie to me. I
think that there is some dodgy pointer issue when freeing memory at the
end or something like this but I'd really appreciate some help as to where
to place spies in the source. Htdig is big enough to be a full time
project and I do not have that. I might try to unpack the library source
(esp. malloc.c) and then attach gdb and step through the code. This will
require some time as you imagine.

>If all else fails, you could also run indexes on smaller chunks of your tree
>and merge them together to get the full index.

How do I do that ? Run htdig with -ias on chunks using individual database
files (*.work01 ...) and then htmerge all the resulting databases ? Can
you indicate a sequence of commands that will implement this ? I do not
remember seeing anything like this on or in the documentation.

>Andrew Scherpbier, Chief Scientist
>Evoke Communications (
>8334 Clairemont Mesa Blvd #204, San Diego, CA 92111
>phone +1.858.278.2329 x115 fax +1.858.278.2502

thank you,


To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.
List archives: <>
FAQ: <>

This archive was generated by hypermail 2b28 : Wed Oct 11 2000 - 13:25:08 PDT