BOUNCE htdig: Admin request


owner-htdig@sdsu.edu
Thu, 20 Aug 1998 11:53:28 -0700 (PDT)


>From andrew@contigo.com Thu Aug 20 11:53:25 1998
Received: from mail2.noc.netcom.net (mail2.noc.netcom.net [199.183.9.3])
        by sdsu.edu (8.8.7/8.8.7) with ESMTP id LAA07812
        for <htdig@sdsu.edu>; Thu, 20 Aug 1998 11:53:20 -0700 (PDT)
From: plucas@frost.com
Received: from notes.frost.com (notes.frost.com [207.221.223.20])
        by mail2.noc.netcom.net (8.8.5/8.8.5) with SMTP id MAA26441
        for <htdig@sdsu.edu>; Thu, 20 Aug 1998 12:01:36 -0700 (PDT)
Received: by notes.frost.com(Lotus SMTP MTA v4.6.1 (569.2 2-6-1998)) id 88256666.00677A6E ; Thu, 20 Aug 1998 11:50:15 -0700
X-Lotus-FromDomain: FROSTUS
To: htdig@sdsu.edu
Message-ID: <88256666.0064EA72.00@notes.frost.com>
Date: Thu, 20 Aug 1998 11:50:57 -0700
Subject: Virtual Memory Exceeded
Mime-Version: 1.0
Content-type: multipart/mixed;
        Boundary="0__=lEmApGoUm3o8xEhH0eIZJfKfXD56QE6pRdM0Wrqje1zMD5KbeGdvspsk"
Content-Disposition: inline

--0__=lEmApGoUm3o8xEhH0eIZJfKfXD56QE6pRdM0Wrqje1zMD5KbeGdvspsk
Content-type: text/plain; charset=us-ascii
Content-Disposition: inline

I am hoping that someone can give me some help with this.

We have just moved from a Sun SPARC-5/110 running Solaris 2.5 with 256MB
RAM to a multi-processor Sun Enterprise 450/2250 running Solaris 2.6 with
640MB RAM. We have 575,000 fairly small (2-4k) documents.

On our old server it was taking about 48 hours to run htdig and htmerge but
it was completing OK.

Running on our new server htmerge failed with "gdbm fatal: malloc error" so
I increased swap space from 3GB (enough on the old server) to first 4GB and
then 5GB. I have edited the config file so that it creates its temporary
sort files in a different directory on a different disk from /tmp which I
have been told would prevent them from competing for the virtual memory.

Each time I rerun it htmerge fails with:

htmerge...
htmerge: Sorting...
htmerge: Merging...
htmerge: 339200:
--0__=lEmApGoUm3o8xEhH0eIZJfKfXD56QE6pRdM0Wrqje1zMD5KbeGdvspsk
Content-type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-transfer-encoding: quoted-printable

=DCberwachungs
htmerge: Total word count: 339250
Virtual memory exceeded in `new'

I asked our system admin to help and after running vmstat he told me...=

"I checked the vmstat logs at the times indicated. At about 19:11,
there was 5GB of swap available, and a lot of memory suddenly freed
up. This would indicate a process exited (either a normal
termination or it aborted).

<<Sort finishing?>>

At about 00:24, available swap space was still at about 2.9GB, then
it jumped back up to about 5GB (along with RAM freeing up), again
indicating a process exited.

<<Merge failing?>>

At this point, we need some support from the software vendor. I
don't know how this indexing application works, but is it limited to
signed 32-bit integers? That's a limit of 2GB, so it's possible the
indexing runs are exceeding the software's capacity. However, only
the developers could say for certain."

Can anyone shed any light on this?

Many thanks in advance,

Paul Lucas e-mail: plucas@frost.com
Systems Developer
Frost & Sullivan
Mountain View, California
=

--0__=lEmApGoUm3o8xEhH0eIZJfKfXD56QE6pRdM0Wrqje1zMD5KbeGdvspsk--



This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:27:17 PST