RE: [htdig] Virtual Memory error

Subject: RE: [htdig] Virtual Memory error
Date: Wed Nov 24 1999 - 09:54:35 PST

When I run "df -k" at my UNIX prompt, it tells me this for my tmp directory:

swap 159344 20 159324 1% /tmp

Do I need to adjust this allocation so I have more swap space on the device
itself... or can I tell htdig to use a different directory for swapping?


> -----Original Message-----
> From: Geoff Hutchison []
> Sent: Wednesday, November 24, 1999 10:20 AM
> To: KHumpf@IITRI.ORG
> Cc:
> Subject: Re: [htdig] Virtual Memory error
> At 8:35 AM -0500 11/24/99, KHumpf@IITRI.ORG wrote:
> >I've got a rather large page (over 200,000 links to very
> simple pages with
> >no links), that I am trying to index. Whenever I run htdig
> against it, I
> >eventually get the error "virtual memory exceeded in 'new'".
> >
> >I really don't want to split the page into lots of smaller
> ones because it's
> >dynamically generated from a database so... is there any way
> I can index
> >these files without getting that error?
> Yes, but only if you increase the amount of swap space...
> Here's the problem--in order to do the indexing, htdig must assemble
> a TODO list of links. Since indexing that page just added 200,000
> links to the TODO list, it needs to grab a bunch more memory. But
> that's the problem it mentioned--it ran out of memory (in this case
> swap), so it's dead.
> Even splitting the links a little might help. Even better, you could
> break the links into "batches" and use htmerge to merge the database
> together later.
> -Geoff Hutchison
> Williams Students Online

To unsubscribe from the htdig mailing list, send a message to
You'll receive a message confirming the unsubscription.

This archive was generated by hypermail 2b25 : Wed Nov 24 1999 - 10:07:24 PST