Re: [htdig] Virtual Memory error

Subject: Re: [htdig] Virtual Memory error
From: Geoff Hutchison (
Date: Wed Nov 24 1999 - 07:19:44 PST

At 8:35 AM -0500 11/24/99, KHumpf@IITRI.ORG wrote:
>I've got a rather large page (over 200,000 links to very simple pages with
>no links), that I am trying to index. Whenever I run htdig against it, I
>eventually get the error "virtual memory exceeded in 'new'".
>I really don't want to split the page into lots of smaller ones because it's
>dynamically generated from a database so... is there any way I can index
>these files without getting that error?

Yes, but only if you increase the amount of swap space...

Here's the problem--in order to do the indexing, htdig must assemble
a TODO list of links. Since indexing that page just added 200,000
links to the TODO list, it needs to grab a bunch more memory. But
that's the problem it mentioned--it ran out of memory (in this case
swap), so it's dead.

Even splitting the links a little might help. Even better, you could
break the links into "batches" and use htmerge to merge the database
together later.

-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig mailing list, send a message to
You'll receive a message confirming the unsubscription.

This archive was generated by hypermail 2b25 : Wed Nov 24 1999 - 07:38:38 PST