Subject: Re: [htdig] Virtual Memory error
From: Geoff Hutchison (email@example.com)
Date: Wed Nov 24 1999 - 07:19:44 PST
At 8:35 AM -0500 11/24/99, KHumpf@IITRI.ORG wrote:
>I've got a rather large page (over 200,000 links to very simple pages with
>no links), that I am trying to index. Whenever I run htdig against it, I
>eventually get the error "virtual memory exceeded in 'new'".
>I really don't want to split the page into lots of smaller ones because it's
>dynamically generated from a database so... is there any way I can index
>these files without getting that error?
Yes, but only if you increase the amount of swap space...
Here's the problem--in order to do the indexing, htdig must assemble
a TODO list of links. Since indexing that page just added 200,000
links to the TODO list, it needs to grab a bunch more memory. But
that's the problem it mentioned--it ran out of memory (in this case
swap), so it's dead.
Even splitting the links a little might help. Even better, you could
break the links into "batches" and use htmerge to merge the database
Williams Students Online
To unsubscribe from the htdig mailing list, send a message to
You'll receive a message confirming the unsubscription.
This archive was generated by hypermail 2b25 : Wed Nov 24 1999 - 07:38:38 PST