Re: [htdig] Multiple merging

Geoff Hutchison (
Wed, 3 Nov 1999 00:02:16 -0600

At 11:23 AM +0100 11/2/99, Andrea Carpani wrote:
>I have dug separately 1200 sites and this operation took a few hours. The
>problem is that the merging of the whole data is taking more than 10 days.
>Is this normal? How long would it take to build a single database
>from a single
>dig (feeding ht://dig with the whole list of sites at once)?

It's hard to know what's "normal" or which option would be faster.
Remember we're all digging very different servers, pages, etc. For
example, you don't mention how many URLs you have or the size of your

I'm guessing the merging is taking a while because either (or both):
a) 1200 sites => many, many URLs => large databases
b) the machine you're using doesn't have much RAM and is swapping to merge

These are obviously intertwined. The amount of RAM you need is
related to the size of your databases...


To unsubscribe from the htdig mailing list, send a message to containing the single word unsubscribe in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Tue Nov 02 1999 - 22:12:35 PST