Re: [htdig] Multiple servers in one database

Geoff Hutchison (
Mon, 7 Jun 1999 11:12:25 -0400 (EDT)

On Mon, 7 Jun 1999, Marcin Owsiany wrote:

> (or maybe it isn't - will htdig be able to read several-hundred-kilobytes-long line)?
> so i have to make multiple config files and run htdigs serially.

The biggest problem with having many URLs in the start_url is that htdig
will have to build the list of these URLs right at the beginning. This
means it will consume a lot of memory right from the start. See my next

(BTW, the comment about including them from a file is correct and would be
my suggestion for entering that many servers. You can also include
*multiple* files, just as you'd include multiple URLs.)

> the most important question is: do i have to make multiple databases and
> then merge them

This would probably be the fastest. Working with many servers introduces
lag time between each request. Work is underway on using HTTP/1.1
persistent connections in the development code, which may help slightly.

> or (preferably)
> make htdig append more and more information to the single file and then run
> htmerge once?

This might work, but I don't think it's been tested.

-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Mon Jun 07 1999 - 07:30:37 PDT