Geoff Hutchison (email@example.com)
Mon, 7 Jun 1999 11:12:25 -0400 (EDT)
On Mon, 7 Jun 1999, Marcin Owsiany wrote:
> (or maybe it isn't - will htdig be able to read several-hundred-kilobytes-long line)?
> so i have to make multiple config files and run htdigs serially.
The biggest problem with having many URLs in the start_url is that htdig
will have to build the list of these URLs right at the beginning. This
means it will consume a lot of memory right from the start. See my next
(BTW, the comment about including them from a file is correct and would be
my suggestion for entering that many servers. You can also include
*multiple* files, just as you'd include multiple URLs.)
> the most important question is: do i have to make multiple databases and
> then merge them
This would probably be the fastest. Working with many servers introduces
lag time between each request. Work is underway on using HTTP/1.1
persistent connections in the development code, which may help slightly.
> or (preferably)
> make htdig append more and more information to the single file and then run
> htmerge once?
This might work, but I don't think it's been tested.
Williams Students Online
To unsubscribe from the htdig mailing list, send a message to
firstname.lastname@example.org containing the single word "unsubscribe" in
the SUBJECT of the message.
This archive was generated by hypermail 2.0b3 on Mon Jun 07 1999 - 07:30:37 PDT