Subject: [htdig] Rate limiting spidering?
From: Albert Lunde (Albert-Lunde@northwestern.edu)
Date: Thu Aug 10 2000 - 14:53:27 PDT
Is there a way to control something like the number of hits per
minute htdig makes on a give server?
One of our "wish list" items is that we'd like to spider small
servers more slowly and large servers more aggressively (given some
externally supplied list of what are "large" and "small" servers.
--
Albert Lunde Albert-Lunde@northwestern.edu (new address)
Albert-Lunde@nwu.edu (old address)
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-unsubscribe@htdig.org
You will receive a message to confirm this.
This archive was generated by hypermail 2b28 : Thu Aug 10 2000 - 04:54:26 PDT