[htdig] Rate limiting spidering?

Subject: [htdig] Rate limiting spidering?
From: Albert Lunde (Albert-Lunde@northwestern.edu)
Date: Thu Aug 10 2000 - 14:53:27 PDT

Is there a way to control something like the number of hits per
minute htdig makes on a give server?

One of our "wish list" items is that we'd like to spider small
servers more slowly and large servers more aggressively (given some
externally supplied list of what are "large" and "small" servers.

     Albert Lunde         Albert-Lunde@northwestern.edu (new address)
                          Albert-Lunde@nwu.edu (old address)

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig-unsubscribe@htdig.org You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Thu Aug 10 2000 - 04:54:26 PDT