Re: [htdig] Rate limiting spidering?

Subject: Re: [htdig] Rate limiting spidering?
From: Geoff Hutchison (
Date: Thu Aug 10 2000 - 15:04:38 PDT

On Thu, 10 Aug 2000, Albert Lunde wrote:

> Is there a way to control something like the number of hits per
> minute htdig makes on a give server?

In the production code, no. In the 3.2 code, the config parser allows
per-server and per-URL attributes, though not all attributes are "turned
on." (Unfortunately, the 3.2 documentation indicates the current wishlist
and not whether they actually work this way yet.)

But to answer your question, yes server_wait_time is among those that
works in this fashion in the current 3.2 code. Unfortunately there's no
way to do it in 3.1.x w/o multiple config files.

-Geoff Hutchison
Williams Students Online

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Thu Aug 10 2000 - 05:04:41 PDT