Re: [htdig] 3.1.5 -- Limiting Duration / Resources for HTDIG

Subject: Re: [htdig] 3.1.5 -- Limiting Duration / Resources for HTDIG
From: Gilles Detillieux (
Date: Wed May 24 2000 - 12:00:20 PDT

According to
> I am running into some situations where HTDIG appears to be going into an
> infinite loop. The pattern is that it either does, or does not, occur when
> searching a specific website; re-attempting the HTDIG--against the same
> site--usually produces the same result.
> Is there some straightforward way to limit the number of bytes which can be
> written into either wordlist or the log file; to ensure unconditional
> termination of processing once this limit is exceeded? (In practice, HTDIG
> is invariably still producing output when it's externally terminated; I
> haven't run into any instances of looping without producing output).

There's a server_max_docs config attribute that will limit the number of
documents htdig will index per server. However, you may want to run htdig
with one or more -v options to get some feedback on what it is doing on the
servers that seem problematic. It might give you some hints as to how you
can tweak your exclude_urls and limit_urls_to attributes to avoid indexing
certain branches on these servers that you'd rather avoid.

See for descriptions of all these attributes.

Gilles R. Detillieux              E-mail: <>
Spinal Cord Research Centre       WWW:
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Wed May 24 2000 - 09:48:57 PDT