Re: ht://dig

Weston Houghton (
Thu, 5 Mar 1998 17:39:40 -0700

I was under the understanding that ht://dig always looked for a robots.txt
file, and that to avoid that I would have to change the source. Can you set
a preference to look for or ignore the robots.txt file in the configuration?

If so, I apologize, I shouldn't have bothered you with this question, but I
thought I could only change what the robot appeared to be.

Thanks again,

>Weston Houghton wrote:
>> Andrew,
>> Thanks for the great tool in ht://dig. I have a quick question for you. Is
>> it possible to have two versions of htdig around. I would like one for our
>> intranets that does not pay attention to the robots.txt. file, and one that
>> does for external use.
>> I see in the htdig-3.0.8b2/htdig folder there is the code. By
>> changing this can I compile it without the robots.txt functionality? How
>> could I then differentiate the two versions of htdig?
>You can have them use different config files. The same goes for the search
>You can specify the config file as a hidden parameter on the search forms.
>Andrew Scherpbier <>
>Contigo Software <>

| Weston Houghton \_ _/ _/ |
| AnaPraxis R+D \_ _/____/ |
| \_ \_ _/ _/ |
| \_ \_/ _/ |

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:25:49 PST