Re: [htdig] robots.txt line:Disallow: /

Torsten Neuer (
Wed, 2 Jun 1999 15:48:05 +0200

According to Geoff Hutchison:
>On Tue, 1 Jun 1999, Yang Yang wrote:
>> I couldn't search my site at all, when I run "rundig -vvv", I got a
>> message which said: "robots.txt line: Disallow:/ ". Is there a way to
>> modify htdig's source code to overcome this problem? I asked the system
>You could, but the resulting program would violate the standard for robots
>exclusion. If there's no other way and you really don't care about it, you
>can edit the htdig/ file to ignore the directives.
>> administrator of our site, he don't want to change the robots.txt file but
>> he has no objection for me to search the website.
>Sheesh. The least he could do is add a "User-Agent: htdig" section with
>"Disallow: " which would let you search but no one else.

Agreed. But there is still the possibility to write a dummy robots.txt
file and configure ht://Dig to use that file with the robotstxt_name
configuration directive. Yang's site is at a root directory level,
that should work, too ,-)


InWise - Wirtschaftlich-Wissenschaftlicher Internet Service GmbH
Waldhofstraße 14                            Tel: +49-4101-403605
D-25474 Ellerbek                            Fax: +49-4101-403606
E-Mail:            Internet:

------------------------------------ To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Wed Jun 02 1999 - 06:12:08 PDT