Torsten Neuer (tneuer@inwise.de)
Wed, 2 Jun 1999 15:48:05 +0200
According to Geoff Hutchison:
>On Tue, 1 Jun 1999, Yang Yang wrote:
>
>> I couldn't search my site at all, when I run "rundig -vvv", I got a
>> message which said: "robots.txt line: Disallow:/ ". Is there a way to
>> modify htdig's source code to overcome this problem? I asked the system
>
>You could, but the resulting program would violate the standard for robots
>exclusion. If there's no other way and you really don't care about it, you
>can edit the htdig/Server.cc file to ignore the directives.
>
>> administrator of our site, he don't want to change the robots.txt file but
>> he has no objection for me to search the website.
>
>Sheesh. The least he could do is add a "User-Agent: htdig" section with
>"Disallow: " which would let you search but no one else.
Agreed. But there is still the possibility to write a dummy robots.txt
file and configure ht://Dig to use that file with the robotstxt_name
configuration directive. Yang's site is at a root directory level,
that should work, too ,-)
cheers,
Torsten
-- InWise - Wirtschaftlich-Wissenschaftlicher Internet Service GmbH Waldhofstraße 14 Tel: +49-4101-403605 D-25474 Ellerbek Fax: +49-4101-403606 E-Mail: info@inwise.de Internet: http://www.inwise.de------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig@htdig.org containing the single word "unsubscribe" in the SUBJECT of the message.
This archive was generated by hypermail 2.0b3 on Wed Jun 02 1999 - 06:12:08 PDT