Re: [htdig] robots.txt line:Disallow: /


Geoff Hutchison (ghutchis@wso.williams.edu)
Tue, 1 Jun 1999 16:56:31 -0400 (EDT)


On Tue, 1 Jun 1999, Yang Yang wrote:

> I couldn't search my site at all, when I run "rundig -vvv", I got a
> message which said: "robots.txt line: Disallow:/ ". Is there a way to
> modify htdig's source code to overcome this problem? I asked the system

You could, but the resulting program would violate the standard for robots
exclusion. If there's no other way and you really don't care about it, you
can edit the htdig/Server.cc file to ignore the directives.

> administrator of our site, he don't want to change the robots.txt file but
> he has no objection for me to search the website.

Sheesh. The least he could do is add a "User-Agent: htdig" section with
"Disallow: " which would let you search but no one else.

-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word "unsubscribe" in
the SUBJECT of the message.



This archive was generated by hypermail 2.0b3 on Tue Jun 01 1999 - 13:13:41 PDT