[htdig] robots.txt line:Disallow: /


Yang Yang (yayst6+@pitt.edu)
Tue, 1 Jun 1999 16:26:41 -0400 (EDT)


Hello, everyone,

I couldn't search my site at all, when I run "rundig -vvv", I got a
message which said: "robots.txt line: Disallow:/ ". Is there a way to
modify htdig's source code to overcome this problem? I asked the system
administrator of our site, he don't want to change the robots.txt file but
he has no objection for me to search the website.

Thanks for your help.

Yang

University of Pittsburgh

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word "unsubscribe" in
the SUBJECT of the message.



This archive was generated by hypermail 2.0b3 on Tue Jun 01 1999 - 12:40:13 PDT