htdig: no robots tag


simon (simon@earthlink.com.au)
Mon, 07 Sep 1998 22:31:02 +0930


Greetings from downunder,

Is there a way to turn the robot exclusions protocol off.

We have no robots tags to stop external bots on some pages but want
htdig to go in.

Thanks for all the work that has gone into this project.

Cheers,
Simon

----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-request@sdsu.edu containing the single word "unsubscribe" in
the body of the message.



This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:27:42 PST