Re: htdig: Feature request


Chuq Von Rospach (chuqui@plaidworks.com)
Mon, 14 Sep 1998 21:47:01 -0700


At 5:56 PM -0700 9/14/98, Geoff Hutchison wrote:

> Sorry, the robots exlusion standard people (and myself) are ahead of you.

> <meta name="robots" content="noindex">

Brings up an interesting point I've been sort of researching. What's
the best way to write up a robots.txt file that allows the local htdig
to index, but nobody else? I know using a user-agent of htdig should
allow it to index, but seems to allow remote htdig's also. I guess I
could disable robot.txt looking in my local copy, but even though I
trust myself not to abuse that...

--
Chuq Von Rospach (Hockey fan? <http://www.plaidworks.com/hockey/>)
Apple Mail List Gnome (mailto:chuq@apple.com)
Plaidworks Consulting (mailto:chuqui@plaidworks.com)
<http://www.plaidworks.com/> + <http://www.lists.apple.com/>
----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-request@sdsu.edu containing the single word "unsubscribe" in
the body of the message.



This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:27:48 PST