Subject: [htdig] what robots.txt actually does
From: Kari Suomela (ks@karicobs.com)
Date: Fri Dec 10 1999 - 17:58:45 PST
Friday December 10 1999 11:16, Gilles Detillieux wrote to Daniel
MacKay:
GD> I think the key sentence that Kari is looking for is this:
GD> The presence of an empty "/robots.txt" file has no explicit
That's the one! Thx for good responses! :)
KS
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-unsubscribe@htdig.org
You will receive a message to confirm this.
This archive was generated by hypermail 2b28 : Fri Dec 10 1999 - 09:58:48 PST