Re: [htdig3-dev] Re: robots.txt bug (was [ANNOUNCE] ht://Dig

Subject: Re: [htdig3-dev] Re: robots.txt bug (was [ANNOUNCE] ht://Dig
Date: Tue Feb 08 2000 - 15:18:57 PST

> Found it!

 Great. Thanks a lot for this good lesson about robots.txt. I'm
downloading ~2000 robots.txt and store them in The list of servers that have robots.txt
files is and was extracted
from our search engine.
 This will, at least, give us an idea of what people actually use in their
robots.txt. My intuitive understanding was that each section was self
contained. Despite of that I did not suggest a solution to the reported
problem. I'm in favour of the most restrictive interpretation because I
think most people will think this way.
 I know for sure that some site use the Allow tag and would be in favour of
supporting it.


		Loic Dachary

24 av Secretan 75019 Paris Tel: 33 1 42 45 09 16 e-mail: URL:

------------------------------------ To unsubscribe from the htdig3-dev mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Tue Feb 08 2000 - 13:58:52 PST