Subject: Re: [htdig3-dev] Re: robots.txt bug (was [ANNOUNCE] ht://Dig
Date: Tue Feb 08 2000 - 15:18:57 PST
> Found it!
Great. Thanks a lot for this good lesson about robots.txt. I'm
downloading ~2000 robots.txt and store them in
http://www.senga.org/htdig/robots/. The list of servers that have robots.txt
files is http://www.senga.org/htdig/robots/robots-list and was extracted
from our search engine.
This will, at least, give us an idea of what people actually use in their
robots.txt. My intuitive understanding was that each section was self
contained. Despite of that I did not suggest a solution to the reported
problem. I'm in favour of the most restrictive interpretation because I
think most people will think this way.
I know for sure that some site use the Allow tag and would be in favour of
-- Loic Dachary
24 av Secretan 75019 Paris Tel: 33 1 42 45 09 16 e-mail: email@example.com URL: http://www.senga.org/
------------------------------------ To unsubscribe from the htdig3-dev mailing list, send a message to firstname.lastname@example.org You will receive a message to confirm this.
This archive was generated by hypermail 2b28 : Tue Feb 08 2000 - 13:58:52 PST