Re: HTDig & Robots.txt


David A. Karp (daaron@creativelement.com)
Mon, 29 Sep 1997 11:04:10 -0700


Got it - thanks for the response.

Is there any way to get some information on the new configuration file
directives associated with searching a local filesystem in 3.0.8b2? Thanks
in advance.

-David

At 09:44 AM 9/29/1997 -0700, you wrote:
>No, there isn't.
>You could take out the code from htdig/Retriever.cc
>(The easiest would be to change the "robots.txt" to something like
>"xrobots.txt")

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
David A. Karp daaron@creativelement.com
Creative Element http://www.creativelement.com/dak/
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=



This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:25:05 PST