Re: [htdig] robotstxt_name


Subject: Re: [htdig] robotstxt_name
From: Geoff Hutchison (ghutchis@wso.williams.edu)
Date: Sun Jan 07 2001 - 10:30:43 PST


At 9:25 AM -0600 1/7/01, Mike Paradis wrote:
>Is this directive so that htdig will know to look for robots.txt or a custom
>name file that does the same. Or, is this to make our digger look
>like something else?

It sets the name htdig uses when parsing the robots.txt file and only
that. So this will affect how htdig considers robot-specific
directives in the robots.txt file. This is useful, for example, when
you want to make directives specific to your copy of htdig, etc.

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig-unsubscribe@htdig.org You will receive a message to confirm this. List archives: <http://www.htdig.org/mail/menu.html> FAQ: <http://www.htdig.org/FAQ.html>



This archive was generated by hypermail 2b28 : Sun Jan 07 2001 - 10:53:04 PST