Re: [htdig] Going for the big dig


Subject: Re: [htdig] Going for the big dig
From: Geoff Hutchison (ghutchis@wso.williams.edu)
Date: Mon Dec 18 2000 - 17:41:21 PST


At 10:14 AM +1100 12/19/00, Terry Collins wrote:
>And make sure you don't ignore robots.txt

Yes, though someone would need to alter the code to do this.

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig-unsubscribe@htdig.org You will receive a message to confirm this. List archives: <http://www.htdig.org/mail/menu.html> FAQ: <http://www.htdig.org/FAQ.html>



This archive was generated by hypermail 2b28 : Mon Dec 18 2000 - 19:25:08 PST