Subject: Re: [htdig] Going for the big dig
From: Geoff Hutchison (ghutchis@wso.williams.edu)
Date: Mon Dec 18 2000 - 17:41:21 PST
At 10:14 AM +1100 12/19/00, Terry Collins wrote:
>And make sure you don't ignore robots.txt
Yes, though someone would need to alter the code to do this.
-- -Geoff Hutchison Williams Students Online http://wso.williams.edu/------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig-unsubscribe@htdig.org You will receive a message to confirm this. List archives: <http://www.htdig.org/mail/menu.html> FAQ: <http://www.htdig.org/FAQ.html>
This archive was generated by hypermail 2b28 : Mon Dec 18 2000 - 19:25:08 PST