Re: [htdig] Going for the big dig


Subject: Re: [htdig] Going for the big dig
From: Terry Collins (terryc@woa.com.au)
Date: Mon Dec 18 2000 - 19:42:30 PST


Geoff Hutchison wrote:
>
> At 10:14 AM +1100 12/19/00, Terry Collins wrote:
> >And make sure you don't ignore robots.txt
>
> Yes, though someone would need to alter the code to do this.

If you are doing an external site, it shouldn't be to much effort to
just read this and set the excludes.

Courtesy thing.

--
   Terry Collins {:-)}}} Ph(02) 4627 2186 Fax(02) 4628 7861  
   email: terryc@woa.com.au  www: http://www.woa.com.au  
   WOA Computer Services <lan/wan, linux/unix, novell>

"People without trees are like fish without clean water"

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig-unsubscribe@htdig.org You will receive a message to confirm this. List archives: <http://www.htdig.org/mail/menu.html> FAQ: <http://www.htdig.org/FAQ.html>



This archive was generated by hypermail 2b28 : Mon Dec 18 2000 - 20:53:42 PST