Re: [htdig] Going for the big dig


Subject: Re: [htdig] Going for the big dig
From: Terry Collins (terryc@woa.com.au)
Date: Mon Dec 18 2000 - 15:14:08 PST


> > 1) Is ht://Dig able to efficiently index external sites?

> Depends mostly on the response of the site and the speed of your network
> connection! It's also usually a very good idea to be a good neighbor and
> ask the sysadmin of the box as far as typical low-usage periods, set
> server_wait_time high, etc.

And make sure you don't ignore robots.txt

--
   Terry Collins {:-)}}} Ph(02) 4627 2186 Fax(02) 4628 7861  
   email: terryc@woa.com.au  www: http://www.woa.com.au  
   WOA Computer Services <lan/wan, linux/unix, novell>

"People without trees are like fish without clean water"

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig-unsubscribe@htdig.org You will receive a message to confirm this. List archives: <http://www.htdig.org/mail/menu.html> FAQ: <http://www.htdig.org/FAQ.html>



This archive was generated by hypermail 2b28 : Mon Dec 18 2000 - 16:25:20 PST