Re: htdig: Feature request


webmaster@www.nisu.flinders.edu.au
Tue, 15 Sep 1998 14:54:31 +0930 (CST)


On 14 Sep, Chuq Von Rospach wrote:
> At 5:56 PM -0700 9/14/98, Geoff Hutchison wrote:
>
>> Sorry, the robots exlusion standard people (and myself) are ahead of you.
>
>> <meta name="robots" content="noindex">
>
>
> Brings up an interesting point I've been sort of researching. What's
> the best way to write up a robots.txt file that allows the local htdig
> to index, but nobody else? I know using a user-agent of htdig should
> allow it to index, but seems to allow remote htdig's also. I guess I
> could disable robot.txt looking in my local copy, but even though I
> trust myself not to abuse that...
>

In theory, you can use a configuration file entry

robotstxt_name: yourhtdig

to make htdig advertise itself as that name. In practice (at least in
my experience) that doesn't seem to work. You could also modify the
default in common/defaults.cc.

Cheers

-- 
David Robley

WEBMASTER | Phone +61 8 8374 0970 RESEARCH CENTRE FOR INJURY STUDIES | http://www.nisu.flinders.edu.au/ AusEinet | http://auseinet.flinders.edu.au/ Flinders University, ADELAIDE, SOUTH AUSTRALIA Visit the PHP mirror at http://au.php.net:81/

---------------------------------------------------------------------- To unsubscribe from the htdig mailing list, send a message to htdig-request@sdsu.edu containing the single word "unsubscribe" in the body of the message.



This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:27:49 PST