Re: htdig: Feature request
Tue, 15 Sep 1998 14:54:31 +0930 (CST)

On 14 Sep, Chuq Von Rospach wrote:
> At 5:56 PM -0700 9/14/98, Geoff Hutchison wrote:
>> Sorry, the robots exlusion standard people (and myself) are ahead of you.
>> <meta name="robots" content="noindex">
> Brings up an interesting point I've been sort of researching. What's
> the best way to write up a robots.txt file that allows the local htdig
> to index, but nobody else? I know using a user-agent of htdig should
> allow it to index, but seems to allow remote htdig's also. I guess I
> could disable robot.txt looking in my local copy, but even though I
> trust myself not to abuse that...

In theory, you can use a configuration file entry

robotstxt_name: yourhtdig

to make htdig advertise itself as that name. In practice (at least in
my experience) that doesn't seem to work. You could also modify the
default in common/


David Robley

WEBMASTER | Phone +61 8 8374 0970 RESEARCH CENTRE FOR INJURY STUDIES | AusEinet | Flinders University, ADELAIDE, SOUTH AUSTRALIA Visit the PHP mirror at

---------------------------------------------------------------------- To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:27:49 PST