Re: htdig: no robots tag
Tue, 8 Sep 1998 11:15:28 +0930 (CST)

On 7 Sep, simon wrote:
> Greetings from downunder,
> Is there a way to turn the robot exclusions protocol off.
> We have no robots tags to stop external bots on some pages but want
> htdig to go in.
> Thanks for all the work that has gone into this project.
> Cheers,
> Simon

You'll find a description of how to do this at

Essentially, you need a section in robots.txt that gives permissions to
your htdig robot which aren't available to others. Example:

#Keep out the nasties
User-agent: *
Disallow: whatever

# Let htdig at everything
User-agent: htdig (or whatever you have called it)


David Robley

WEBMASTER | Phone +61 8 8374 0970 RESEARCH CENTRE FOR INJURY STUDIES | AusEinet | Flinders University, ADELAIDE, SOUTH AUSTRALIA Visit the PHP mirror at

---------------------------------------------------------------------- To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:27:46 PST