Re: htdig: no robots tag


webmaster@www.nisu.flinders.edu.au
Tue, 8 Sep 1998 11:15:28 +0930 (CST)


On 7 Sep, simon wrote:
> Greetings from downunder,
>
> Is there a way to turn the robot exclusions protocol off.
>
> We have no robots tags to stop external bots on some pages but want
> htdig to go in.
>
> Thanks for all the work that has gone into this project.
>
> Cheers,
> Simon
>

You'll find a description of how to do this at

http://info.webcrawler.com/mak/projects/robots/norobots.html

Essentially, you need a section in robots.txt that gives permissions to
your htdig robot which aren't available to others. Example:

#Keep out the nasties
User-agent: *
Disallow: whatever

# Let htdig at everything
User-agent: htdig (or whatever you have called it)
Disallow:

Cheers

-- 
David Robley

WEBMASTER | Phone +61 8 8374 0970 RESEARCH CENTRE FOR INJURY STUDIES | http://www.nisu.flinders.edu.au/ AusEinet | http://auseinet.flinders.edu.au/ Flinders University, ADELAIDE, SOUTH AUSTRALIA Visit the PHP mirror at http://au.php.net:81/

---------------------------------------------------------------------- To unsubscribe from the htdig mailing list, send a message to htdig-request@sdsu.edu containing the single word "unsubscribe" in the body of the message.



This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:27:46 PST