[htdig] robotstxt_name

Subject: [htdig] robotstxt_name
From: Mike Paradis (mike@newforce.ca)
Date: Sun Jan 07 2001 - 07:25:10 PST

In the directives list, this is explained...

type: string
used by: htdig
default: htdig
description: Sets the name that htdig will look for when parsing robots.txt files. This
can be used to make htdig appear as a different spider than ht://Dig. Useful to
distinguish between a private and a global index.
example: robotstxt_name: myhtdig

However, I am still not clear on this because of the mention os what htdig can be made to
look like. Is this directive so that htdig will know to look for robots.txt or a custom
name file that does the same. Or, is this to make our digger look like something else?


To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.
List archives: <http://www.htdig.org/mail/menu.html>
FAQ: <http://www.htdig.org/FAQ.html>

This archive was generated by hypermail 2b28 : Sun Jan 07 2001 - 07:37:25 PST