Re: htdig: Feature request

Geoff Hutchison (
Mon, 14 Sep 1998 20:56:36 -0400

At 5:31 PM -0400 9/14/98, Michael Graff wrote:
>There is a way to limit what pages are indexed in htdig, using meta
>tags. I propose an additional flag that says "collect URLs from this
>page, but do NOT index this page."

Sorry, the robots exlusion standard people (and myself) are ahead of you.
In version 3.1.0b1, htdig accepts the following tag (check for details)

<meta name="robots" content="noindex">

This means exactly as you say. Follow links from the page but don't index
them. There's also <meta name="robots" content="nofollow"> for those pages
that you don't want the URLs followed. :-)

-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:27:48 PST