Re: htdig: Feature request


Geoff Hutchison (Geoffrey.R.Hutchison@williams.edu)
Mon, 14 Sep 1998 20:56:36 -0400


At 5:31 PM -0400 9/14/98, Michael Graff wrote:
>There is a way to limit what pages are indexed in htdig, using meta
>tags. I propose an additional flag that says "collect URLs from this
>page, but do NOT index this page."

Sorry, the robots exlusion standard people (and myself) are ahead of you.
In version 3.1.0b1, htdig accepts the following tag (check
http://www.htdig.org/meta.html for details)

<meta name="robots" content="noindex">

This means exactly as you say. Follow links from the page but don't index
them. There's also <meta name="robots" content="nofollow"> for those pages
that you don't want the URLs followed. :-)

-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-request@sdsu.edu containing the single word "unsubscribe" in
the body of the message.



This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:27:48 PST