Re: [htdig] exclude documents, but still index links on it?


Subject: Re: [htdig] exclude documents, but still index links on it?
From: Geoff Hutchison (ghutchis@wso.williams.edu)
Date: Wed Jan 05 2000 - 17:27:34 PST


On Wed, 5 Jan 2000, John Caldwell wrote:

> I'm working on a site with some dynamic content, and i want to be able to
> point htdig at an index of pages so that it can spider them, but I don't
> want the main index page to ever show up in the search results. Is there
> way to 'exclude' a file but still follow the links on it?

Sure:
<meta name="robots" content="noindex,follow">

This works with any robot that follows the META robot standard (e.g.
ht://Dig ;-)

-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-unsubscribe@htdig.org
You will receive a message to confirm this.



This archive was generated by hypermail 2b28 : Wed Jan 05 2000 - 17:43:13 PST