Re: [htdig] Excluding URL


Subject: Re: [htdig] Excluding URL
From: Geoff Hutchison (ghutchis@wso.williams.edu)
Date: Thu Mar 23 2000 - 08:21:41 PST


On Thu, 23 Mar 2000, Paul Rhoades wrote:

> Where each page is identified by a unique ArticleNumber and
> ArticleVersion number. An ArticleVersion of 0 always relates to the
> most recent (highest) version of an article and therefore I'd like to
> index the site using the following "wildcard type" URL, such that
> only the latest version of all articles are included :
>
> http://123.123.123.123/Area/*/0/index.html

Versions before 3.2.0b1 do not allow regexp in config files. Furthermore,
valid_extensions would only work on the ending \.* portion of the URL.

You could probably manage a workaround using noindex tags for older
articles--I assume you have some sort of script that rotates them. If so,
you could have it inject a META robots tag:

<meta name="robots" content="nofollow,noindex">

-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-unsubscribe@htdig.org
You will receive a message to confirm this.



This archive was generated by hypermail 2b28 : Thu Mar 23 2000 - 07:19:37 PST