htdig: Excluding mailing list archive indexes


Jeff Breidenbach (jeff@multivac.jab.org)
Thu, 12 Nov 1998 11:08:08 -0500


Erno,

www.mail-archive.com does this by putting HTML meta tags on index
pages that instruct robots not to archive the page. This assumes you
have control over the HTML pages.

Jeff

> search http://www.foo.org/mailing-list/*
> not return as search results as urls that match */date.html, or
> */thread.html, or end in '/'.
>
> but it should still follow the links in those files and index
> the linked-to files (mailing list messages).

----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-request@sdsu.edu containing the single word "unsubscribe" in
the body of the message.



This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:28:47 PST