htdig: Excluding mailing list archive indexes

Jeff Breidenbach (
Thu, 12 Nov 1998 11:08:08 -0500

Erno, does this by putting HTML meta tags on index
pages that instruct robots not to archive the page. This assumes you
have control over the HTML pages.


> search*
> not return as search results as urls that match */date.html, or
> */thread.html, or end in '/'.
> but it should still follow the links in those files and index
> the linked-to files (mailing list messages).

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:28:47 PST