Re: [htdig] htdig won't ignore the files I want ignored

Subject: Re: [htdig] htdig won't ignore the files I want ignored
From: Geoff Hutchison (
Date: Sat Dec 18 1999 - 14:43:08 PST

On Fri, 17 Dec 1999, Paul Johnson wrote:

> Q: How can I prevent htdig's index from including files like index.html
> that are automatically found by the web browser when doing a crawl over
> a directory structure.
> A: In each index.html file you want to exclude, add the following
> between the <HEAD> and </HEAD> tags:
> <META NAME="robots" CONTENT="noindex, follow">
> The insertion of this line can be made automatic in MhonArc by inserting
> that line in the resource file in the sections IDXPGBEGIN and

Fair enough, we'll hopefully include it in the FAQ soon. (I'm home for
the holidays and can't edit things very well :-)

-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Sat Dec 18 1999 - 14:57:02 PST