Re: [htdig] excluding file trees from indexing process

Subject: Re: [htdig] excluding file trees from indexing process
From: Jens Moellenhoff (
Date: Thu Dec 02 1999 - 00:12:45 PST schrieb:

> > As I stated at the beginning of this thread, I want absolutely no search
> > result showing a directory tree.
> That's where the robots exclusion standard comes in and that's why you
> need to customize this default document.
> Of course, you can also have another tool, gathering the URLs (i.e.
> documents) to be indexed from the directory structure and include
> this URL list in the start_urls directive of your Ht://Dig conf.
> But I'm not sure if this is really required, since any auto-index
> document which has a <META NAME="robots" CONTENT="noindex,follow">
> in its header should do just that automatically.

My colleague had that idea with <meta name="robots"...> before, i just
wanted to check if that's okay with you and others on the list. He also
said that it might be necessary to put a <meta http-equiv="refresh"...>
into every index page, because when i put that <meta name="robots"
content="noindex,follow"> into the index file, it does not show the file
tree, and therefore the search engine might not know which link to
follow. With the "refresh" option, I tell the robot where to go to next.

So as far as I understand it, I have to work both with the <meta
name="robot"...> solution AND the program gathering the URL. What do you

Jens Moellenhoff

To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Thu Dec 02 1999 - 00:25:18 PST