Re: [htdig] Directory Listing Indexed? Why?

Subject: Re: [htdig] Directory Listing Indexed? Why?
From: Geoff Hutchison (
Date: Wed Jun 14 2000 - 07:40:35 PDT

At 9:46 AM -0700 6/12/00, Kevin Obbayi wrote:
>1. The directory structure is also being indexed, so that after I index a
>number of files that I have in the when I type in a
>search query like the word "index" it returns the following link "index
>of/mydir" and then when I click on this link I get the directory listing of
>that directory. How can I prevent this?

Sure. You probably want to put a META robots tag in the header of
your Apache directory index files.

<meta name="robots" content="follow,noindex">

>2. IF I have one "index.html" file in the directory, rundig will not index
>any other files that directory.

Well htdig only follows links. So if you make an index.html file and
it doesn't have links to anything then you won't see anything
indexed. If the server generates an on-the-fly index, it will have
links to the individual files.

-Geoff Hutchison
Williams Students Online

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Wed Jun 14 2000 - 05:34:41 PDT