Re: [htdig] Words and files not being found or indexed

Subject: Re: [htdig] Words and files not being found or indexed
From: Dennis Director (
Date: Thu Dec 14 2000 - 07:50:19 PST

On Wed, 13 Dec 2000, crosstar wrote:

> I am not too technical, so I hope this sounds clear.
> I have htdig installed. But, although it works fine with no
> errors, many files and words are being left out of the search and
> indexing.

I'm not any kind of htdig expert, but ..
on my sites, I discovered, (the slow way) that htdig recurses down from
the start page that you give it in the config file. If there is not a
path of links on pages, starting at the start page, that gets you to the
subdirectory you mention, then those pages will not be fetched and
indexed. This happened to me when I set <noindex> tags around my
navigation bar on all pages. I needed to leave the nav bar indexable, at
least on the front page, so that there were references to all sections.
Hope this helps!

To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.
List archives: <>
FAQ: <>

This archive was generated by hypermail 2b28 : Thu Dec 14 2000 - 07:49:02 PST