Re: [htdig] Words and files not being found or indexed

Subject: Re: [htdig] Words and files not being found or indexed
From: crosstar (
Date: Thu Dec 14 2000 - 01:48:29 PST

Thanks for taking the time to reply. :)

I might sound a bit dense (sorry), but bear with me, as I hope
to get this thing operable.

What do you mean by:

"Path of links on pages?"

"Starting at the start page?"

For example, in htdig.conf, at present, there is:

# This specifies the URL where the robot (htdig) will start. You can specify# multiple URLs here. Just separate them by some whitespace.# The example here will cause the ht://Dig homepage and related pages to be# indexed.# You could also index all the URLs in a file like so:# start_url: `${common_dir}/start.url`#start_url:

Are you saying that this is incorrect? Do I need to add
something here?

If so, what?

Or, are you stating that something must be added to each page
in the site? There are thousadns of pages and this would be
a rather difficult proposition (hope that wouldn't be necessary).

If you are saying that something has to be added on each page,
what -- exactly -- needs to be added, please.

What do you mean by:

"leave the nav bar indexable, at least on the front page"

The reason I am in the dark is that about 40% of the site appears
to be indexing all right. But the other 60% is lost.

I am trying to understand your analysis, but could you,
perhaps, simply tell me what exactly to do (such as,
"type this," "cut and paste" that, or something practical
(rather than just theoretical).

As I said, I am not very technical.

Much appreciated.

At 09:50 AM 12/14/00 -0600, you wrote:
>On Wed, 13 Dec 2000, crosstar wrote:
>> I am not too technical, so I hope this sounds clear.
>> I have htdig installed. But, although it works fine with no
>> errors, many files and words are being left out of the search and
>> indexing.
>I'm not any kind of htdig expert, but ..
>on my sites, I discovered, (the slow way) that htdig recurses down from
>the start page that you give it in the config file. If there is not a
>path of links on pages, starting at the start page, that gets you to the
>subdirectory you mention, then those pages will not be fetched and
>indexed. This happened to me when I set <noindex> tags around my
>navigation bar on all pages. I needed to leave the nav bar indexable, at
>least on the front page, so that there were references to all sections.
>Hope this helps!

The Nationalist Movement
PO Box 2000
Learned MS 39154
(601) 885-2288
Home Page:
ICQ: 5429992
Newsgroup: alt.national
Views not necessarily those of The Nationalist Movement
2000 by The Nationalist Movement


To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.
List archives: <>
FAQ: <>

This archive was generated by hypermail 2b28 : Thu Dec 14 2000 - 08:02:00 PST