Re: htdig: Pages get indexed, but no results: BUG?

Netsolution Internet Consulting (
Thu, 17 Dec 1998 18:18:38 +0100

Gilles Detillieux wrote:

> Who says it's deleting anything? Does an htdig -vvv seem to suggest that?
> What I'm suggesting is that htdig sees the href to PRODUCTS.HTM before any
> href to products.htm, and so it queues up the upper-case URL, but marks
> the lower-case URL as visited (because all visits are recorded in lower
> case). So, it tries to get PRODUCTS.HTM, and fails, so it never sees the
> real file. Whenever it sees any of the good hrefs to products.htm, it
> thinks the file was already visited, so it doesn't queue it up again.
> Do you have any hard evidence that htdig is indeed fetching products.htm
> from the server, and deleting its hrefs?

I actually did run htdig -sivvv and I did see that the pages which are linked from
products.htm were defenitely indexed - there are at least 100 pages linked from
products.htm so its certain that they have been indexed.

But when I search with keywords from these pages, htsearch does not find any results - I
made several tests, so Im sure.

htsearc finds pages from that site which do not start from products.htm - no problem

That is why Im assuming that the pages get indexed and then deleted again.

That is also why I think that it does not help when I take a start URL starting directly
from products.htm.

Also I dig several sites, so would it make sense to use limit urls to?


To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:29:54 PST