Re: [htdig] Debugging "why didn't this get indexed?" questions

Doug (
Mon, 25 Oct 1999 11:32:45 -0700

Daniel MacKay wrote:
> Howdy.
> I have just released an index of our site -- 115 servers, 100,000 pages --
> and am bracing for the "Why didn't my site get indexed properly?"
> questions.
> How do you debug these questions?

        It depends on the questioner's definition of "properly." In general,
setting up htdig was an education for us both in the sense of search
engines in general, and also in the sense of the little nits and nats on
our site that needed cleanup. I would be armed with pointers to the htdig
FAQ for things that you think will come up often (keywords, descriptions,
etc.) and then let the chips fall where they may.

> I ran htdig in -v mode but I don't think
> there's enough info there to answer that question; should I run it in -vv
> ?? More v's??

        More -vvvv's equal more detail. I'd start with 5, that should give you all
the info you need. Also, robots.txt files were a big hindrance on our site,
but the -vvvvv gives you the info in detail as to what it's doing to parse

Good luck,


To unsubscribe from the htdig mailing list, send a message to containing the single word unsubscribe in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Mon Oct 25 1999 - 11:42:12 PDT