Re: [htdig] SSI's & Not indexing docs

Geoff Hutchison (
Fri, 29 Jan 1999 23:43:57 -0400

>seen. Does this version of htdig support ssi's? If not Any patches? Any

If you're indexing through HTTP, ht://Dig couldn't care whether it's SSI,
ASP, CGI, or whatever. All it will look at over HTTP is the end-result file
that the user would see.

>don't have any exclusions for htdig. Is it possible the larger web
>hosting's company server has a htdig exclusion? Similarly I didn't put

That shouldn't matter. When indexing, ht://Dig (or any standard indexer)
will grab /robots.txt before starting on the specific server. So it's your
virtual host that determines the /robots.txt file.

>any exclusions on indexing .shtml files in the htdig.conf file. I also
>tried deleting htdig and reinstalling it on the server and it that was
>no help.

One of the best ways to see what ht://Dig is up to is to add debugging output.
For example:

htdig -vvv (flags of your choice)

This will generate a bunch of messages, including results of parsing
robots.txt, and HTTP headers.

-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Sun Jan 31 1999 - 10:43:20 PST