Re: [htdig] htdig-3.1.1; stopping it from indexing scripts

Geoff Hutchison (
Wed, 24 Feb 1999 13:38:00 -0500

>Finally got it to compile (don't know why, it just did). Now I noticed that
>it seems to index CGI script data (with ?:s and &:s in the path) by default.
>How do I stop this? I tried to add the path to the script to the
>exclude-urls configuration option, and restart htdig, but it continued to
>index it, which isn't too good, since the script has about 216^5
>combinations... :-/

You don't say what you set exclude_urls to. Here's mine:

exclude_urls: cgi-bin .cgi ? .pl

This excludes any URLs in our cgi-bin directory, as well as Apache's
directory sorting features and every URL with a ? character, which
indicates the start of a query.


-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Fri Feb 26 1999 - 14:34:12 PST