RE: [htdig] htdig-3.1.1; stopping it from indexing scripts


Budd, S. (s.budd@ic.ac.uk)
Thu, 25 Feb 1999 09:56:47 -0000


Unfortunately this "?" will also exclude "active server pages"
for as much as we feel about it a lot of people are
now using the NT server.

> -----Original Message-----
> From: Geoff Hutchison [SMTP:ghutchis@wso.williams.edu]
> Sent: 24 February 1999 18:38
> To: htdig@htdig.org
> Cc: htdig@htdig.org
> Subject: Re: [htdig] htdig-3.1.1; stopping it from indexing scripts
>
>
> >Finally got it to compile (don't know why, it just did). Now I noticed
> that
> >it seems to index CGI script data (with ?:s and &:s in the path) by
> default.
> >How do I stop this? I tried to add the path to the script to the
> >exclude-urls configuration option, and restart htdig, but it continued to
> >index it, which isn't too good, since the script has about 216^5
> >combinations... :-/
>
> You don't say what you set exclude_urls to. Here's mine:
>
> exclude_urls: cgi-bin .cgi ? .pl
>
> This excludes any URLs in our cgi-bin directory, as well as Apache's
> directory sorting features and every URL with a ? character, which
> indicates the start of a query.
>
> Cheers,
>
> -Geoff Hutchison
> Williams Students Online
> http://wso.williams.edu/
>
>
> ------------------------------------
> To unsubscribe from the htdig mailing list, send a message to
> htdig@htdig.org containing the single word "unsubscribe" in
> the SUBJECT of the message.
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word "unsubscribe" in
the SUBJECT of the message.



This archive was generated by hypermail 2.0b3 on Fri Feb 26 1999 - 14:34:13 PST