Re: [htdig3-dev] [ANNOUNCE] ht://Dig 3.2.0b1

Subject: Re: [htdig3-dev] [ANNOUNCE] ht://Dig 3.2.0b1
From: Geoff Hutchison (
Date: Sat Feb 05 2000 - 16:50:11 PST

At 10:29 PM +0200 2/5/00, Valdas Andrulis wrote:
>User-agent: *
>Disallow: /cat/
>User-agent: htdig
>Disallow: /cat/
>but htdig doesn't pay attention on that, it still indexes pages like

This is a bit strange. The robots.txt you mentioned is redundant--it
says "no one should index /cat/" and then "htdig should not index

Nevertheless, it shouldn't do this. What version is this? 3.2.0b1?
Does it seem to work under 3.1.4?


-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig3-dev mailing list, send a message to
You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Sat Feb 05 2000 - 16:53:12 PST