[htdig3-dev] Re: robots.txt bug (was [ANNOUNCE] ht://Dig 3.2.0b1)

Subject: [htdig3-dev] Re: robots.txt bug (was [ANNOUNCE] ht://Dig 3.2.0b1)
From: Geoff Hutchison (ghutchis@wso.williams.edu)
Date: Sun Feb 06 2000 - 13:25:35 PST

At 7:40 PM +0200 2/6/00, Valdas Andrulis wrote:
>I doesnt work with 3.2.0b1, and do work with 3.1.4, with that redundant
>robots.txt configurations, or just
>User-agent: *
>Disallow: /cat/
>User-agent: htdig
>Disallow: /cat/

Hmph. This one's harder than your other bug report. The pattern is
set correctly and the code looks correct. The new version switched to
using regex internally so I bet this is a bug somewhere in the logic.

First off, have you set case_sensitive to anything in your config file?

Secondly, could you try setting another (bogus) pattern in your robots.txt?

User-agent: htdig
Disallow: /foobar/

Then let us know what pattern it sets in the debug output--I don't
really want the whole thing but I want to see if it's setting the
pattern OK.

I'll try something like this on my machine to see if I can reproduce it.


To unsubscribe from the htdig3-dev mailing list, send a message to
You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Sun Feb 06 2000 - 14:29:37 PST