Re: [htdig3-dev] Re: robots.txt bug (was [ANNOUNCE] ht://Dig 3.2.0b1)


Subject: Re: [htdig3-dev] Re: robots.txt bug (was [ANNOUNCE] ht://Dig 3.2.0b1)
From: Valdas Andrulis (valdand@soften.ktu.lt)
Date: Mon Feb 07 2000 - 01:15:36 PST


On Sun, 6 Feb 2000, Geoff Hutchison wrote:

GH> At 7:40 PM +0200 2/6/00, Valdas Andrulis wrote:
GH> >I doesnt work with 3.2.0b1, and do work with 3.1.4, with that redundant
GH> >robots.txt configurations, or just
GH> >User-agent: *
GH> >Disallow: /cat/
GH> >
GH> >or
GH> >
GH> >User-agent: htdig
GH> >Disallow: /cat/
GH>
GH> Hmph. This one's harder than your other bug report. The pattern is
GH> set correctly and the code looks correct. The new version switched to
GH> using regex internally so I bet this is a bug somewhere in the logic.
GH>
GH> First off, have you set case_sensitive to anything in your config file?

No.

GH>
GH> Secondly, could you try setting another (bogus) pattern in your robots.txt?
GH>
GH> Add:
GH> User-agent: htdig
GH> Disallow: /foobar/
GH>
GH> Then let us know what pattern it sets in the debug output--I don't
GH> really want the whole thing but I want to see if it's setting the
GH> pattern OK.

Trying to retrieve robots.txt file
Parsing robots.txt file using myname = htdig
Found 'user-agent' line: htdig
Found 'disallow' line: /cat/
Found 'user-agent' line: htdig
Found 'disallow' line: /foobar/
Pattern: /foobar/

VAldas

GH>
GH> I'll try something like this on my machine to see if I can reproduce it.
GH>
GH> -Geoff
GH>
GH>
GH> ------------------------------------
GH> To unsubscribe from the htdig3-dev mailing list, send a message to
GH> htdig3-dev-unsubscribe@htdig.org
GH> You will receive a message to confirm this.
GH>
GH>
GH>

------------------------------------
To unsubscribe from the htdig3-dev mailing list, send a message to
htdig3-dev-unsubscribe@htdig.org
You will receive a message to confirm this.



This archive was generated by hypermail 2b28 : Mon Feb 07 2000 - 01:18:46 PST