Re: [htdig] won't make url encoded queries?


Subject: Re: [htdig] won't make url encoded queries?
From: Torsten Neuer (tneuer@inwise.de)
Date: Thu Nov 11 1999 - 23:42:09 PST


Chris Gough wrote:
>
> Geoff Hutchison wrote:
>
> > At 2:47 PM +1100 11/12/99, Chris Gough wrote:
> > >It seems my new htdig is not following links with "url encoded"
> > >addresses, such as
> > >http://161.50.26.253/WebSearch/program.php3?id=DP-AJ66NA
> >
> > There's a patch on the website:
> >
> > <http://www.htdig.org/files/contrib/other/htdig-3.1.3-urlparmbug.patch>
>
> To me it reads as though that patch is for a bug that turns
> "..?foo=123&bar=456" into "..?foo=456". My URL has no "&" in it. Have i
> misunderstood the scope of the bug? Could their be some other reason why
> htdig is not following the links?

You're right - the bug is about multiple parameters. But there
might also be some side effects which also affect single parameters.
To be sure, it is not the case, I'd suggest you apply the patch
nevertheless.

Second, you should check "robots.txt", the "robots" META tag and
the settings of "bad_querystr" and "exclude_urls" in your Ht://Dig
configuration file.

Third, Ht://Dig won't follow the links if they are written using
some client side scripting language (like Java or Javascript) or
if they are triggered by <FORM>s (I have a couple of pages where
I use this to exclude pages from being indexed).

hth,
  Torsten

-- 
InWise - Wirtschaftlich-Wissenschaftlicher Internet Service GmbH
Waldhofstraße 14                            Tel: +49-4101-403605
D-25474 Ellerbek                            Fax: +49-4101-403606
E-Mail: info@inwise.de            Internet: http://www.inwise.de

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig@htdig.org containing the single word unsubscribe in the SUBJECT of the message.



This archive was generated by hypermail 2b25 : Thu Nov 11 1999 - 23:52:58 PST