Re: htdig: Slashes in query string causing loops

Gilles Detillieux (
Tue, 22 Dec 1998 14:10:14 -0600 (CST)

According to Matthew Whalen:
> When I was running 3.1.0b2, I had exactly the same problem. Everything was
> working fine for weeks, then one night this started happening. My
> configuration hadn't changed, nor had any of the scripts which create
> dynamic content on my web server. I ended up crafting some excludes
> which solved the problem, but it was annoying and bothersome. I have no
> idea what caused it.

In the 3.1.0b2 release, htdig stripped everything after the "?", so I'm
assuming you used one of the patches posted to this list to fix that,
so you could index CGI pages. You mentioned that you didn't change the
configuration or the scripts, but did you install other patches to
htdig just before this started happening?

> I think I heard "Adam Coyne" say:
> >I just upgraded to 3.1.0b3 (and applied the memory leak patches), and I'm
> >having a problem where htdig gets into an infinite loop. My home page is
> >/index.asp. It has a bunch of links to itself with different dates in the
> >query string, such as:
> >
> >/index.asp?date=11/21/98
> >
> >After htdig finishes indexing the pages for each date, it begins indexing
> >urls looking like this:
> >
> >/index.asp?date=11/21/index.asp?date=12/1/98
> >
> >It appears to be interpreting the final slash to mean a subdirectory, even
> >though it's after the question-mark. Doesn't seem quite right.

htdig interprets the final slash to mean a subdirectory when the URL matches
a path in local_urls, so it gets the document from the local file system.
Otherwise, I believe it's supposed to leave the interpretation of the
final slash up to the HTTP server. Are you using local_urls at all, and
if so, are you sure it's set not to interfere with any paths of CGI scripts?

You may want to try an htdig -vvv to see where things go awry.

Gilles R. Detillieux              E-mail: <>
Spinal Cord Research Centre       WWW:
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930
To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:29:55 PST