htdig: not following cgi urls ?

denis filipetti (
Thu, 17 Dec 1998 16:02:46 -0500

Hi Folks,

        I am trying to evaluate htdig. Our site is almost entirely generated from
a Database via servlets. I have noticed that a url like
"http://blahblah/cpg.jrun?pageId=926" gets rejected. I have commented out
exclude_urls to no avail (actually that didn't seem to be the prob anyway).
Unfortunately this means that we can never get our site indexed by htdig.
Is there a way to get around this ? Preferably by a config setting since I
am working on NT and don't have a compiler installed there. Below is an
example of this from running htdig. The "blahblah" is actually a numeric IP
addr if that matters.

Many thanks for any help on this,

url rejected: (level 1)http://blahblah/cpg.jrun
image: http://blahblah/graphics/hackett/pgnumber_bg.gif
A tag: pos = 2, position = ="/cpg.jrun?pageId=926">
href: http://blahblah/cpg.jrun ()

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:29:53 PST