Re: [htdig] indexing files generated at runtime


Subject: Re: [htdig] indexing files generated at runtime
From: Geoff Hutchison (ghutchis@wso.williams.edu)
Date: Tue Jan 04 2000 - 17:15:31 PST


At 3:32 PM +0100 12/16/99, Jens Moellenhoff wrote:
>Hello all,
>
>We have some problems with the indexing of html files that are generated
>at runtime.
>
> >From a webserver (www.server), links go to a data bank server
>(www.databaseserver.ora) which itself gives out the result via the
>following link as html files:
>http://www.databaseserver.ora:1234/XXX/aaa/anzeige.produktlisten?spra
>ch_id=de&produkt_art=Lebensmittel).
>The server www.server is fully indexed, but not the HTML file as it
>returns the result of the query from the database server.
>
>I added the data bank server to the htdig config file:
>limit_urls_to: http://www.server \
> http://databaseserver.ora:1234

As you've written it, the second line of limit_urls_to disallows this URL.

>Why doesn't htdig index these pages? Can't htdig index files that are
>generated at runtime at all?

Yes, it can. I'd need to see the HTTP responses (say from running
htdig -vvvv) to see what was going on, but I'm assuming you send out
a HTTP redirect code. I also assume the URL returned is good
regardless of whether someone came from www.server or not. Do you
make any assumptions about the browser? Cookies?

Seeing some more debugging information would really help. From what
you say, it's hard to know whether it's rejecting the redirect (which
would most likely be a config problem or a bug in htdig) or something
on the server side.

-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-unsubscribe@htdig.org
You will receive a message to confirm this.



This archive was generated by hypermail 2b28 : Tue Jan 04 2000 - 17:33:02 PST