[htdig] How to get htdig to search through java servlets


Lim Swee Tat IS NCS (limst@mbox.ncs.com.sg)
Wed, 16 Jun 1999 14:57:07 +0800


Basically, I'm supposed to implement a search engine in the company's
intranet. But due to the way the intranet is working now, I'm having a bit
of major trouble getting any spider/robot or agent to set up the basic
database at all.

Basically, when the user makes a request to the site, on a prespecified port
number say, 81, a servlet thread class is called to set up the connection
and authenticate the user. Thereafter, each page he goes to is basically
generated by the different servlets.

The basic problem here is that coz the servlet produces output directly to
the server, the search robot/spider or agent will be best placed to search
information by placing the request directly to the server, but there is no
method to authenticate the robot/spider or agent. And, since the servlets
are the ones generating the output, how can the robot/spider or agent search
that particular output rather than the .java file which it is not searching
anyway.

Help!!!! Major help!! would be appreciated.

Ciao
ST Lim
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word "unsubscribe" in
the SUBJECT of the message.



This archive was generated by hypermail 2.0b3 on Tue Jun 15 1999 - 23:10:22 PDT