Re: htdig: Digging both internal and external sites


Frank Guangxin Liu (frank@ctcqnx4.ctc.cummins.com)
Thu, 3 Dec 1998 14:36:55 -0500 (EST)


What I did is
1) setup an internal proxy/cache server using "squid".
2) configure "squid" so that it connects directly to Intranet hosts
   and uses firewall proxy for Internet hosts.
3) tell htdig to use "squid" for all hosts.

>
> According to Geoff Hutchison:
> > At 9:50 AM -0500 12/2/98, Gilles Detillieux wrote:
> > >The other option would be to get all the local files from the local file
> > >system rather than going through HTTP, using the local_urls parameter.
> > >Than would mean all your LAN sites' content would need to be mounted
> > >(directly or via NFS) on the host running htdig.
> >
> > If you can wait, the final option is to wait until I've written the merge
> > database code and use two config files with smaller databases. One config
> > uses certain sites and the proxy, and the other does the rest. Then you
> > merge the databases together afterwards.
>
> I still think that the http_proxy parameter should have a companion
> http_proxy_exclude parameter, or something like that, which would list
> the hosts or URLs that shouldn't be fetched through the proxy server.
>
> I suspect it's fairly common to have an intranet behind a firewall, which
> can't or shouldn't be accessed through the proxy server. If you want to
> index that along with external servers, all in the same database, then
> it would sure be handy to be able to do it all in one dig, rather than
> building separate DBs in two digs, and merging them.
>
> It's probably an easy addition, but I can't really justify the time to
> do it myself right now, as we don't need it ourselves.
>
> --
> Gilles R. Detillieux E-mail: <grdetil@scrc.umanitoba.ca>
> Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/~grdetil
> Dept. Physiology, U. of Manitoba Phone: (204)789-3766
> Winnipeg, MB R3E 3J7 (Canada) Fax: (204)789-3930
> ----------------------------------------------------------------------
> To unsubscribe from the htdig mailing list, send a message to
> htdig-request@sdsu.edu containing the single word "unsubscribe" in
> the body of the message.
>

----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-request@sdsu.edu containing the single word "unsubscribe" in
the body of the message.



This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:29:46 PST