Re: htdig: Digging both internal and external sites

Gilles Detillieux (
Thu, 3 Dec 1998 12:08:51 -0600 (CST)

According to Geoff Hutchison:
> At 9:50 AM -0500 12/2/98, Gilles Detillieux wrote:
> >The other option would be to get all the local files from the local file
> >system rather than going through HTTP, using the local_urls parameter.
> >Than would mean all your LAN sites' content would need to be mounted
> >(directly or via NFS) on the host running htdig.
> If you can wait, the final option is to wait until I've written the merge
> database code and use two config files with smaller databases. One config
> uses certain sites and the proxy, and the other does the rest. Then you
> merge the databases together afterwards.

I still think that the http_proxy parameter should have a companion
http_proxy_exclude parameter, or something like that, which would list
the hosts or URLs that shouldn't be fetched through the proxy server.

I suspect it's fairly common to have an intranet behind a firewall, which
can't or shouldn't be accessed through the proxy server. If you want to
index that along with external servers, all in the same database, then
it would sure be handy to be able to do it all in one dig, rather than
building separate DBs in two digs, and merging them.

It's probably an easy addition, but I can't really justify the time to
do it myself right now, as we don't need it ourselves.

Gilles R. Detillieux              E-mail: <>
Spinal Cord Research Centre       WWW:
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930
To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:29:46 PST