Re: htdig: Digging both internal and external sites

Denis Bazinet (
Thu, 03 Dec 1998 21:46:06 -0500

Since I thought this was such an AWESOME solution, I thought it should make in the thread with a brief
explanation for the not so technically inclined. (I'm you are all very smart!)

The problem: I was trying to set up one of my SPARC systems to be a search engine for distinct sites:
some on the Internet and some on our intranet (protected by a firewall/proxy). Htdig seems to be able
to ONLY dig internal sites if the config file has no "http_proxy" setting, and digs ONLY external sites,
if there is a proxy setting. So, the problem is ONE CONFIG FILE TO DO BOTH!!!

The answer: (First of all, I'd like to metion that I thought of doing this too, but hadn't had time to
try it!) Frank and Gilles proposed to "push" the firewall to inside the computer. That is, to put a
proxy server close to the htdig machine, so that it thinks all sites are external to this internal proxy
server. Frank took this one step further and mentionned that this new proxy server could be installed
on the same machine as htdig. Therefore, before any communication out to the web server(s) would
happen, it would go through its proxy server process.
     One note to add, in the new internal proxy server, you must specify which addresses are local and
define the old proxy server to chain. And don't forget to tell htdig to use this new proxy server.

Thanx again Frank and Gilles,


Denis Bazinet, System Administrator
Online Strategy & Development
Bell Emergis (613) 781-3974

---------------------------------------------------------------------- To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:29:47 PST