htdig: Having problems with slow WEB servers?

Maren S. Leizaola (
Tue, 4 Aug 1998 08:08:34 +0800 (CST)

        I would like to share a little solution I have found to one of my
problems. I am indexing a site which is short of bandwidth and I got
messages constantly "could not connect to site" in my logs. Also indexing
was taking about 4 hours.

        I found the solution of using a proxy server. I used squid and
found that squid did not time out as quickly under congested network
conditions. Indexing time went down to 2 hours, not as good as one would
have thought in terms of bandwidth saving but at least it seems to get the
site down in one piece for the digger to build.

        Indexing seems to be accurate so far.


Maren S. Leizaola Main : 1-519-7664375 US Voice Mail : 1-888-3236009
United MTA HK : 852 2791-2547 Fax : 1-416-3525177
--------------------------------- Coming soon

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:27:15 PST