Re: [htdig] rundig does not finish

Subject: Re: [htdig] rundig does not finish
From: Doug Barton (
Date: Thu Nov 11 1999 - 11:22:48 PST

Howard Ha wrote:
> Hi there,
> I am experiencing a weird problem and I'm not sure what the cause might
> be. If I am digging a relatively small site everything works fine. I can
> dig pretty large numbers of pages without problems, but when I try to add
> more sites to dig I get this problem where everything just stops.

        Have you tried digging them one at a time? I suspect that you're
probably running into a robots.txt problem.

Good luck,


To unsubscribe from the htdig mailing list, send a message to containing the single word unsubscribe in
the SUBJECT of the message.

This archive was generated by hypermail 2b25 : Thu Nov 11 1999 - 11:33:19 PST