Re: [htdig] rundig does not finish


Subject: Re: [htdig] rundig does not finish
From: Doug Barton (DougB@simplenet.com)
Date: Thu Nov 11 1999 - 11:22:48 PST


Howard Ha wrote:
>
> Hi there,
>
> I am experiencing a weird problem and I'm not sure what the cause might
> be. If I am digging a relatively small site everything works fine. I can
> dig pretty large numbers of pages without problems, but when I try to add
> more sites to dig I get this problem where everything just stops.

        Have you tried digging them one at a time? I suspect that you're
probably running into a robots.txt problem.

Good luck,

Doug

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word unsubscribe in
the SUBJECT of the message.



This archive was generated by hypermail 2b25 : Thu Nov 11 1999 - 11:33:19 PST