Re: [htdig] pause

Subject: Re: [htdig] pause
From: Franck Horlaville (
Date: Thu May 25 2000 - 05:43:38 PDT

>>I'd like to pause the htdig crawl and restart it where it left off, is
>>there any way to do this ?
>Sure. Pass the -l flag when you call htdig and it will write out the
>"todo" list of URLs before it quits. Then when you restart with this
>flag, it will read this list in and resume.

thanks - I guess I had missed it in the docs ... Which btw don't mention
what signal is to be sent to interrupt : a simple kill or a kill -s INT ?

Next question : how do I know when the crawl has finished to do merge etc

My idea :

repeat forever
start crawl at 1.15 am (using a work db)
pause crawl at 6.15 am
until done
merge databases
end repeat

thanks a bunch

Franck Horlaville,

Technical Director - Athena Online s.a.
Web site creation and hosting


------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Thu May 25 2000 - 03:30:18 PDT