Re: [htdig] Unable to open word list file ...Hmm -- A new twist?


Subject: Re: [htdig] Unable to open word list file ...Hmm -- A new twist?
From: hsinclai (hsinclai@linux1091.dn.net)
Date: Thu Jun 01 2000 - 14:36:41 PDT


Hi,

On Thu, 1 Jun 2000, Geoff Hutchison wrote:

> At 9:24 PM -0400 5/31/00, hsinclai wrote:
> >bytes, as should be (stderr is dumped out via 2&>1 >> myfile.somewhere).

> Have you been timing the runs? How long does it usually take?

Yes, the longest takes less than 3 minutes -- these are small sites, but they will grow exponentially in a few months.

> >Suddenly one night they don't run and htmerge spits out the message in the
> >log file. The databases are gone, and only a .new file is in each
> >directory.
>
> What happens if you run this by hand now? Have you tried adding in
> more debugging output?

I wasn't watching when this happened (4AM)... what was left in the
database directory were some 0-byte files and a .new file. I tried running
my script again and htmerge failed with the same error. So I just blew the
files away and redid my initial htdig.

> >P700) and the disk I/O so fast, and the net connection is so fast that the
> >second job might be starting before the first one finished. Or something
> >like that.
>
> Let me get this straight... You're running these as daily cron jobs,
> right? If the cron job is really stepping on itself, you would expect
> that it takes longer than 24 hours to run the job. Since you say this
> is "sudden," was it taking anywhere *close* to 24 hours before?

Sorry -- I didn't express this very well. There are about 10 scripts in
the /etc/cron.daily directory. I was under the impression that cron would
wait for the parent PID of the first script to exit before launching the
next one. Perhaps this is not the case, I'll have to research it. I am
guessing that one of my rundig scripts had not fully finished before the
next one was launched by cron.

> My suggestion is to disable the cron job for now, run it by hand and
> see what happens, how long it takes, etc. (This is pretty much
> standard procedure for me when *any* cron job starts acting flaky.)

Thanks and yes that's what I did. However everything ran just hunky dory
manually so I dropped the scripts back in cron.daily and they ran last
night without a hitch!

Nevertheless I'm getting these out of the cron daily. I'm working on a
script (it's rather long) that will run the jobs from root's crontab with
some delay in between digs merges and fuzzies and a truckload of error
checking. If I get it working correctly I'll be glad to post it should
anyone be interested.. maybe someone out there can make my crappy bash
scripting a little more elegant!

Sorry for the length of this post, I'll just have to see if it happens
again and try to gather evidence more useful than what I got this time :)

Regards,

-Harold

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-unsubscribe@htdig.org
You will receive a message to confirm this.



This archive was generated by hypermail 2b28 : Thu Jun 01 2000 - 12:25:57 PDT