Re: [htdig] The 4:02 crash

Subject: Re: [htdig] The 4:02 crash
From: Geoff Hutchison (
Date: Tue Sep 12 2000 - 08:46:42 PDT

On Tue, 12 Sep 2000, Vincent QUERU wrote:

> But looking through Apache's access log I noticed the following 2 lines :
> my_server - - [12/Sep/2000:04:02:00 +0200] "GET /robots.txt HTTP/1.0" 404 278
> my_server - - [12/Sep/2000:04:02:00 +0200] "GET
> /r2_admin/robot_init_page/?ht_dig_robot=1 HTTP/1.0" 401
> 471
> I did not do a "robots.txt" file as my server is the only one to index
> the site.

That's fine, but htdig will still fetch it. It's required to do so by 'net
standards. It does this first off when it finds a server. I assume the
next line is your start_url?

> It looks as if there is some kind of automatic indexing (of course 4:02 is
> nowhere to be found in my crontab)

Well it has to be launched somehow, either from 'cron' or 'at' since htdig
cannot launch by itself. What time is in your crontab?

> that after it my db.wordlist file is
> empty...

And if you run the script yourself from the command-line it works fine?
What cron program/version do you use?

-Geoff Hutchison
Williams Students Online

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this. List archives: <> FAQ: <>

This archive was generated by hypermail 2b28 : Tue Sep 12 2000 - 08:49:08 PDT