Re: [htdig] MODIFIED date

Subject: Re: [htdig] MODIFIED date
From: Torsten Neuer (
Date: Tue Sep 26 2000 - 10:16:38 PDT

Roger Weiss wrote:
> OK, I'll bite. HOW?
> For background:
> Before running htdig, I run a perl script which creates an html file that
> contains links (in the form of hrefs) to all the index.html files in my
> domain.

It is much more convenient to have the start URLs listed in the
tag of the configuration file. You can include an external file for this

> Htdig then starts at this file and follows the links to generate the db.
> Is that technique getting in my way at all?
> Or do I just need to change a parm in my apache server?

What kind of pages are you serving? Pure HTML or server-parsed stuff?

If the latter is the case, you have to emit the proper HTTP header
your scripts. Emitting a Last-Modified header from your script varies
the script engine you use. If instead your pages are pure HTML,
should be fine without changing anything - except if you have
handling compiled in and activated, in which case the server will issue
cookies to tracks users which might cause the Last-Modified header not
be emitted by Apache.

What does the average HEAD request return? (Try telnet'ing the server on
port 80 and issue some HEAD request to see what HTTP headers are sent by
the server).



InWise - Wirtschaftlich-Wissenschaftlicher Internet Service GmbH
Waldhofstraße 14                            Tel: +49-4101-403605
D-25474 Ellerbek                            Fax: +49-4101-403606
E-Mail:            Internet:

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this. List archives: <> FAQ: <>

This archive was generated by hypermail 2b28 : Tue Sep 26 2000 - 10:20:18 PDT