Subject: RE: [htdig] META tags and mod_autoindex feature of Apache, htdig fails to fol low links
From: Vergonet, Henk (henk.vergonet@ben.nl)
Date: Mon May 01 2000 - 03:07:52 PDT
Forgive me,
Htdig does seem to follow META tags.
Regards,
Henk
-----Original Message-----
From: Vergonet, Henk [mailto:henk.vergonet@ben.nl]
Sent: Monday, May 01, 2000 10:37 AM
To: 'htdig@htdig.org'
Subject: [htdig] META tags and mod_autoindex feature of Apache, htdig
fails to fol low links
Hi,
I've patched the module mod_autoindex (the automatic generation of
dirlistings where no index file exists) to generate a robots META tag.
Apache now generates as listings code:
...
<HTML>
<HEAD>
<META name="robots" content="noindex,follow">
<TITLE>Index of /data/IT-development</TITLE>
</HEAD>
<BODY>
...
<IMG SRC="/icons/back.gif" ALT="[DIR]"> <A HREF="/data/">Parent
Directory</A>
<IMG SRC="/icons/text.gif" ALT="[TXT]"> <A HREF="test.txt">test.txt</A>
...
</BODY></HTML>
I had hoped this would stop the indexing of the dir-listings, and
indeed it does, but it also stops indexing of the file test.txt
and this I don't want.
Is this a bug in htdig-3.1.5?
Regards,
Henk Vergonet
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-unsubscribe@htdig.org
You will receive a message to confirm this.
This archive was generated by hypermail 2b28 : Mon May 01 2000 - 00:55:05 PDT