Re: [htdig] META NAME="robots" ignored.

Martin Pohl (
Mon, 22 Feb 1999 15:34:07 +0100

Yann Doussot wrote:

> I've just installed htdig 3.1.1 on a Linux server. Works fine
> except the META TAGS. I've added <META NAME="robots" VALUE="noindex"> to
> several pages, rebuilt database but these are still returned by a search.
> Ideas ? May be I missunderstand the meaning of the tag (I want links in
> the page to be followed but I don't want to index it).

As far as I know, your Meta-tag is only supported by very few robots.
Obviously not by ht://Dig. Another method would be to create a file
'robot.txt' in the root directory of the WWW server, which contains

User-agent:<robot-name> | "*"
[Allow: /path/[file.html] ]
Disallow: /path/[file.html]

All lines can appear more than once. If you use the asterisk in the
first line, all robots should regard the exclusions.

Let me know, if it had worked.


Dr. Martin Pohl                  	
Institute for Orthopaedic Research and Biomechanics
Helmholtzstr. 14
89081 ULM, Germany, EU
phone: +49-(0)731-50-23494 / FAX: +49-(0)731-50-23498
e-mail: mailto://
To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Mon Feb 22 1999 - 07:08:23 PST