Re: [htdig] META NAME="robots" ignored.


Martin Pohl (martin.pohl@medizin.uni-ulm.de)
Mon, 22 Feb 1999 15:34:07 +0100


Yann Doussot wrote:

> I've just installed htdig 3.1.1 on a Linux server. Works fine
> except the META TAGS. I've added <META NAME="robots" VALUE="noindex"> to
> several pages, rebuilt database but these are still returned by a search.
> Ideas ? May be I missunderstand the meaning of the tag (I want links in
> the page to be followed but I don't want to index it).

As far as I know, your Meta-tag is only supported by very few robots.
Obviously not by ht://Dig. Another method would be to create a file
'robot.txt' in the root directory of the WWW server, which contains

User-agent:<robot-name> | "*"
[Allow: /path/[file.html] ]
Disallow: /path/[file.html]

All lines can appear more than once. If you use the asterisk in the
first line, all robots should regard the exclusions.

Let me know, if it had worked.

Martin

-- 
_____________________________________________________
Dr. Martin Pohl                  	
Institute for Orthopaedic Research and Biomechanics
Helmholtzstr. 14
89081 ULM, Germany, EU
phone: +49-(0)731-50-23494 / FAX: +49-(0)731-50-23498
WWW:    http://pollux.medizin.uni-ulm.de 
e-mail: mailto://martin.pohl@medizin.uni-ulm.de
_____________________________________________________
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word "unsubscribe" in
the SUBJECT of the message.



This archive was generated by hypermail 2.0b3 on Mon Feb 22 1999 - 07:08:23 PST