[htdig] Re: Ignore robots.txt and META-Tags.

Subject: [htdig] Re: Ignore robots.txt and META-Tags.
From: Sven Hartge (hartge@ds9.argh.org)
Date: Wed Jan 26 2000 - 07:35:28 PST

Gilles Detillieux <grdetil@scrc.umanitoba.ca> schrieb:

[Ignore META-noindex on own local pages]

> No, the only way to get htdig to ignore META noindex tags is to patch
> the source. If you want to allow some search engines but disallow
> others, the place to do this is in robots.txt, which allows different
> rules for different user agents. META tags are meant to apply to any
> web client smart enough to use them.

Ok, I will remove any META-noindex tag from our pages and set up a
correct robots.txt. Anyone a good search'n'replace tool, which can
handle about 10,000 files without choking?

> No, unfortunately, that's one of the big problems with locales -
> if they're not set up correctly on your system, you're out of luck,

Well, then it seems we have to spend some money in beer. Bribing
Sysadmins with pizza or beer ... ;->



Letzte Worte des Mensakochs: "Merkwuerdig ruhig da draussen ..."

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig-unsubscribe@htdig.org You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Wed Jan 26 2000 - 07:37:41 PST