Subject: [htdig] Re: Ignore robots.txt and META-Tags.
From: Sven Hartge (firstname.lastname@example.org)
Date: Wed Jan 26 2000 - 07:35:28 PST
Gilles Detillieux <email@example.com> schrieb:
[Ignore META-noindex on own local pages]
> No, the only way to get htdig to ignore META noindex tags is to patch
> the source. If you want to allow some search engines but disallow
> others, the place to do this is in robots.txt, which allows different
> rules for different user agents. META tags are meant to apply to any
> web client smart enough to use them.
Ok, I will remove any META-noindex tag from our pages and set up a
correct robots.txt. Anyone a good search'n'replace tool, which can
handle about 10,000 files without choking?
> No, unfortunately, that's one of the big problems with locales -
> if they're not set up correctly on your system, you're out of luck,
Well, then it seems we have to spend some money in beer. Bribing
Sysadmins with pizza or beer ... ;->
-- Letzte Worte des Mensakochs: "Merkwuerdig ruhig da draussen ..."
------------------------------------ To unsubscribe from the htdig mailing list, send a message to firstname.lastname@example.org You will receive a message to confirm this.
This archive was generated by hypermail 2b28 : Wed Jan 26 2000 - 07:37:41 PST