[htdig] Re: Ignore robots.txt and META-Tags.

Subject: [htdig] Re: Ignore robots.txt and META-Tags.
From: Geoff Hutchison (ghutchis@wso.williams.edu)
Date: Wed Jan 26 2000 - 08:25:33 PST

At 4:35 PM +0100 1/26/00, Sven Hartge wrote:
>Ok, I will remove any META-noindex tag from our pages and set up a
>correct robots.txt. Anyone a good search'n'replace tool, which can
>handle about 10,000 files without choking?

I usually use sed, but you can do this with perl or awk as well.

>Well, then it seems we have to spend some money in beer. Bribing
>Sysadmins with pizza or beer ... ;->


Good luck!

-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Wed Jan 26 2000 - 08:32:10 PST