[htdig] line endings in robots.txt?

Subject: [htdig] line endings in robots.txt?
From: nets@searchtools.com
Date: Thu Dec 02 1999 - 10:38:10 PST

Next message: Geoff Hutchison: "Re: [htdig] Indexing proteted servers"
Previous message: Paul DuBois: "Re: [htdig] Indexing proteted servers"
In reply to: Paul DuBois: "Re: [htdig] Indexing proteted servers"
Next in thread: Geoff Hutchison: "Re: [htdig] line endings in robots.txt?"
Next in thread: Geoff Hutchison: "Re: [htdig] Indexing proteted servers"
Reply: nets@searchtools.com: "[htdig] line endings in robots.txt?"
Reply: Geoff Hutchison: "Re: [htdig] line endings in robots.txt?"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Hi all,

I just saw a report that ht://Dig has trouble with non-PC line
endings in the robots.txt file -- it requires CR/LF.

Is this current? I searched the site and could not tell. If so,
could you add checks for just CR (Mac line end character) and just LF
(Unix line end character)?

Avi

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-unsubscribe@htdig.org
You will receive a message to confirm this.

Next message: Geoff Hutchison: "Re: [htdig] Indexing proteted servers"
Previous message: Paul DuBois: "Re: [htdig] Indexing proteted servers"
In reply to: Paul DuBois: "Re: [htdig] Indexing proteted servers"
Next in thread: Geoff Hutchison: "Re: [htdig] line endings in robots.txt?"
Next in thread: Geoff Hutchison: "Re: [htdig] Indexing proteted servers"
Reply: nets@searchtools.com: "[htdig] line endings in robots.txt?"
Reply: Geoff Hutchison: "Re: [htdig] line endings in robots.txt?"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

This archive was generated by hypermail 2b28 : Thu Dec 02 1999 - 10:51:34 PST