Re: htdig: geocities robots.txt


Geoff Hutchison (ghutchis@wso.williams.edu)
Mon, 14 Dec 1998 21:42:29 -0500


At 4:54 PM -0500 12/14/98, Ryan Scott wrote:
>I still cannot index geocities pages. Here's what they put in their
>robots.txt file:
>
># htdig knows where to go.
>User-agent: htdig/3.1.0b1

>I'm not too familiar with how it is all supposed to be but it appears this
>doesn't cut it. I'm trying to index various neighborhoods on request of the
>folks running those neighborhoods, in case you were a wonderin.

Either get Geocities to set the User-agent: htdig or set the robotstxt_name
option to "htdig/3.1.0b1" in your conf file. The problem is that the
robots.txt name must match *exactly* and it doesn't.

-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-request@sdsu.edu containing the single word "unsubscribe" in
the body of the message.



This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:29:53 PST