Re: [htdig] "skipped"


Geoff Hutchison (ghutchis@wso.williams.edu)
Fri, 29 Oct 1999 12:59:49 -0400 (EDT)


On Fri, 29 Oct 1999, Kari Suomela wrote:

> This happens only on *my* own site. The others work ok. What is causing
> it to be 'skipped'? There is no "robots.txt". What is the reason for
> looking for one?

There is a standard for search engines. You specify the pages that can or
cannot be indexed using a file at the root of the server called
'robots.txt' so this m must be retrieved from all servers.

-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word unsubscribe in
the SUBJECT of the message.



This archive was generated by hypermail 2.0b3 on Fri Oct 29 1999 - 10:09:25 PDT