Re: [htdig] "skipped"

Geoff Hutchison (
Fri, 29 Oct 1999 12:59:49 -0400 (EDT)

On Fri, 29 Oct 1999, Kari Suomela wrote:

> This happens only on *my* own site. The others work ok. What is causing
> it to be 'skipped'? There is no "robots.txt". What is the reason for
> looking for one?

There is a standard for search engines. You specify the pages that can or
cannot be indexed using a file at the root of the server called
'robots.txt' so this m must be retrieved from all servers.

-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig mailing list, send a message to containing the single word unsubscribe in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Fri Oct 29 1999 - 10:09:25 PDT