[htdig] "skipped"

Kari Suomela (ftnx@karicobs.com)
Fri, 29 Oct 1999 13:26:27 EST5EDT

Friday October 29 1999 12:59, Geoff Hutchison wrote to Kari Suomela:

>> causing it to be 'skipped'? There is no "robots.txt". What is the
>> reason for looking for one?

GH> There is a standard for search engines. You specify the pages that
GH> can
GH> or cannot be indexed using a file at the root of the server
GH> called 'robots.txt' so this m must be retrieved from all servers.

I found some "info" on it, but no clear instructions as to the contents
of it.

"Disallow: /" obviously does what it says, but how do I allow htdig to
do its stuff?


To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word unsubscribe in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Fri Oct 29 1999 - 10:40:12 PDT