Re: [htdig] problem with frames ?


Torsten Neuer (tneuer@inwise.de)
Tue, 24 Aug 1999 09:05:50 +0200


According to Geoff Hutchison:
>> that I get when I turn the debugging on. Is there any way in htdig to
>> overwrite this ?
>
>No. That would violate the robot exclusion standard.
>
>> Robots.txt line: User-agent: *
>> Found 'user-agent' line: *
>> Robots.txt line: Disallow: /
>
>This tells all robots to ignore all files on this server. So ht://Dig is
>doing exactly what it's told.
>
>My suggestion would be to have the server administrator change the
>robots.txt file.

ANother solution would be to tell ht://Dig to use an alternate
robots.txt file. See http://www.htdig.org/attrs.html#robotstxt_name
for more information.

hth,
  Torsten

--
InWise - Wirtschaftlich-Wissenschaftlicher Internet Service GmbH
Waldhofstraße 14                            Tel: +49-4101-403605
D-25474 Ellerbek                            Fax: +49-4101-403606
E-Mail: info@inwise.de            Internet: http://www.inwise.de

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig@htdig.org containing the single word unsubscribe in the SUBJECT of the message.



This archive was generated by hypermail 2.0b3 on Tue Aug 24 1999 - 00:09:49 PDT