Torsten Neuer (tneuer@inwise.de)
Tue, 24 Aug 1999 09:05:50 +0200
According to Geoff Hutchison:
>> that I get when I turn the debugging on. Is there any way in htdig to
>> overwrite this ?
>
>No. That would violate the robot exclusion standard.
>
>> Robots.txt line: User-agent: *
>> Found 'user-agent' line: *
>> Robots.txt line: Disallow: /
>
>This tells all robots to ignore all files on this server. So ht://Dig is
>doing exactly what it's told.
>
>My suggestion would be to have the server administrator change the
>robots.txt file.
ANother solution would be to tell ht://Dig to use an alternate
robots.txt file. See http://www.htdig.org/attrs.html#robotstxt_name
for more information.
hth,
Torsten
-- InWise - Wirtschaftlich-Wissenschaftlicher Internet Service GmbH Waldhofstraße 14 Tel: +49-4101-403605 D-25474 Ellerbek Fax: +49-4101-403606 E-Mail: info@inwise.de Internet: http://www.inwise.de------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig@htdig.org containing the single word unsubscribe in the SUBJECT of the message.
This archive was generated by hypermail 2.0b3 on Tue Aug 24 1999 - 00:09:49 PDT