Re: [htdig] problem with frames ?

Torsten Neuer (
Tue, 24 Aug 1999 09:05:50 +0200

According to Geoff Hutchison:
>> that I get when I turn the debugging on. Is there any way in htdig to
>> overwrite this ?
>No. That would violate the robot exclusion standard.
>> Robots.txt line: User-agent: *
>> Found 'user-agent' line: *
>> Robots.txt line: Disallow: /
>This tells all robots to ignore all files on this server. So ht://Dig is
>doing exactly what it's told.
>My suggestion would be to have the server administrator change the
>robots.txt file.

ANother solution would be to tell ht://Dig to use an alternate
robots.txt file. See
for more information.


InWise - Wirtschaftlich-Wissenschaftlicher Internet Service GmbH
Waldhofstraße 14                            Tel: +49-4101-403605
D-25474 Ellerbek                            Fax: +49-4101-403606
E-Mail:            Internet:

------------------------------------ To unsubscribe from the htdig mailing list, send a message to containing the single word unsubscribe in the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Tue Aug 24 1999 - 00:09:49 PDT