Re: [htdig] problem with frames ?

Geoff Hutchison (
Mon, 23 Aug 1999 17:47:23 -0500

> that I get when I turn the debugging on. Is there any way in htdig to
> overwrite this ?

No. That would violate the robot exclusion standard.

> Robots.txt line: User-agent: *
> Found 'user-agent' line: *
> Robots.txt line: Disallow: /

This tells all robots to ignore all files on this server. So ht://Dig is
doing exactly what it's told.

My suggestion would be to have the server administrator change the
robots.txt file.

-Geoff Hutchison
Williams Students Online

------------------------------------ To unsubscribe from the htdig mailing list, send a message to containing the single word unsubscribe in the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Mon Aug 23 1999 - 15:50:15 PDT