[htdig] how to ignore robots.txt

p0222 (p0222@odb.rhein-main.de)
Sun, 28 Mar 1999 14:28:11 +0200

• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
• Next message: Markus Rietzler: "[htdig] problem after updating database"
• Previous message: Bob Nugent: "[htdig] Hosts providing ht://Dig service"
• Next in thread: Geoff Hutchison: "Re: [htdig] how to ignore robots.txt"

Hello People !!!

How can I tell htdig to *ignore* the robots.txt-files, on the whole web or
on specified servers ?

That's my problem:
title: ReferateFundus
href: http://www.fundus.org/index1.htm ()
resolving 'http://www.fundus.org/index1.htm'

   pushing http://www.fundus.org/index1.htm
+href: http://www.fundus.org/indexrechts.htm ()
resolving 'http://www.fundus.org/indexrechts.htm'

   pushing http://www.fundus.org/indexrechts.htm
+A tag: pos = 2, position =
="http://www.fundus.org/cgi/ref_anz.cgi?Biographien">
href: http://www.fundus.org/cgi/ref_anz.cgi?Biographien (Biographien [290])

   Rejected: Item in the exclude list: item # 2 length: 4

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^EXLCUDE LIST ?!?
How can i turn this exlcude list *OFF* ?!?

Thank you,
Gunther Stammwitz

url rejected: (level 1)http://www.fundus.org/cgi/ref_anz.cgi?Biographien
A tag: pos = 2, position =
="http://www.fundus.org/cgi/ref_anz.cgi?Biologie">
href: http://www.fundus.org/cgi/ref_anz.cgi?Biologie (Biologie [238])

   Rejected: Item in the exclude list: item # 2 length: 4

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word "unsubscribe" in
the SUBJECT of the message.

• Next message: Markus Rietzler: "[htdig] problem after updating database"
• Previous message: Bob Nugent: "[htdig] Hosts providing ht://Dig service"
• Next in thread: Geoff Hutchison: "Re: [htdig] how to ignore robots.txt"

This archive was generated by hypermail 2.0b3 on Sun Mar 28 1999 - 06:25:29 PST