[htdig] robots.txt results in not indexing a whole site?


Subject: [htdig] robots.txt results in not indexing a whole site?
From: boerio@arocknid.com
Date: Thu Aug 17 2000 - 16:52:49 PDT


Hi,

I'm using ht://Dig 3.1.4 on a Linux platform, and noticed that only one
single site from some of my URL entries were getting indexed. I turned on
all the debugging information, and this appears throughout:

  Rejected: Item in the exclude list: item # 3 length: 1

  url rejected: (level 1)http://www.DOMAIN.com/index.html

My problem is likely in this "exclude list" but I don't know where that's
coming from. There's nothing in the htdig.conf file that would indicate
such a list, and I don't think I'm intentionally doing anything.

I perused htdig.org and the faq, and perhaps I missed something, or perhaps
its fixed in a different version, or more likely, is just something I don't
have a clue about :-)

Suggestions?

     - Jeff

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-unsubscribe@htdig.org
You will receive a message to confirm this.
List archives: <http://www.htdig.org/mail/menu.html>
FAQ: <http://www.htdig.org/FAQ.html>



This archive was generated by hypermail 2b28 : Thu Aug 17 2000 - 16:52:49 PDT