Subject: [htdig] robots.txt results in not indexing a whole site?
Date: Thu Aug 17 2000 - 16:52:49 PDT
I'm using ht://Dig 3.1.4 on a Linux platform, and noticed that only one
single site from some of my URL entries were getting indexed. I turned on
all the debugging information, and this appears throughout:
Rejected: Item in the exclude list: item # 3 length: 1
url rejected: (level 1)http://www.DOMAIN.com/index.html
My problem is likely in this "exclude list" but I don't know where that's
coming from. There's nothing in the htdig.conf file that would indicate
such a list, and I don't think I'm intentionally doing anything.
I perused htdig.org and the faq, and perhaps I missed something, or perhaps
its fixed in a different version, or more likely, is just something I don't
have a clue about :-)
To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.
List archives: <http://www.htdig.org/mail/menu.html>
This archive was generated by hypermail 2b28 : Thu Aug 17 2000 - 16:52:49 PDT