Christophe BAEGERT (cbaegert@biographie.net)
Mon, 18 Oct 1999 02:26:50 +0200
Hello,
I tried to htdig a certain number of sites, and I had the message : "url
rejected (level 1)".
I know the reason of that : my start_url is
http://www.domain.com/datas/index.html, and this page links to documents whose
the URL is http://www.domain.com/documents/page*.html. So of course, it doesn't
match.
(i didn't exclude any urls, there wasn't any robots.txt)
I used a max hop of 1, and of course, the -i option.
It's not possible to use directly http://www.domain.com as a start_url, because
it would index too much pages.
Is there a solution ?
Regards,
--
Christophe BAEGERT webmestre@biographie.net
>>>>>>> http://biographie.net <<<<<<<
The first biographical search engine
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word unsubscribe in
the SUBJECT of the message.
This archive was generated by hypermail 2.0b3 on Sat Oct 23 1999 - 09:48:06 PDT