[htdig] URL doesn't match

Christophe BAEGERT (cbaegert@biographie.net)
Mon, 18 Oct 1999 02:26:50 +0200


I tried to htdig a certain number of sites, and I had the message : "url
rejected (level 1)".
I know the reason of that : my start_url is
http://www.domain.com/datas/index.html, and this page links to documents whose
the URL is http://www.domain.com/documents/page*.html. So of course, it doesn't
(i didn't exclude any urls, there wasn't any robots.txt)

I used a max hop of 1, and of course, the -i option.

It's not possible to use directly http://www.domain.com as a start_url, because
it would index too much pages.

Is there a solution ?

Christophe BAEGERT webmestre@biographie.net

>>>>>>> http://biographie.net <<<<<<<
 The first biographical search engine

To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word unsubscribe in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Sat Oct 23 1999 - 09:48:06 PDT