Re: [htdig] Recrusiv Digging

Michael Reutlinger (
Tue, 22 Jun 1999 19:44:21 +0200 (METDST)

Hi ...

 Thanx for your answer ...

> It does realize it saw a page. However, it's criteria is based on the URL.
> So if you have several URLs pointing to the same document, you're going to
> get duplicates. More powerful duplicate elimination code is in the works.

 On our Webserver System we have some MHonArc Archives, all with
 the same url !
 The whole Sever is about 3.500 Documents large and the last
 run told me, that htDig has about 55.000 Documents in its
 Database .. I think this is way to much ;)

 We had one "cross link" with different directories and only
 Filesystem Symlinks, but we eliminated this one (so there should
 be 100 files less ;))

 Actually i don't see a message like "already habe this document
 scipping everything inside ..."

 Do you have any idea about this ?



To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Tue Jun 22 1999 - 09:59:53 PDT