Subject: Re: [htdig3-dev] Rejecting duplicates on htdig
From: Toivo Pedaste (toivo@ucs.uwa.edu.au)
Date: Mon Aug 21 2000 - 19:44:00 PDT
In local.h3dig-dev you write:
>>
>> That was my thinking too. I also felt that if duplicate detection was a
>> configuration attribute (potentially on a pre-URL or per-server basis),
>> then it wouldn't be as big of an issue.
>Oh, right. It didn't occur to me when reading Toivo's patches that this
>was not an optional feature. Making this feature selectable by a config
>attribute is a must, IMHO!
Yes, it think it should be optional, the duplicate detection and also
the extracting links from duplicate pages.
I've been running my htdig version against the web here for about
a day (it takes two days to complete) and looking at the output
the relative link problem is not a great one but I have found
one case of it happening so there will have to be a way of handling
it if it causes a problem, it won't be in the first version of
my code though.
-- Toivo Pedaste Email: toivo@ucs.uwa.edu.au University Communications Services, Phone: +61 8 9 380 2605 University of Western Australia Fax: +61 8 9 380 1109 "The time has come", the Walrus said, "to talk of many things"...------------------------------------ To unsubscribe from the htdig3-dev mailing list, send a message to htdig3-dev-unsubscribe@htdig.org You will receive a message to confirm this.
This archive was generated by hypermail 2b28 : Mon Aug 21 2000 - 19:44:30 PDT