Subject: Re: [htdig3-dev] symlink bug
From: Geoff Hutchison (email@example.com)
Date: Fri Jul 28 2000 - 07:59:47 PDT
On Fri, 28 Jul 2000, Jonathan Bartlett wrote:
> I once wrote a spider program that ran into the same problem. The way I
> fixed it there was to have an option of the maximum URL size. This should
> prevent such a loop. The default could be infinite, or just a really huge
Nah, max_hop_count is IMHO a more elegant way of doing it. Who knows why
you might want to have some very long URL, but there's probably no reason
to be desending beyond some number of hops from your top page.
Of course a duplicate detection scheme (i.e. checksum the pages) would be
nice, but it doesn't look like that's going to happen unless someone
volunteers to do it soon.
To unsubscribe from the htdig3-dev mailing list, send a message to
You will receive a message to confirm this.
This archive was generated by hypermail 2b28 : Thu Jul 27 2000 - 21:58:22 PDT