Subject: [htdig] A few points.
From: gil cohen (email@example.com)
Date: Mon May 15 2000 - 15:22:37 PDT
I have two questions:
Is there any way to benchmark htdig? Performance seems to degrade as it
crawls through more and more URLs. I've tried to use db.urls as a way to
benchmark, but I realize it's too inaccurate because the URLs that are in
there were just "seen," htdig wasn't there.
If it is true that htdig slows down after a while, do any of you know anyway
to make it faster? (I honestly don't care about the searching use for htdig.
All I'm using it for is getting a list of unique URLs from a site, and
passing it through my parser)
Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com
To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.
This archive was generated by hypermail 2b28 : Mon May 15 2000 - 13:10:57 PDT