Re: [htdig] Suggetions on Speeding up Search

Subject: Re: [htdig] Suggetions on Speeding up Search
From: Chad Phillips (
Date: Mon Jul 31 2000 - 12:28:08 PDT

I have run it from the server and from the command line and the speed is about the same.

>So if you're doing
>searches for very common words, it's bound to take longer since more
>matches must be scored.

I did play around with the searches, and your right. When I use criteria that causes fewer hits the search is much faster. So it looks like everything is normal. Would there be any advantage in using 3.2, since a lot of my searches may return many hits?


>>> Geoff Hutchison <> 07/31/00 01:38PM >>>
At 12:00 PM -0500 7/31/00, Chad Phillips wrote:
>I installed another version of htdig and set it to dig about 30
>sites. Everything went ok, but when I run a search it is extremely

How does the server respond? Does it seem like it's starting to page?
Have you tried running htsearch from the command-line? If so, does it
seem like the slowdown is from htsearch itself or from some sort of
webserver lag?

>The db for the second htdig, is about twice as big as the db for the
>first htdig. But the search takes about 5-10 longer. I set the
>search_algorithm to

Remember that it's not going to scale linearly with the disk space of
the database. That's an important factor since many searches are
bounded by the number of disk accesses. But the searches are more
closely bound to the number of word matches. So if you're doing
searches for very common words, it's bound to take longer since more
matches must be scored.

You said you're running this on a Sun box, so I can't say you should
switch to a faster disk. :-) But how much RAM does it have?

-Geoff Hutchison
Williams Students Online

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Mon Jul 31 2000 - 02:27:34 PDT