[htdig] Re: "locale" problem in 3.2.0b2 (PR#885)

Subject: [htdig] Re: "locale" problem in 3.2.0b2 (PR#885)
From: Geoff Hutchison (ghutchis@wso.williams.edu)
Date: Mon Aug 21 2000 - 06:38:58 PDT

At 11:09 AM -0500 8/14/00, Gilles Detillieux wrote:
> > database that I have currently is 2,5 GB big. I am running it on
>RH 6.2, x86
> > . I am using Mylex RAID controller with 8 MB cache and 3x 18G 10.000 rpm
> > drives with 4 MB cache on each, RAID level 5 for database. system
>drives are
> > on level 1. Processor is 333 MHz with 192 MB ram (getting upgraded to dual
> > P3 733 , 512 MB ram this week). Searching takes very long time, 43 seconds
> > for word that gives 4700 hits. Word with 620 hits takes 6 seconds. And that
> > is with only "exact" method used.
> > How can I improve performance? What kind of indexing does htdig use
> > currently? Is there any plan for indexing in more levels?
> > Final database is going to be 4x bigger, and I cant affor to let
>people wait
> > 2+ minutes for hit :(
>I'm not really the person to ask, so if you don't get any other response,
>I suggest you repost to htdig@htdig.org as a separate thread. We've heard
>some reports about 3.2.0b2 being slow for searches (one said it was slow
>for the first few then sped up), but we haven't really gotten to the bottom
>of this. Other reports indicated 3.2 was faster than 3.1.x, which is more
>what we expected, due to the new database structure.

I'm taking this to the list myself because it merits response.
Searches in 3.2.x will definitely be faster, though certainly words
with few hits will almost always be faster. Benchmarking by Loic and
myself suggest that while indexing may be slower for some people,
truly large databases will see a performance improvement because the
new compression reduces disk I/O.

I'll also point out that performance is one of the areas that we're
working on heavily. The final 3.2.0 code will have result caching and
a few other methods to speed up searching and the final code will
also have faster indexing (than the betas) with some new database
code we're developing. When developing 3.2, we've been concerned with
improving the scaling and we hope we're getting there. It remains to
be seen how much will make 3.2.0b3 and how much is still under

-Geoff Hutchison
Williams Students Online

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig-unsubscribe@htdig.org You will receive a message to confirm this. List archives: <http://www.htdig.org/mail/menu.html> FAQ: <http://www.htdig.org/FAQ.html>

This archive was generated by hypermail 2b28 : Mon Aug 21 2000 - 06:48:02 PDT