Re: [htdig] htdig performance on large databases

Geoff Hutchison (
Sat, 15 May 1999 16:30:38 -0400

>about 1.6 Gigs. Any less 'popular' comes up relatively faster. How do
>I improve the perfomance ?

For one, see

>I was thinking of a way of NOT sorting the results. That should make it
>faster. I mean, leave out the RELEVANCE factor and sorting out
>compleatly and spit out the results of the search. I think Inktomi does
>that - they don't sort for relevance or date or anything at all.

Maybe I'm just misunderstanding you, but what would you plan on doing with
unsorted results? IMHO users want search results sorted by *something*,
especially something they can understand. Relevance, Date and Title are on
the top of my lists as a user.

>Has anybody tried to put the database into a HUGE RAM disk (1 or 2 GB of
>RAM) to increase the spead and performance ? Should I switch to SCSI ?

Both of these would help. From what you say, your searches may be swapping
as well. If so, I'd start with at least 64MB more RAM.

>Is there a way of caching the results so that next time the search does
>not take so long ?

Not at the moment. It's a good idea, but it would also depend on how
frequently searches duplicate results. To some degree, the OS should do
this already--parts of the database files will be cached into RAM.

-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Sat May 15 1999 - 13:49:08 PDT