Re: [htdig3-dev] htdig-3.2.0b1 slower than htdig-3.1.3 ?


Subject: Re: [htdig3-dev] htdig-3.2.0b1 slower than htdig-3.1.3 ?
From: March Walter Addison (wmarch@haverford.edu)
Date: Wed Feb 23 2000 - 10:56:38 PST


> > htdig 3.2.0b1 took
> > Tue Feb 22 09:08:44 EST 2000
> > Tue Feb 22 16:21:56 EST 2000
> > what is that? 7hrs and 10 minutes?
> >
>
> You can dramatically improve indexing speed by setting the following
> two parameters :
>
> wordlist_compress: true
> wordlist_cache_size: <the highest value you can afford on your machine>

Because I have a Solaris 2.6 system I compiled with
--without-zlib --disable-shared
 and when I ran htdig this time I got:

WordDB: not compiled with libz, compression not available
WordDB: Db::open: Invalid argument

Is this error from the without-zlib and can I recompile with zlib on a
solaris 2.6 machine?

> Are you indexing from a server on your intranet or from the internet ?

The indexer is a server on our intranet... in fact, it IS the server where
most of that stuff resides.

> Are you 100% sure the crawl/indexing time only depends on htdig ?

Well... no, of course not b/c the web servers are dealing with traffic and
the systems are doing their mysterious system things but I've run the
3.1.3 htdig under what should have been heavier usage then the 3.2 code
and 3.1.3 has consistantly taken less than 1.5 hrs.

thanks for your help.

------------------------------------
To unsubscribe from the htdig3-dev mailing list, send a message to
htdig3-dev-unsubscribe@htdig.org
You will receive a message to confirm this.



This archive was generated by hypermail 2b28 : Wed Feb 23 2000 - 11:01:49 PST