RE: [htdig] Large websites


plucas@frost.com
Mon, 1 Feb 1999 09:09:18 -0800


>> >I want to install an internal search on our webserver.
>> >We have currently over 10 thousand html-files available.
>> >Is this amount still managable with ht://Dig?
>>
>> Easily. I have no problems with around 58,000 URLs. I've received
>> reports
>> of close to 600,000 URLs. Some depends on the amount of disk space
>> you're
>> willing to give to the databases. :-)
>>
>> -Geoff
>>
>[Fels, Christian] How long does it take to search 600,000 URLs ? Which
>machine is used for this database ?

We have 600,000+ HTML documents now and searchtimes are in the 5-10 second
range on a Sun Enterprise 450/2250 running Solaris 2.6 with 640MB RAM and
multiple 9.1GB disks in a RAID 6 configuration.

The searches could be a little quicker but we have added an overhead with
some additional Java cgi programs.

Hope this helps.

Paul
==
Paul Lucas
Online System Administrator
Frost & Sullivan

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word "unsubscribe" in
the SUBJECT of the message.



This archive was generated by hypermail 2.0b3 on Wed Feb 10 1999 - 17:09:05 PST