Barry Zubel (firstname.lastname@example.org)
Tue, 16 Mar 1999 09:50:06 -0000
I have a site that currently has, uhh, (quick peek) 90k documents on it,
(which increase by up to 1000 a day) and htdig is admirable. I run an update
every hour (admittedly the site is on a remote machine: the search machine
is a dedicated PII-400 with 256MB Ram) and it loves it. I've never had it
If I ever start having problems with volumes (unlikely, plenty of HDD space
and memory) I know that I'll get a quick response from the htdig developers:
thats what its all about.
City Mutual Ltd
From: Philip Jenkins <email@example.com>
To: firstname.lastname@example.org <email@example.com>
Date: Tuesday, March 16, 1999 6:11 AM
Subject: [htdig] scalability of htdig
>I was wondering if you could answer a couple of questions for me.
>I am looking for a search engine for a site that I am putting up that
>will cover a specific subject and I want the search engine to search
>certain sites that are either submitted or linked from relating sites.
>will be indexing about 1,000 to 3,000 remote sites, and probably around
>to 25,000 documents. I have been looking at SWISH-E, SWISH++ and
>ht://dig search engines. I have more impressed by far by ht://dig then
>that I have seen, and I am trying to stay with GNU software.
>I noticed that some sites that use ht://dig have
>over 5,000 items indexed. I was wondering if you could tell me how well
>scales to larger sites. Also if you think the engine could handle as
>as I am needing to do. Does ht://dig handle both Indexes like Yahoo and
>How well does it crawl sites to index them, does it crash on large
>One last question, I wanted to add a link to
>have people submit there own sites, if I do this does ht://dig
>automatically index them?
>Thank you for you time.
>To unsubscribe from the htdig mailing list, send a message to
>firstname.lastname@example.org containing the single word "unsubscribe" in
>the SUBJECT of the message.
To unsubscribe from the htdig mailing list, send a message to
email@example.com containing the single word "unsubscribe" in
the SUBJECT of the message.
This archive was generated by hypermail 2.0b3 on Wed Mar 17 1999 - 10:05:13 PST