[htdig] scalability of htdig


Philip Jenkins (phil@ourside.com)
Mon, 15 Mar 1999 20:55:43 -0700


I was wondering if you could answer a couple of questions for me.
I am looking for a search engine for a site that I am putting up that
will cover a specific subject and I want the search engine to search
certain sites that are either submitted or linked from relating sites.
I
will be indexing about 1,000 to 3,000 remote sites, and probably around
5,000
to 25,000 documents. I have been looking at SWISH-E, SWISH++ and
ht://dig search engines. I have more impressed by far by ht://dig then
any others
that I have seen, and I am trying to stay with GNU software.
I noticed that some sites that use ht://dig have
over 5,000 items indexed. I was wondering if you could tell me how well
it
scales to larger sites. Also if you think the engine could handle as
many documents
as I am needing to do. Does ht://dig handle both Indexes like Yahoo and
normal searching?
How well does it crawl sites to index them, does it crash on large
sites?
One last question, I wanted to add a link to
have people submit there own sites, if I do this does ht://dig
automatically index them?

Thank you for you time.

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word "unsubscribe" in
the SUBJECT of the message.



This archive was generated by hypermail 2.0b3 on Wed Mar 17 1999 - 10:05:12 PST