htdig: Re: indexing large seach engine data

Rolf Diederichs (
Tue, 8 Dec 1998 15:08:09 -0800 (PST)

>Date: Tue, 8 Dec 1998 15:31:05 -0600 (CST)
>From: <>
>Is/has anyone used htdig for indexing large seach engine data like yahoo
>or altavista? I understand the resources that are required to index such
>a large database, but I would still like to know if someone is/has doing

Recently we started to use htdig remote features for our Virtual Library.
Once we had some wrong Limit_URLs set, so we got some problems.
At the moment we don't take any risk and fetsh 40 MB of 2427 documents
of 165 Server. It took approx. 2 h, but most of the pages have been already
in the proxy server.
Everday we increase the content, we would like to do more, however not
We have in mind to do more than 10 000 pages,
I also would like to know how to estimate the limits.

   The e-Journal of Nondestructive Testing & Ultrasonics
                   Plus NDT online Exhibition
                * NDTnet - *
  NDT Internet Publishing Tel: +49(0)5221-769314
  Rolf Diederichs FAX: +49(0)5221-769731
  Tacheniusweg 8 Email:
  D-32052 Herford

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:29:49 PST