[htdig] Index of More that 2 Million html files

Lee Mathers (lee@help.nt.net)
Fri, 14 May 1999 13:11:19 -0400

I was wondering if someone could help me out with tuning up htdig to index
more than 2 million + straight html files, all the files are very small
about 200-500 bytes in size, the problem that I have is the directory tree
in which they are stored is massive, ie for each file the is about 10
directories ie. I've tried making an index.html for everything and pointing
it htdig at the index file but I seems to not read the complete index file,
(index file is extreamly large about 63mb's or 2.3million lines) So What I
have to do is allow apache to generate a dir listing and then point htdig
at it, The problem with doing this is that htdig then also indexes the
index which adds a whole lot of bloat to the db. If someone wants to help
with this I sure would appreciate it.
Lee Mathers
Operations Manager
AmikTech Inc.

To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word "unsubscribe" in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Fri May 14 1999 - 10:27:05 PDT