htdig: indexing a site

Adam Crews (
Tue, 8 Sep 1998 19:00:02 -0700 (PDT)


Is there any way to index a site without actually having the htdig program
request all the documents via the web daemon?

I bill my clients by the amount of bandwidth that they use. I have one
site that is about 40mb or so.. The indexing of the site is causing a
large skew in the actual bandwidth that they use. The search engine is a
"free" benefit of their site. I would like to be able to have htdig simply
read the html pages from a starting directory and then go from there...
Is this possible, and I just missed it in the the docs?


To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:27:42 PST