Re: htdig: indexing a site

Geoff Hutchison (
Wed, 09 Sep 1998 14:12:42 -0400

>I bill my clients by the amount of bandwidth that they use. I have one
>site that is about 40mb or so.. The indexing of the site is causing a
>large skew in the actual bandwidth that they use. The search engine is a
>"free" benefit of their site. I would like to be able to have htdig simply
>read the html pages from a starting directory and then go from there...

I'm not quite sure what you mean. Do you mean reading the files off of the
disk directly (rather than going through HTTP)? That's already here in the
form of a patch and will be here in the hopefully-soon-to-be-released

For those worried about such things, I don't forsee ht://Dig ever giving up
HTTP access. It will likely add additional access methods (like through the
filesystem), but once you have the code written, why would you give it up?

-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:27:46 PST