htdig: Tapping the data-stream from the digger - how?

Soeren A. Brandbyge (
Thu, 2 Apr 98 01:00:11 +0200

Hi all,

I have just been gazing through the last few month of messages from this
list but so far without any luck. I'm sure that someone, sometime within a
reasonable past mentioned htdig's ability to provide for "plug-in's". But I
can't find the description of how to do...
What I'm in need of:
I'm using htDig (that lovely little beast) daily to index a few sites. Some
pages get updated with some pre formatted data every day. The exact url's is
not known in advance (and it's beyond my control (customers customer site))
but I know the signature of the data signaling a page containing data. I
would like to "tap" the pages from the digger while it dig it's way through
the sites together with the exact url. It would be nice if the data could be
streamed into a file, or (preferably) into a pipe, but how to.....?

Have anybody a hint or (better still) some working code that could
accomplish the task?

Thank's in advance

To all of You who have contributed to htdig: I admire the work. I'm no
C-programmer - but that nice structured code is (almost) self explanatory.

Soeren Brandbyge

To unsubscribe from the htdig mailing list, send a message to containing the single word "unsubscribe" in
the body of the message.

This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:26:00 PST