Re: [htdig] digging in external links

Subject: Re: [htdig] digging in external links
From: Torsten Neuer (
Date: Thu Mar 30 2000 - 06:55:26 PST

Matthias Kleine - Patzschke + Rasp Software AG wrote:
> Hi there!
> Our internal document-system uses a lot of external links. Up to now,
> I didn't find a possibility to tell htdig to dig in the external
> document links, too. Is this possible?

Two ways to accomplish this:

- Clear the limit_urls_to directive
  Of course, this is a dangerous thing to do, as Ht://Dig will then
  all external links in the external documents as well >:-]

- Have the external links extracted by a script and put into a file that
  can be included into the start_url part of the configuration file.
  Extracting the external links can be achieved by creating an URL-list
  from the htdig run, pipe it through sort and uniq, then eliminate the
  local URLs by piping the file further through sed or an awk script.


InWise - Wirtschaftlich-Wissenschaftlicher Internet Service GmbH
Waldhofstraße 14                            Tel: +49-4101-403605
D-25474 Ellerbek                            Fax: +49-4101-403606
E-Mail:            Internet:

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Thu Mar 30 2000 - 05:54:56 PST