Re: [htdig] digging in external links


Subject: Re: [htdig] digging in external links
From: Torsten Neuer (tneuer@inwise.de)
Date: Thu Mar 30 2000 - 06:55:26 PST


Matthias Kleine - Patzschke + Rasp Software AG wrote:
>
> Hi there!
>
> Our internal document-system uses a lot of external links. Up to now,
> I didn't find a possibility to tell htdig to dig in the external
> document links, too. Is this possible?

Two ways to accomplish this:

- Clear the limit_urls_to directive
  Of course, this is a dangerous thing to do, as Ht://Dig will then
follow
  all external links in the external documents as well >:-]

- Have the external links extracted by a script and put into a file that
  can be included into the start_url part of the configuration file.
  Extracting the external links can be achieved by creating an URL-list
  from the htdig run, pipe it through sort and uniq, then eliminate the
  local URLs by piping the file further through sed or an awk script.

cheers,
  Torsten

-- 
InWise - Wirtschaftlich-Wissenschaftlicher Internet Service GmbH
Waldhofstraße 14                            Tel: +49-4101-403605
D-25474 Ellerbek                            Fax: +49-4101-403606
E-Mail: info@inwise.de            Internet: http://www.inwise.de

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig-unsubscribe@htdig.org You will receive a message to confirm this.



This archive was generated by hypermail 2b28 : Thu Mar 30 2000 - 05:54:56 PST