Subject: Re: [htdig] digging in external links
From: Torsten Neuer (tneuer@inwise.de)
Date: Thu Mar 30 2000 - 06:55:26 PST
Matthias Kleine - Patzschke + Rasp Software AG wrote:
>
> Hi there!
>
> Our internal document-system uses a lot of external links. Up to now,
> I didn't find a possibility to tell htdig to dig in the external
> document links, too. Is this possible?
Two ways to accomplish this:
- Clear the limit_urls_to directive
Of course, this is a dangerous thing to do, as Ht://Dig will then
follow
all external links in the external documents as well >:-]
- Have the external links extracted by a script and put into a file that
can be included into the start_url part of the configuration file.
Extracting the external links can be achieved by creating an URL-list
from the htdig run, pipe it through sort and uniq, then eliminate the
local URLs by piping the file further through sed or an awk script.
cheers,
Torsten
-- InWise - Wirtschaftlich-Wissenschaftlicher Internet Service GmbH Waldhofstraße 14 Tel: +49-4101-403605 D-25474 Ellerbek Fax: +49-4101-403606 E-Mail: info@inwise.de Internet: http://www.inwise.de------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig-unsubscribe@htdig.org You will receive a message to confirm this.
This archive was generated by hypermail 2b28 : Thu Mar 30 2000 - 05:54:56 PST