Re: [htdig] ISO Info on Funtionality


Subject: Re: [htdig] ISO Info on Funtionality
From: Gilles Detillieux (grdetil@scrc.umanitoba.ca)
Date: Thu Dec 16 1999 - 14:13:14 PST


According to Michael Johnson:
> I need a little information about how htdig handle header redirection.
> Here is my situation. htdig is running on a domain and is limited to
> searching only that domain. We want it to also index the first external
> page from any level. Example:
>
> 1. www.mysite.com is the site with htdig running
>
> 2. www.mysite.com/index.html links to www.someothersite.com/page.html
>
> 3. In an attempt to make htdig index www.someothersite.com/page.html I
> have done created a perl script that accepts 1 parameter, a url, and then
> prints 'Location: [url]. This causes a redirection to [url].
>
> So my ultimate question is will htdig index this page or not? Does it see
> that is a redirection and not follow the link?

htdig will follow redirects, but only if they fall within the scope of
limit_urls_to, which is set to the same as your start_url by default.
You'd need to change this to allow digging files from other sites, but
then it won't be limited to the first external page only.

-- 
Gilles R. Detillieux              E-mail: <grdetil@scrc.umanitoba.ca>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig-unsubscribe@htdig.org You will receive a message to confirm this.



This archive was generated by hypermail 2b28 : Thu Dec 16 1999 - 14:27:04 PST