Subject: Re: [htdig] Spidering .asp sites?
From: Gilles Detillieux (email@example.com)
Date: Wed Aug 02 2000 - 11:23:01 PDT
According to John Dispirito:
> I'm currently attempting to spider a site which uses .asp files, and it
> seems like
> the spider isn't working properly. I know its a fairly big site, and most of
> the time the output lists
> maybe at the most 20 -30 spidered entries from the site, and then it just
I'd suggest running htdig -i -vvv and looking at the output to see if it's
seeing links to all the pages it should, and if so, why it would be rejecting
many of them. It could be a problem with your exclude_urls or limit_urls_to
-- Gilles R. Detillieux E-mail: <firstname.lastname@example.org> Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/~grdetil Dept. Physiology, U. of Manitoba Phone: (204)789-3766 Winnipeg, MB R3E 3J7 (Canada) Fax: (204)789-3930
------------------------------------ To unsubscribe from the htdig mailing list, send a message to email@example.com You will receive a message to confirm this.
This archive was generated by hypermail 2b28 : Wed Aug 02 2000 - 01:21:59 PDT