Re: [htdig] Spidering .asp sites?

Subject: Re: [htdig] Spidering .asp sites?
From: Gilles Detillieux (
Date: Wed Aug 02 2000 - 11:23:01 PDT

According to John Dispirito:
> I'm currently attempting to spider a site which uses .asp files, and it
> seems like
> the spider isn't working properly. I know its a fairly big site, and most of
> the time the output lists
> maybe at the most 20 -30 spidered entries from the site, and then it just
> stops...

I'd suggest running htdig -i -vvv and looking at the output to see if it's
seeing links to all the pages it should, and if so, why it would be rejecting
many of them. It could be a problem with your exclude_urls or limit_urls_to
settings. Note also that htdig only follows HTML links, not JavaScript.

Gilles R. Detillieux              E-mail: <>
Spinal Cord Research Centre       WWW:
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Wed Aug 02 2000 - 01:21:59 PDT