Re: htdig: There has to be an easier way


Geoff Hutchison (Geoffrey.R.Hutchison@williams.edu)
Tue, 03 Nov 1998 08:54:43 -0500


At 10:16 AM -0500 10/24/98, Wayne Spivak wrote:
>I can't for the life of me get Htdig to dig though the entire tree.
>These are the pertinent parts of my config file:

I would run "htdig -vvv" and redirect this to a file. Then look through
this file to see why it is rejecting certain URLs. My first impression is
that since you're using the local_urls, that it's rejecting files that
don't end in .html since it can't parse .cgi files.

-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

>database_dir: /www/htdig/db_com
>start_url: http://com.sb.usps.org/
>limit_urls_to: com.sb.usps.org
>exclude_urls: /cgi-bin/
>max_head_length: 60000
>search_algorithm: exact:1 synonyms:0.5 endings:0.1 soundex:0.5
>metaphone:0.5
>maintainer: webmaster@sb.usps.org
>allow_numbers: true
>allow_virtual_hosts: true
>local_urls: http://com.sb.usps.org=/nat/
>create_url_list: true
>url_list: /www/htdig/url_list_com
>timeout: 42

----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-request@sdsu.edu containing the single word "unsubscribe" in
the body of the message.



This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:28:44 PST