Re: [htdig] SSL site index


Subject: Re: [htdig] SSL site index
From: gil cohen (cohengil4@hotmail.com)
Date: Tue Aug 29 2000 - 21:52:58 PDT


Look in the contrib dir, there are some patches for htdig that let it do SSL
sites/pages.

signed,
gil

>From: Walls Rob W MSgt 75 CS/SCX <Rob.Walls@HILL.af.mil>
>To: "'htdig@htdig.org'" <htdig@htdig.org>
>Subject: [htdig] SSL site index
>Date: Tue, 29 Aug 2000 17:56:34 -0600
>
>I have found htdig to be the fastest and most configurable indexer around.
>Now, just like everyone else it seems, my site must go secure. This
>question has been asked many times, but I'll ask it again...
>
>Can htdig index a secure site?
>
>One answer to this question from Browny said that htdig would now index
>https sites in version 3.1.4. I have 3.1.4. and htdig won't talk SSL.
>Would recompiling htdig with the openssl libs fix this?
>I am running Apache and the entire site is using SSL on port 443 (not even
>listening on port 80)
>
>Doing an "htdig -i -vvv" returns:
>------------------------------------------------------
>[root]:/web/htdig/bin >./htdig -i -vvv
> 1:0:"https://commweb.hill.af.mil:443/"
>New server: commweb.hill.af.mil, 443
>Retrieval command for http://commweb.hill.af.mil:443/robots.txt: GET
>/robots.t
>xt HTTP/1.0
>User-Agent: htdig/3.1.4 (root@commweb.hill.af.mil)
>Host: commweb.hill.af.mil
>
>Header line: HTTP/1.1 403 Forbidden
>Header line: Date: Tue, 29 Aug 2000 23:17:44 GMT
>Header line: Server: Apache/1.3.12 (Unix) mod_perl/1.24 mod_ssl/2.6.4
>OpenSSL/
>0.9.5a
>Header line: Connection: close
>Header line: Content-Type: text/html; charset=iso-8859-1
>Header line:
>returnStatus = 1
> pushed
>pick: commweb.hill.af.mil, # servers = 1
>0:0:0:"https://commweb.hill.af.mil:443/": Retrieval command for
>"https://commw
>eb.hill.af.mil:443/": GET /" HTTP/1.0
>User-Agent: htdig/3.1.4 (root@commweb.hill.af.mil)
>Host: commweb.hill.af.mil
>
>Header line: HTTP/1.1 403 Forbidden
>Header line: Date: Tue, 29 Aug 2000 23:17:44 GMT
>Header line: Server: Apache/1.3.12 (Unix) mod_perl/1.24 mod_ssl/2.6.4
>OpenSSL/
>0.9.5a
>Header line: Connection: close
>Header line: Content-Type: text/html; charset=iso-8859-1
>Header line:
>returnStatus = 1
> not found
>pick: commweb.hill.af.mil, # servers = 1
>--------------------------------------------------------
>
>Apache's error log contains the related errors:
>
>--------------------------------------------------------
>[Tue Aug 29 17:17:44 2000] [error] mod_ssl: SSL handshake failed: HTTP
>spoken
>on HTTPS port; trying to send HTML error page (OpenSSL library error
>follows)
>[Tue Aug 29 17:17:44 2000] [error] OpenSSL: error:1407609C:SSL
>routines:SSL23_
>GET_CLIENT_HELLO:http request [Hint: speaking HTTP to HTTPS port!?]
>[Tue Aug 29 17:17:44 2000] [error] access to
>/web/htdocs/mod_ssl:error:HTTP-re
>quest failed for commweb.hill.af.mil, reason: SSL connection required
>[Tue Aug 29 17:17:44 2000] [error] (2)No such file or directory: access to
>/we
>b/htdocs/error/403.htm failed for commweb.hill.af.mil, reason: SSL
>connection
>required
>[Tue Aug 29 17:17:44 2000] [error] mod_ssl: SSL handshake failed: HTTP
>spoken
>on HTTPS port; trying to send HTML error page (OpenSSL library error
>follows)
>[Tue Aug 29 17:17:44 2000] [error] OpenSSL: error:1407609C:SSL
>routines:SSL23_
>GET_CLIENT_HELLO:http request [Hint: speaking HTTP to HTTPS port!?]
>[Tue Aug 29 17:17:44 2000] [error] access to
>/web/htdocs/mod_ssl:error:HTTP-re
>quest failed for commweb.hill.af.mil, reason: SSL connection required
>[Tue Aug 29 17:17:44 2000] [error] (2)No such file or directory: access to
>/we
>b/htdocs/error/403.htm failed for commweb.hill.af.mil, reason: SSL
>connection
>required
>------------------------------------------------------
>
>So, does htdig do SSL and if so how?
>I would really hate to have to quit using the world's best indexer!
>
>Also, it seems a seperate port directive would be nice, unless the port is
>extracted from the start_url and used for every request (the links in the
>pages don't specify port #)
>
>Thanks,
>
>MSgt Rob W. Walls
>75CS/SCX
>DSN 777-2230
>CML (801) 777-2230
>rob.walls@hill.af.mil
>
>
>------------------------------------
>To unsubscribe from the htdig mailing list, send a message to
>htdig-unsubscribe@htdig.org
>You will receive a message to confirm this.
>List archives: <http://www.htdig.org/mail/menu.html>
>FAQ: <http://www.htdig.org/FAQ.html>
>

_________________________________________________________________________
Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com.

Share information about yourself, create your own public profile at
http://profiles.msn.com.

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-unsubscribe@htdig.org
You will receive a message to confirm this.
List archives: <http://www.htdig.org/mail/menu.html>
FAQ: <http://www.htdig.org/FAQ.html>



This archive was generated by hypermail 2b28 : Tue Aug 29 2000 - 21:54:35 PDT