Re: [htdig] Basics - Already searched archives

Subject: Re: [htdig] Basics - Already searched archives
From: Geoff Hutchison (
Date: Fri Oct 20 2000 - 12:46:47 PDT

On Fri, 20 Oct 2000, Melon, Jack wrote:

> I seem to be missing something to get it to drill down the directories. The
> index page on localhost is 3frames and its indexing the header.htm,
> main.htm, footer.htm - but that's all. The "header" and "main" frames both
> contain links to vast documentation in *.html, *.doc, *.txt, etc.; but it's
> not being retrieved. I've left the "limit_url_to:" attribute set to the
> default ${start_url}

I'd run htdig with the -vvv flag and take a look at the debugging output.
It will give you specific reasons for rejecting links.

> Also I'm getting an "Invalid Header- WWW Authenticate NTLM / Not Authorized"
> from the non-secured Microsoft IIS Server here. I've tried the -u
> username:password with no success. I understand that htdig is using BASIC
> encription and the IIS server is using "NT Challenge Response" so that's
> probably the problem, but I don't know a fix.

Wait. I thought you said you were using a "non-secured" server. Or are you
saying that it has password authorization but not HTTPS?

There's really nothing you can do on the ht://Dig side. Unless someone
steps forward to offer support for NT authentication, it's unlikely to
happen. I don't know of any of the active developers with access to an
NT-IIS machine and I wouldn't have the faintest idea how you'd support
that authentication.

You can always do a few things:
1) Convince the admin of the web box to make a "special case" in your
server config for the indexer. Keep in mind it's probably coming from a
paritcular IP and can have a specific user-agent field.
2) Grab the CygWin pakcage for running UNIX apps on Win32 machines and try
using local_filesystem digging. (No, I don't know if this works.)

-Geoff Hutchison
Williams Students Online

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this. List archives: <> FAQ: <>

This archive was generated by hypermail 2b28 : Fri Oct 20 2000 - 12:53:54 PDT