RE: [htdig] Indexing MANY directories

Subject: RE: [htdig] Indexing MANY directories
From: Geoff Hutchison (
Date: Mon Jul 31 2000 - 11:43:20 PDT

At 1:26 PM -0400 7/31/00, Roger Weiss wrote:
>We're using Java Web Server, and I'm not aware of any files with built-in
>dir listings.
>However, wouldn't I need more than a dir listing? Wouldn't they have to be
>in url (hyperlink) format so htdig could follow it?

Webservers like Apache (in particular) generate an automatic
directory listing with links to directories and files inside that
directory. I believe Netscape's webserver as well as IIS do this too.

>This seems like a pretty basic feature...Am I missing something?

It's not really a "basic feature" of a web spider. There's no way
ht://Dig (or any other spider) can know anything about your
underlying directory structure through the webserver unless it sees
URL links. No links, no indexing, it's that simple. Since ht://Dig
can be used on remote webservers, this is the only way it can
operate. (It's a whole different ball of wax if you want to index
your local hard drive. But that's not your question.)

It IS, however, a fairly typical feature for webservers.

-Geoff Hutchison
Williams Students Online

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Mon Jul 31 2000 - 01:43:14 PDT