Re: [htdig] CGI Question

Subject: Re: [htdig] CGI Question
From: Jeff (
Date: Thu Mar 30 2000 - 13:48:09 PST

On Thu, 30 Mar 2000, Tod Thomas wrote:

> I'm brand new to htdig so sorry for my newbiness...
> We have an internal site (a rather large one) driven completely by CGI.
> All pages are rendered dynamically, except for the first which acts as a
> doorway into the site and is standard HTML. When I run htdig it only
> indexes that top page and nothing else. Is there a way around this or
> does htdig only work with static html ?

You might want to look at the contents of your htdig.conf file, especially
in the areas of limit_urls_to, exclude_urls, and bad_extensions. The
mime-type of the content handed back from the server can affect the
indexing process as well -- this is a little less likely in your

Also, if your site uses/requires cookies for session management, you might
have some issues. I don't believe that htdig supports cookie usage at this
time (I've never had the occasion to check).

If your site uses authentication (ala basic auth, etc), you'll want to
look into the options that can enable this in your config file. Check the
htdig docs.

Hope this helps,


Jeff Godin
Network Specialist
Traverse Area District Library / Traverse Community Network - 231/932-8546 voice

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Thu Mar 30 2000 - 12:49:28 PST