Re: [htdig] Indexing scope

Subject: Re: [htdig] Indexing scope
From: Geoff Hutchison (
Date: Mon Apr 17 2000 - 11:08:01 PDT

On Sun, 16 Apr 2000, Dave Lers wrote:

> So the second dig is always adding one hop to the local database_one, that
> works (I assume the local hops to local files/dirs that were already indexed
> pose no problems*). Do I have to mess with htdig-dbgen? That file makes just
> about 0 sense to me.

Yes, as I described it, the second dig is always adding one hop to the
previous one.

I'm not sure what you're talking about with "htdig-dbgen." It sounds like
a script provided by a binary package--it's not part of the source

> *How does Htdig handle those foo/?=D type auto indexes (an Apache thing?)?
> Watching dig I seem to remember a long run of *'s (I ran one search script that
> indexed these as separate URL's)

Sigh. If you have Apache's FancyIndexing turned on, you'll get links at
the top. Since these are links to "new pages" you'll get essentially
duplicate copies of these indexes, though the pages linked from them
aren't affected.

I usually add "?" to exclude_urls to get rid of these. There's not much
the indexer can do since they really are different pages.

> Pipes on both sides, no spaces, one URL per record/line. I could create a file,
> w/ just the URL's and add in the spaces, from the db...OT newbie question,
> could I put that Perl script in cron?

Yep. Or you could call it from the script that calls htdig, but just

-Geoff Hutchison
Williams Students Online

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Mon Apr 17 2000 - 08:53:54 PDT