Re: [htdig] Including Pull-Down Menu Pages

Subject: Re: [htdig] Including Pull-Down Menu Pages
From: Geoff Hutchison (
Date: Fri Oct 20 2000 - 14:05:03 PDT

On Fri, 20 Oct 2000, Douglas Kline wrote:

> All this boils down to the fact that, in the case of these menu bars, Web pages
> are reached by the search engine by a different means than that by which they
> are reached by the user. That is the ultimate source of the problems I have
> described. As these menu bars become more common, more pages will not be found
> by search engines which their users expect to find them.

You make fine points and they're all valid. The problem is that there are
hundreds of non-standard ways of indicating links on the web now and only
a very few can be followed by search engines (ht://Dig is not alone in
this). It's all well and good to code a JavaScript navigation menu or an
actual form/CGI pair.

But there's no way for a search engine to fully parse JavaScript or know
that a specific form is for navigation and another isn't. For ht://Dig to
know, for example that a JavaScript menu is a navigation mechanism, it
would need to contain at least a significant portion of a full
interpreter to run the code. But then you hit the wall of
computation--even if you run the JavaScript code itself, you can't mimic
the user interaction. It's an "undecidable" problem to use the
mathematical term.

Now some have proposed that search engines should just "slurp" up all URLs
in a document as possible branch points. This would be possible if all
URLs were full and included, e.g. http://. But since most of us use
relative URLs, it would still need to recognize that "foo/" was likely to
be a relative URL.

Since you are running your own search engine with ht://Dig you also have
control over things like the start_url. In some cases, it may be more
efficient to generate a list of URLs outside of ht://Dig and put them in a
file before indexing or things along these lines. So beyond things like
tags, you can provide URLs through other means.

You also have access to the source and are free to modify it. (That's the
"free" of freedom in free software.) So if you feel your website needs
custom modifications to recognize your style of navigation, go to it.
We're willing to offer pointers into the code.

In short, it IS a problem, but it can only be solved by better web
standards from the HTML side of things (e.g. a stanrdard for representing
URLs for navigation purposes). But IMHO, you're *much* futher ahead of the
game with ht://Dig than with other packages for the reasons I just


-Geoff Hutchison
Williams Students Online

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this. List archives: <> FAQ: <>

This archive was generated by hypermail 2b28 : Fri Oct 20 2000 - 14:10:46 PDT