Re: [htdig] Including Pull-Down Menu Pages

Subject: Re: [htdig] Including Pull-Down Menu Pages
From: Gilles Detillieux (
Date: Thu Oct 26 2000 - 12:00:13 PDT

Hey, guys. One of the great things about the htdig mailing list
is the lack of flame wars, so let's try to avoid accusations and
counter-accusations. Doug's request is a good one, but unfortunately
too technically difficult for us to consider.

According to Douglas Kline:
> > I don't think you quite understand the magnitude of your request.
> > Have you looked at all the other search engines?
> I pointed out a potentially serious disadvantage of the search engine. I
> requested nothing.
> > Can you name me
> > *any* which can indeed do that?
> No. I haven't spend the inordinate amount of time required to evaluate all the
> many search engines available. We've already installed ht-Dig and are trying
> to optimize our use of it

I think the word "disadvantage" implied a comparison to other engines,
which prompted Avi's question. As no search engine can index JavaScript
links, it's a level playing field with no one being at a disadvantage
in this regard. While it would be a great _advantage_ if htdig was the
only kid on the block that had this feature, this limitation doesn't
put htdig at a disadvantage with respect to the other engines.

I know that there are other definitions of disadvantage, so I'm not
arguing that your usage was incorrect - just that Avi and I both
understood a different meaning than you intended.

However, as Adam pointed out, a web design that only works on a JavaScript
enabled browser is a disadvantage. It won't work with many browsers other
than the leading two, and for them only if JavaScript isn't disabled
to close the gaping holes it opens. It probably also wouldn't meet
accessibility guidelines, if that's a concern for you. And of course,
there's the obvious disadvantage that no known web spider can traverse
the menus.

> > Blaming the ht://Dig Group that is just silly. It's open source --
> > if you think it should be done, you can always do it yourself.
> I didn't blame anybody. I pointed out a desideratum. What you do about it is
> another issue. This if-you-want-it-done-do-it-yourself idea is itself rather
> silly in my opinion. It would mean fully understanding the extensive workings
> of the ht-Dig code and the intricacies of all the contingencies with which it
> has to deal which essentially would put me in the business of single-handedly
> writing a search engine, not a decision one would arbitrarily make if one had
> anything else to do for the next few months.

I certainly didn't see any hint of laying blame in your earlier messages.
However, I have to disagree with you on the silliness of the idea of
if-you-want-it-done-do-it-yourself. In fact, as I see it this is the
way most open source software is developed. Sure, some people contribute
because they want to help others out, or because they like the challenge,
but that still comes down to wanting it done. ht://Dig wouldn't have
gone beyond what Andrew had originally written if it wasn't for others
who added features they wanted or needed, and wouldn't have gotten
started at all if it wasn't for Andrew seeing the need for this tool
and building it himself. That doesn't mean you can't benefit from a
feature if someone else wants it too, and gets around to implementing
it before you do, but in many cases a request will wind up back on the
shoulders of the person who made it, just because there's no one else
who wants it badly enough to do the necessary work.

I also disagree that extending the parsing capabilities of ht://Dig
would require fully understanding the extensive workings of the code.
If that were true, I'd never have begun contributing to the development
in the first place, because I'm not there yet. There are a lot of areas
that are out of my grasp. As it turns out, the parsers in ht://Dig are
pretty modular, so you can tweak and enhance one module without having
to worry too much about what the other modules are doing.

In your case, it would be a simple matter, assuming you had this
amazing JavaScript interpreter and predictive execution engine, of
hooking it into the htdig/ code so that the HTML parser would
pass a block of JavaScript to your interpreter, and it would figure
out all the URLs that this block of code could potentially generate.
Each URL can be given to Retriever::got_href(), just as does.
That's about the extent of the intricacies it would have to deal with.
The rest of the engine is done and could be used as-is.

Some changes, such as those which require entending the database, can
get a lot more hairy, but in this case the interconnection is almost
trivial, compared to the omminous task of actually figuring out all the
URLs that any arbitrary JS code can produce. A more site-specific hack,
which just deals with the URL formats you use in JS code at your site,
would likely be quite a bit easier to implement, but not generally useful
for other users, so a DIY solution probably is your best bet if you can't
avoid the JS-dependent menu design.

Gilles R. Detillieux              E-mail: <>
Spinal Cord Research Centre       WWW:
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this. List archives: <> FAQ: <>

This archive was generated by hypermail 2b28 : Thu Oct 26 2000 - 12:06:06 PDT