Re: [htdig] Including Pull-Down Menu Pages

Subject: Re: [htdig] Including Pull-Down Menu Pages
From: Douglas Kline (
Date: Fri Oct 20 2000 - 13:24:35 PDT

Thank you for your responses to my e-mails about security lapses of versions of
ht-Dig prior to v. 3.1.5, the issue of stability of a version, and using <LINK>
tags to enable ht-Dig to find Web pages referenced only through menu bars. I
have looked up these Web page references you mention and as I wrote earlier the
<LINK> tags enable the search engine to find the pages.

I think that the inability of the search engine to find pages referenced
through the menu bar and not by hyper-links is a significant disadvantage. Web
programmers who employ these menu bars may not know that they won't be
traversed by search engines and may not think about it and use <LINK> tags even
if they know. Whoever maintains the search engine may not know either. Even
if they know or find out eventually, actually installing <LINK> tags for all
pages referenced through menu bars and not through hyperlinks might be quite
difficult and even making sure that all Web programmers are aware of the need
for <LINK> tags for future use would be problematic.

Even if you are using the <LINK> tags, you could still be at a disadvantage.
You might put the tags for a menu file's links in any page which uses that menu
file but then you would have multiple copies of the <LINK> tags when only one
was necessary and, if the menu file changes or any of the links in it changes,
all files with the corresponding <LINK> tags have to change too. You could put
the <LINK> tags in just one file which calls the menu file but then if that
file changes and doesn't call the menu file or it's removed or the links by
which it is reached are removed, you would then have to put the <LINK> tags in
another file. If a page is reached by both a hyper-link and a menu bar and the
search engine finds it through the link and the link is later removed, then a
<LINK> tag will have to be added so that the search engine can still find it.
Ultimately all this is possible but keeping track of it and avoiding mistakes
will be much more difficult and the more Web pages and people involved, the
more difficult.

All this boils down to the fact that, in the case of these menu bars, Web pages
are reached by the search engine by a different means than that by which they
are reached by the user. That is the ultimate source of the problems I have
described. As these menu bars become more common, more pages will not be found
by search engines which their users expect to find them.

Nonetheless, we will be using the <LINK> tags for lack of an alternative.

The point you made in an earlier message about shopping carts and the
undesirability of indexing their contents suggests that a way of marking these
menus to be followed or not followed by search engines would be useful.

Douglas Kline

[Previous e-mails included in their entirety because they were sent nearly a
month ago. I'm just ready to respond now.]

> At 8:09 PM -0400 9/21/00, Douglas Kline wrote:
> >Fixed a nasty security hole in htsearch, which would allow users to view any
> >file on your site that had read permission.
> >
> >I would like to ask whether this security hole applies to all installations
> >ht-Dig v. 3.1.1
> Yes. AFAIK, it applies to absolutely every installation that's not
> using 3.1.5 or later.
> >Also, I would like to ask how the later version improves stability.
> By "stability" I would usually refer to the ability to crash things.
> I can't think of many changes that stopped segfaults (which we take
> quite seriously), but I can think of a few fixes (listed in
> RELEASE.html) that fixed potential infinite loops, a few fixes in the
> connection code that should keep connections from piling up, etc. If
> you want more details, see the release notes or the ChangeLog.
> [snip]
> >pull-down menus. Simple hyperlinks wouldn't do as well. They take up much
> >more space and to get a menu, you'd have to switch to the page with the menu
> >through a hyper-link and then, if you want to get to a page listed on the
> [snip]
> I'm not saying you need to use hyperlinks. I was just saying that
> there isn't a way for a search engine to read your menus of links.
> >However I could find no mention of these <LINK> tags in the Web page
> > you cite. Is there another reference or
> >did I miss a link or something?
> They're in all of the HTML specifications, I believe dating back to
> HTML 2.0. See in the sections on "Links." If you want a nice exact
> URL, try: <>
> --
> -Geoff Hutchison

> According to Douglas Kline:
> > Pursuant to your suggestion, we have installed v. 3.1.5. The only referenc
e to
> > an improvement in security of this version over v, 3.1.1 in the Release Not
> > to which you give Web page reference is:
> >
> > Fixed a nasty security hole in htsearch, which would allow users to view an
> > file on your site that had read permission.
> >
> > I would like to ask whether this security hole applies to all installations
> > ht-Dig v. 3.1.1 and, if not, how one can determine if it applies to a
> > particular installation. It isn't clear to me how a user could use htsearc
h to
> > view files which aren't Web pages which are indexed by the search engine's
> > database. This is of concern because there are other installations of ht-D
> > v.3.1.1 at this institution and we need to evaluate their security.
> This is explained in more detail in the FAQ, and the advisory which the
> FAQ points to. See
> --
> Gilles R. Detillieux E-mail: <>

Douglas Kline

To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.
List archives: <>
FAQ: <>

This archive was generated by hypermail 2b28 : Fri Oct 20 2000 - 13:30:08 PDT