Re: [htdig] robot meta tag?, index page?, "?" ?

Subject: Re: [htdig] robot meta tag?, index page?, "?" ?
From: Geoff Hutchison (
Date: Wed Feb 09 2000 - 19:25:35 PST

At 10:03 PM -0500 2/9/00, bob wrote:
>The site is a directory site of related sites. What do I do to stop
>ht://Dig from following links to other sites? Put "?" at the end of

You can easily take care of this using the limit_urls_to attribute.
It is initially set to ${start_url} which will normally restrict it
to one server.

>Should I put "?" after the URL of the home page? Or, should I start
>an index page (for ht://Dig to start at) of all the URLs of the
>pages of the site that I want ht://Dig to index? And, not have a
>robot meta tag on the home page?

I don't see what the "?" would do unless you set that as a pattern in
exclude_urls. Besides, as a user I would find it really annoying. If
you really don't want ht://Dig to index the starting page but you do
want other search engines to do this, you will probably have to at
least put the links from the starting page into a separate file.

-Geoff Hutchison
Williams Students Online

To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Wed Feb 09 2000 - 19:37:28 PST