[htdig] robot meta tag?, index page?, "?" ?

Subject: [htdig] robot meta tag?, index page?, "?" ?
From: bob (gigs@ici.net)
Date: Wed Feb 09 2000 - 19:03:04 PST

I just put up a directory site and I have the robot no index and no
follow meta tag on some pages of this Web site to stop ht://Dig from
indexing a few pages on the site and following links from those pages.

The site is a directory site of related sites. What do I do to stop
ht://Dig from following links to other sites? Put "?" at the end of
a link to a site outside of the directory site? I tried it and the
link works to the site outside of the directory site. But, I have
no way of telling if ht://Dig stops at that point.

Ideally ht://Dig should not index the home page, but should follow
the links on the page. If I put the no index but follow meta tag on
the home page it will stop ht://Dig from indexing the page but I
should think that it would also stop most of the major Web indexing
robots from indexing the home page which is not good.

Should I put "?" after the URL of the home page? Or, should I start
an index page (for ht://Dig to start at) of all the URLs of the
pages of the site that I want ht://Dig to index? And, not have a
robot meta tag on the home page?

Where the sysad has ht://Dig installed (or parts of it):

/e-golddirectory.com/cgi-bin/htsearch (390832)
/e-golddirectory.com/htdig/mostly .gifs, .pngs, and htmls
/e-golddirectory.com/htsearch.cgi (390832)
/e-golddirectory.com/index.htm (home page)
/e-golddirectory.com/html/ (site's .html pages)



Use gold as money. It's easy. Create a free e-gold account here: http://www.e-gold.com/e-gold.asp?cid=101670

e-goldDirectory's PGP public key: http://www.e-golddirectory.com/assets/DH-DSSkey.txt Fingerprint: 3C4D A63F 3C8B 2D7B 7E1A FFE8 9A2E 4D78 CAD6 66B7

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig-unsubscribe@htdig.org You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Wed Feb 09 2000 - 19:05:40 PST