Subject: Re: [htdig] Somethin' Stupid
From: ATLE VEKA (ACVEKA@coe.edu)
Date: Fri Apr 14 2000 - 17:08:16 PDT
ht://Dig is doing exactly what it's told to, ignore pages that are marked off-limits for search engines. What I think you would be best off doing, is creating a text file with all the links in it and then just refer 'start_url' to that text file (just links, and no <a href> tags). It will then dig the links but not reference the file itself.
You should be able to find documentation on this (I can't remember where it's at right now).
Hope this helps,
>>> "Fanac Webmaster" <firstname.lastname@example.org> 04/14/00 04:31PM >>>
On the fanac.org site there is a cross reference listing of all the names that
I have been able to find that have been mentioned in the various documents that
the site holds. I do not want this listing to be indexed by ht://Dig because
all of these documents are reacable by other paths so I put <META NAME="robots"
CONTENT="noindex,nofollow"> in the header sectuion of its index. However I do
want to let the search engine to be able to find the entries in the index so I
created and html document (allnames.html) that looks like this:
To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.
This archive was generated by hypermail 2b28 : Fri Apr 14 2000 - 14:54:00 PDT