[htdig] Page ignored...

Subject: [htdig] Page ignored...
From: Paul Key (pkey@hcs.sghms.ac.uk)
Date: Tue Jan 04 2000 - 13:35:09 PST

HTDig 3_1_3 Sun Solaris 7

I have installed the above binary and am able to index and
search my website OK apart from one area.

I have a page in the website's document root - i.e. at the
same level as the home page (which in my case - the
homepage is an shtml file - I don't think this makes any
difference though).

I have a page in the same directory which is not linked
directly from the home page but is referenced from a page
'lower' in the site's directory structure.

However when I search for text I know is contained in this
page the page is not returned by the search.

The page being missed has no META noindex tag and I
perfomed the following index build to see the list of URL's

htdig -vvv - the first lines of output are listed below. I
have no robots.txt file and do not understand the
'retrieval command' line. Can I turn the robots.txt
request off?

New server: www.sghms.ac.uk, 80
Retrieval command for http://www.sghms.ac.uk/robots.txt: GET /robots.txt HTTP/1.
User-Agent: htdig/3.1.3 (pkey@sghms.ac.uk)
Host: www.sghms.ac.uk

Header line: HTTP/1.1 404 Not Found
Header line: Date: Tue, 04 Jan 2000 14:18:41 GMT
Header line: Server: Apache/1.3.9 (Unix)
Header line: Connection: close
Header line: Content-Type: text/html
Header line:
returnStatus = 1

Any ideas where I am going wrong and how I can include the
missing page in my search?



To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Tue Jan 04 2000 - 08:50:25 PST