Re: htdig: indexing pages without links


Geoff Hutchison (Geoffrey.R.Hutchison@williams.edu)
Wed, 30 Sep 1998 11:35:58 -0400


>I mean I've got a lot of individual pages in my directories but no page
>have links to these pages. How can I index (dig ?) these pages ? I know
>it's possible with EWS because EWS works with local pages and
>directories and reads the whole directory content. So even if there's no
>link to some pages they are indexed as well as linked ones.

I've discussed this with Andrew Bishop, the author of WWWoffle (a local
HTTP cache). The easiest way to do this is to write a small script that
generates a list of URLs into a document for the start_urls directive.

e.g. (in /bin/sh script)

for file in `ls dir1/* dir2/*`; do
  echo http://www.foo.com/$file >>start_url_file
done

This will produce a file of URLs:
http://www.foo.com/dir1/file.html
http://www.foo.com/dir2/file.html

>I've defined local_urls but it doesn't do what I want.

You'll need this defined to translate the URLs into files.

-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-request@sdsu.edu containing the single word "unsubscribe" in
the body of the message.



This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:27:54 PST