Subject: Re: [htdig] Quick question
From: Geoff Hutchison (firstname.lastname@example.org)
Date: Sun Mar 05 2000 - 13:05:38 PST
At 3:18 PM -0500 3/5/00, Glenn J. Rowe wrote:
>I haven't told it to index any of those pages. It shows hundreds of pages I
>didn't specify. For this reason it takes forever. I want to know if there
>is a way to stop it.
At 3:09 PM -0500 3/5/00, Glenn J. Rowe wrote:
I think you need to look very carefully through your sites.txt file.
It's entirely possible you have some small typo in there. Remember
that while start_url is going to ignore an invalid URL, limit_urls_to
acts as an OR on all the patterns it gets.
For example, let's say the file contains a line:
Well, this will be ignored pretty quickly by start_url, but it will
include almost every server on the web for limit_urls_to.
Williams Students Online
To unsubscribe from the htdig mailing list, send a message to
You will receive a message to confirm this.
This archive was generated by hypermail 2b28 : Sun Mar 05 2000 - 13:14:28 PST