Re: [htdig] Some pages are parsed very slowly.

Subject: Re: [htdig] Some pages are parsed very slowly.
From: Geoff Hutchison (
Date: Thu May 11 2000 - 12:14:20 PDT

On Thu, 11 May 2000, NEPOTE Charles (Neuilly Gestion) wrote:

> I discovered that some pages needs very high CPU activity to be parsed by
> htdig.
> These pages are all about 40 Ko long and contains about 3 or 4 dozen of
> links such as folowing :
> Why ? Maybe the sharp (#) ?

It wouldn't be the sharp because this is ignored on the URL parser.
However, pages with 3 or 4 dozen links could push things up because each
link will need to be checked against the database.

I guess my first question would be if you have pages of 3-4 dozen links
without any sort of # mark. This would provide some element of comparison.

-Geoff Hutchison
Williams Students Online

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Thu May 11 2000 - 10:01:51 PDT