Re: [htdig] Meta data


Subject: Re: [htdig] Meta data
From: Gilles Detillieux (grdetil@scrc.umanitoba.ca)
Date: Fri Jul 07 2000 - 09:16:19 PDT


According to Abdul Jabbar:
> Is there anyway to program Htdig to only search for Metadata and not index
> whole websites?. Some of the websites that I index Htdig absoultely
> cripples them is there anyway to slow down the relentless pressures? i.e.
> limit the number of hops?, please help. Thanks in advance.

See http://www.htdig.org/attrs.html#max_hop_count
    http://www.htdig.org/attrs.html#server_wait_time
and http://www.htdig.org/attrs.html#server_max_docs

You could set factors other than meta_description_factor and
keywords_factor to 0 so that only the meta data is used in searches,
but that will not affect in any way the performance impact htdig has
on the servers you're indexing, whereas the attributes mentioned above,
particularly server_wait_time, should help. You can also reduce load
on the servers by avoiding indexing dynamic content such as CGI scripts
(use exclude_urls), and by indexing via the local filesystem wherever
possible (local_urls).

-- 
Gilles R. Detillieux              E-mail: <grdetil@scrc.umanitoba.ca>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig-unsubscribe@htdig.org You will receive a message to confirm this.



This archive was generated by hypermail 2b28 : Fri Jul 07 2000 - 06:31:45 PDT