[htdig] Performance problems with htdig 3.2.0b2


Subject: [htdig] Performance problems with htdig 3.2.0b2
From: Mathias Rohland (mro@gurus-net.de)
Date: Fri Jan 12 2001 - 10:09:49 PST


Hello to all on this list,

I have a problem with the performance of htdig 3.2.0b2. I'm indexing
about +25.500 HTML-docs at the moment and it takes several (+8) hours
to index them on a machine that's not to busy with outher tasks (PII 233
w/ 512K Cache and 128MB RAM).
All documents are active content coming from a MySQL database and are
being created using a php-script.
The machine is running SuSE Linux 7.0, Apache 1.3.x (no fancy_indexing)
w/ php4 and MySQL 3.22.x.

As all docs are created using one php-script I created a three-level
index stage where the 1st level contains only links to the 2nd that
contains up to 500 links to the script that actually creates the
content.

index.php3
   |
   +---index2.php3?START_ID=x&END_ID=x+499
   | |
   | +---content.php3?x (x being the docid - primary key in db)
   | |
   | +---content.php3?x+1
   | |
   | ...
   | |
   | +---content.php3?x+499
   |
   +---index2.php3?START_ID=x+500&END_ID=x+999
   |
   ...

The documents are not that big. The maximum size of one document is
definitely below 32KB.

Unfortunately I also have to index numbers as docids and several other
informations in this database consist of numbers only.

I need to use htdig 3.2.0b2 as we need phrase searching and a second
machine in another location that runs with solaris won't like 3.2.0b3.

The htdig.conf I'm using can be seen below:

<-- cut here --------------------------------------------------------->
#
# Example config file for ht://Dig.
#
# This configuration file is used by all the programs that make up
ht://Dig.
# Please refer to the attribute reference manual for more details on
what
# can be put into this file. (http://www.htdig.org/confindex.html)
# Note that most attributes have very reasonable default values so you
# really only have to add attributes here if you want to change the
defaults.
#
# What follows are some of the common attributes you might want to
change.
#

#
# Specify where the database files need to go. Make sure that there is
# plenty of free disk space available for the databases. They can get
# pretty big.
#
database_dir: /local/htdig/db

#
# This specifies the URL where the robot (htdig) will start. You can
specify
# multiple URLs here. Just separate them by some whitespace.
# The example here will cause the ht://Dig homepage and related pages to
be
# indexed.
# You could also index all the URLs in a file like so:
# start_url: `${common_dir}/start.url`
#
start_url: http://192.168.1.27/volltext/index.php3

#
# This attribute limits the scope of the indexing process. The default
is to
# set it to the same as the start_url above. This way only pages that
are on
# the sites specified in the start_url attribute will be indexed and it
will
# reject any URLs that go outside of those sites.
#
# Keep in mind that the value for this attribute is just a list of
string
# patterns. As long as URLs contain at least one of the patterns it will
be
# seen as part of the scope of the index.
#
#limit_urls_to: ${start_url}
limit_urls_to: index.php3 index2.php3 content.php3

#
# If there are particular pages that you definitely do NOT want to
index, you
# can use the exclude_urls attribute. The value is a list of string
patterns.
# If a URL matches any of the patterns, it will NOT be indexed. This is
# useful to exclude things like virtual web trees or database accesses.
By
# default, all CGI URLs will be excluded. (Note that the /cgi-bin/
convention
# may not work on your web server. Check the path prefix used on your
web
# server.)
#
exclude_urls: /cgi-bin/ .cgi search.php3

#
# Since ht://Dig does not (and cannot) parse every document type, this
# attribute is a list of strings (extensions) that will be ignored
during
# indexing. These are *only* checked at the end of a URL, whereas
# exclude_url patterns are matched anywhere.
#
# Also keep in mind that while other attributes allow regex, these must
be
# actual strings.
#
bad_extensions: .wav .gz .z .sit .au .zip .tar .hqx .exe .com .gif \
                .jpg .jpeg .aiff .class .map .ram .tgz .bin .rpm .mpg .mov .avi

#
# The string htdig will send in every request to identify the robot.
Change
# this to your email address.
#
maintainer: webmaster

#
# The excerpts that are displayed in long results rely on stored
information
# in the index databases. The compiled default only stores 512
characters of
# text from each document (this excludes any HTML markup...) If you
plan on
# using the excerpts you probably want to make this larger. The only
concern
# here is that more disk space is going to be needed to store the
additional
# information. Since disk space is cheap (! :-)) you might want to set
this
# to a value so that a large percentage of the documents that you are
going
# to be indexing are stored completely in the database. At SDSU we
found
# that by setting this value to about 50k the index would get 97% of all
# documents completely and only 3% was cut off at 50k. You probably
want to
# experiment with this value.
# Note that if you want to set this value low, you probably want to set
the
# excerpt_show_top attribute to false so that the top excerpt_length
characters
# of the document are always shown.
#
max_head_length: 10000

#
# To limit network connections, ht://Dig will only pull up to a certain
limit
# of bytes. This prevents the indexing from dying because the server
keeps
# sending information. However, several FAQs happen because people have
files
# bigger than the default limit of 100KB. This sets the default a bit
higher.
# (see <http://www.htdig.org/FAQ.html> for more)
#
max_doc_size: 200000

#
# Most people expect some sort of excerpt in results. By default, if the
# search words aren't found in context in the stored excerpt, htsearch
shows
# the text defined in the no_excerpt_text attribute:
# (None of the search words were found in the top of this document.)
# This attribute instead will show the top of the excerpt.
#
no_excerpt_show_top: true

#
# Depending on your needs, you might want to enable some of the fuzzy
search
# algorithms. There are several to choose from and you can use them in
any
# combination you feel comfortable with. Each algorithm will get a
weight
# assigned to it so that in combinations of algorithms, certain
algorithms get
# preference over others. Note that the weights only affect the ranking
of
# the results, not the actual searching.
# The available algorithms are:
# accents
# exact
# endings
# metaphone
# prefix
# regex
# soundex
# speling [sic]
# substring
# synonyms
# By default only the "exact" algorithm is used with weight 1.
# Note that if you are going to use the endings, metaphone, soundex,
accents,
# or synonyms algorithms, you will need to run htfuzzy to generate
# the databases they use.
#
search_algorithm: exact:1 synonyms:0.5 endings:0.1

#
# The following are the templates used in the builtin search results
# The default is to use compiled versions of these files, which produces
# slightly faster results. However, uncommenting these lines makes it
# very easy to change the format of search results.
# See <http://www.htdig.org/hts_templates.html for more details.
#
template_map: Normal normal ${common_dir}/normal.html Long long
${common_dir}/long.html Short short ${common_dir}/short.html
# template_name: long

# local_settings
common_dir: /local/htdig/share/htdig
database_base: ${database_dir}/db
doc_db: ${database_base}.docdb
doc_excerpt: ${database_base}.excerpts
word_db: ${database_base}.words.db
script_name: /volltext/search.php3
bin_dir: /local/htdig/bin
allow_numbers: true
logging: true
use_star_image: yes
image_url_prefix: /images/htdig
star_image: ${image_url_prefix}/star.gif
star_blank: ${image_url_prefix}/star_blank.gif
max_stars: 10
matches_per_page: 15
maximum_pages: 9
excerpt_length: 400
end_ellipses: ...
extra_word_characters: -_+
minimum_word_length: 2
no_title_text: "Kein Titel gefunden"
bad_word_list: ${common_dir}/bad_words.deutsch
synonym_db: ${database_base}.synonyms.deutsch.db
synonym_dictionary: ${common_dir}/synonyms.deutsch
endings_word2root_db: ${database_base}.word2root.db
endings_root2word_db: ${database_base}.root2word.db
soundex_db: ${database_base}.soundex.db
endings_affix_file: ${common_dir}/german.aff
endings_dictionary: ${common_dir}/german.0
locale: deutsch
backlink_factor: 0
# /local_settings

#
# The following are used to change the text for the page index.
# The defaults are just boring text numbers. These images spice
# up the result pages quite a bit. (Feel free to do whatever, though)
#
next_page_text: <img src="/images/htdig/buttonr.gif" border="0"
align="middle" width="30" height="30" alt="next">
no_next_page_text:
prev_page_text: <img src="/images/htdig/buttonl.gif" border="0"
align="middle" width="30" height="30" alt="prev">
no_prev_page_text:
page_number_text: '<img src="/images/htdig/button1.gif" border="0"
align="middle" width="30" height="30" alt="1">' \
                        '<img src="/images/htdig/button2.gif" border="0" align="middle"
width="30" height="30" alt="2">' \
                        '<img src="/images/htdig/button3.gif" border="0" align="middle"
width="30" height="30" alt="3">' \
                        '<img src="/images/htdig/button4.gif" border="0" align="middle"
width="30" height="30" alt="4">' \
                        '<img src="/images/htdig/button5.gif" border="0" align="middle"
width="30" height="30" alt="5">' \
                        '<img src="/images/htdig/button6.gif" border="0" align="middle"
width="30" height="30" alt="6">' \
                        '<img src="/images/htdig/button7.gif" border="0" align="middle"
width="30" height="30" alt="7">' \
                        '<img src="/images/htdig/button8.gif" border="0" align="middle"
width="30" height="30" alt="8">' \
                        '<img src="/images/htdig/button9.gif" border="0" align="middle"
width="30" height="30" alt="9">' \
                        '<img src="/images/htdig/button10.gif" border="0" align="middle"
width="30" height="30" alt="10">'
#
# To make the current page stand out, we will put a border around the
# image for that page.
#
no_page_number_text: '<img src="/images/htdig/button1.gif" border="2"
align="middle" width="30" height="30" alt="1">' \
                        '<img src="/images/htdig/button2.gif" border="2" align="middle"
width="30" height="30" alt="2">' \
                        '<img src="/images/htdig/button3.gif" border="2" align="middle"
width="30" height="30" alt="3">' \
                        '<img src="/images/htdig/button4.gif" border="2" align="middle"
width="30" height="30" alt="4">' \
                        '<img src="/images/htdig/button5.gif" border="2" align="middle"
width="30" height="30" alt="5">' \
                        '<img src="/images/htdig/button6.gif" border="2" align="middle"
width="30" height="30" alt="6">' \
                        '<img src="/images/htdig/button7.gif" border="2" align="middle"
width="30" height="30" alt="7">' \
                        '<img src="/images/htdig/button8.gif" border="2" align="middle"
width="30" height="30" alt="8">' \
                        '<img src="/images/htdig/button9.gif" border="2" align="middle"
width="30" height="30" alt="9">' \
                        '<img src="/images/htdig/button10.gif" border="2" align="middle"
width="30" height="30" alt="10">'

# local variables:
# mode: text
# eval: (if (eq window-system 'x) (progn (setq font-lock-keywords (list
'("^#.*" . font-lock-keyword-face) '("^[a-zA-Z][^ :]+" .
font-lock-function-name-face) '("[+$]*:" . font-lock-comment-face) ))
(font-lock-mode)))
# end:

<-- cut here --------------------------------------------------------->

The search process is also slow when I try to use phrase searching.
htsearch seems to eat up all CPU power it gets and uses about 9MB per
running instance.

Is there anything weird with the configuration I use?!? Sorry to ask
but I'm stuck and all I tried didn't help me to speed this up.

Thanks for your help in advance!

Mathias

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig-unsubscribe@htdig.org
You will receive a message to confirm this.
List archives: <http://www.htdig.org/mail/menu.html>
FAQ: <http://www.htdig.org/FAQ.html>



This archive was generated by hypermail 2b28 : Fri Jan 12 2001 - 10:23:48 PST