Virtual web limitation out of one config file?


weblord@lightspeed.net
Wed, 25 Nov 1998 08:44:38 -0800


Box htdig ran on: Pent Pro. 200 with 1/2 gig of ram, 36gig HD space, BSDI
3.1, Apache 1.3.0

I am running the following script on a config file containing approx. 1231
virtual domains.
I have no problem with it digging and compiling 992 domains but if I try
more it gives me
the "virtual memory exceeded" error. This is using version htdig-3.1.0b2. I
have limited the
search to the 0 hops to get only the fist page of each domain. Still no
change in the results. Is
there a limitation in how many virtual domains htdig can dig at once out of
one config file?

I was able to by-pass this by creating 2 seperate config files and running
htdig on each then htmerging them. Although this worked fine it would be
nice to only have to call 1 config file.

Chris Payne
Lightspeed Net System Administration
phone: 800-295-7873 x 343
fax: 805-324-2438
weblord@lightspeed.net

----------OLD Config-------------
# rundig
#
#
if [ "$1" = "-v" ]; then
    verbose=-v
fi

/var/www/docs/search.lightspeed.net/cgi-bin/bin/htdig -c
/var/www/docs/search.lightspeed.net/cgi-bin/conf/domains.conf -i $verbose -s
/var/www/docs/search.lightspeed.net/cgi-bin/bin/htmerge -c
/var/www/docs/search.lightspeed.net/cgi-bin/conf/domains.conf $verbose -s
/var/www/docs/search.lightspeed.net/cgi-bin/bin/htnotify -c
/var/www/docs/search.lightspeed.net/cgi-bin/conf/domains.conf $verbose

#
# Only create the endings database if it doesn't already exist.
# This database is static, so even if pages change, this database will not
# need to be rebuilt.
#
if [ ! -f
/var/www/docs/search.lightspeed.net/cgi-bin/common/word2root.gdbm ]
then
    /var/www/docs/search.lightspeed.net/cgi-bin/bin/htfuzzy -c
/var/www/docs/search.lightspeed.net/cgi-bin/conf/domains.conf $verbose
endings
fi

if [ ! -f /var/www/docs/search.lightspeed.net/cgi-bin/common/synonyms.gdbm ]
then
    /var/www/docs/search.lightspeed.net/cgi-bin/bin/htfuzzy -c
/var/www/docs/search.lightspeed.net/cgi-bin/conf/domains.conf $verbose
synonyms
fi

-----------NEW Config--------------
# rundig
#
#
if [ "$1" = "-v" ]; then
    verbose=-v
fi

/var/www/docs/search.lightspeed.net/cgi-bin/bin/htdig -c
/var/www/docs/search.lightspeed.net/cgi-bin/conf/domains.conf -i $verbose -s
# the above conf file contains less than 990 virtual webs
/var/www/docs/search.lightspeed.net/cgi-bin/bin/htdig -c
/var/www/docs/search.lightspeed.net/cgi-bin/conf/business.conf -i
$verbose -s
# the above conf file contains the remaining 120 + virtual webs
/var/www/docs/search.lightspeed.net/cgi-bin/bin/htmerge -c
/var/www/docs/search.lightspeed.net/cgi-bin/conf/domains.conf $verbose -s
/var/www/docs/search.lightspeed.net/cgi-bin/bin/htnotify -c
/var/www/docs/search.lightspeed.net/cgi-bin/conf/domains.conf $verbose

#
# Only create the endings database if it doesn't already exist.
# This database is static, so even if pages change, this database will not
# need to be rebuilt.
#
if [ ! -f
/var/www/docs/search.lightspeed.net/cgi-bin/common/word2root.gdbm ]
then
    /var/www/docs/search.lightspeed.net/cgi-bin/bin/htfuzzy -c
/var/www/docs/search.lightspeed.net/cgi-bin/conf/domains.conf $verbose
endings
fi

if [ ! -f /var/www/docs/search.lightspeed.net/cgi-bin/common/synonyms.gdbm ]
then
    /var/www/docs/search.lightspeed.net/cgi-bin/bin/htfuzzy -c
/var/www/docs/search.lightspeed.net/cgi-bin/conf/domains.conf $verbose
synonyms
fi



This archive was generated by hypermail 2.0b3 on Sat Jan 02 1999 - 16:28:53 PST