Re: [htdig] Limit to number of files

Subject: Re: [htdig] Limit to number of files
From: Gilles Detillieux (
Date: Wed Jul 12 2000 - 09:20:47 PDT

Do you have any really long lines in /tmp/all.urls? If I'm not mistaken,
the file expansion mechanism still breaks up lines longer than 1000
characters, which may lead to some unpredictable behaviour. A stack
backtrace from the core dump may be enlightening.

According to Robert Morse:
> Thanks for the reply. When I try running htdig -vvv it will
> dump core before printing anything out. I think the problem
> is with machine resources. It seems to be running out of memory
> with more than 2500 or so. I will try to stop some processes
> to get more memory then try running it with all 3500 or so
> files. Thanks.
> Jim Cole wrote:
> >
> > Robert E Morse's bits of Mon, 10 Jul 2000 translated to:
> >
> > >I have 3500 files in several directories that I would like to index,
> > >but they are not all referenced within an index.html. So I created
> > >a file that listed all the URL's seperately and put this line in the
> > >htdig.conf file:
> > >
> > >start_url: `/tmp/all.urls`
> > >
> > >and also put this in:
> > >
> > >max_doc_size: 4000000
> > >
> > >It keeps dumping core when I run the rundig command. When
> > >I trim the file down to about 2500 lines, it works fine. Is there
> > >a limit to how many files can be listed in the external file?
> >
> > Hi - I use external files for some of the sites I maintain. One of the
> > files currently has a little over 57000 lines, so I don't think the
> > number of lines is the problem. Have you tried running the commands
> > (htdig, htmerge, etc.) individually to see where it is dying? Maybe
> > with -vvv to see what is being done when it dies?

Gilles R. Detillieux              E-mail: <>
Spinal Cord Research Centre       WWW:
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Wed Jul 12 2000 - 06:36:45 PDT