Re: [htdig] how to search sub directories


gil (gil@truepath.com)
Fri, 09 Jul 1999 08:21:06 -0700


Hi,

I get a memory error when running htdig:
[root@reliant bin]# ./htdig -i -v -s
Virtual memory exceeded in `new'

I created a list of URLs by changing htdig.conf to include:
start_url: `${common_dir}/start.url`

start.url contains urls likes this:
http://www.cool.com/
http://member.truepath.com/joe/
etc...

Does anyone know why I get this error. The file contains about 12,000
URL's. Is that too many? Are the backticks necessary in the start_url
command?

Gil

On Fri, 9 Jul 1999 09:18:58 +0200
Torsten Neuer <tneuer@inwise.de> wrote:

>
> According to gil:
> >Hi,
> >
> >Is there no way of configuring htdig to look in a subdirectory such as
> >/usr/members that contains all the members' web sites that we manage?
> >
> >>From what I understand from the message below, I need to create an
> >index.html page that contains all of the urls for every member. There
> >are tens of thousands of members. We can write a short perl script to automate
> >the creation of a large index page listing all of the members pages, but
> >before doing so I want to make sure we're not overlooking an a more
> >straight forward approach.
> >
> >Gil
>
> Sure this is possible since everywhere in the configuration file you are
> able to include files (not only with the include directive but also by
> "inlining" them).
>
> Just create such a file for the start_url directive from the content
> of your userspace directories (by "ls -1 /your/memberdir" and some
> sed/awk reformatting) before calling the digger.
>
>
> hth,
> Torsten
>
> --
> InWise - Wirtschaftlich-Wissenschaftlicher Internet Service GmbH
> Waldhofstraße 14 Tel: +49-4101-403605
> D-25474 Ellerbek Fax: +49-4101-403606
> E-Mail: info@inwise.de Internet: http://www.inwise.de
>
> ------------------------------------
> To unsubscribe from the htdig mailing list, send a message to
> htdig@htdig.org containing the single word "unsubscribe" in
> the SUBJECT of the message.
>

in His grip,
Gil
http://www.truepath.com
your Christ-centered web host

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word "unsubscribe" in
the SUBJECT of the message.



This archive was generated by hypermail 2.0b3 on Fri Jul 09 1999 - 07:38:47 PDT