[htdig] limit urls

Charlie Romero (charlie@jumpinternet.com)
Wed, 10 Mar 1999 15:21:34 +0000

• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
• Next message: Miguel A.L. Paraz: "[htdig] command line htsearch"
• Previous message: Gilles Detillieux: "Re: [htdig] Description-problem"

I am still in testing mode on a handful of servers. If you have a
/common/db.txt file for the start urls can you use the limit urls the same
way and reference the same start file for limits or duplicate the file for
limits (ie. /common/limitdb.txt)

Is this necessary to not crawl outside urls or is this overkill?

regarding the same, I saw in the archives that htdig would hang? with to
many start urls and limit urls. In 3.1.0b4 is this still an issue. I'm
having memory problems and I was wondering whether this was partly to blame.

Thanks,

Charlie

______________________________________________________________________
Charlie Romero
Director of Corporate Development
(703) 299-3585
mailto:charlie@jumpinternet.com
______________________________________________________________________
                       J U M P I N T E R N E T

INFORMATION MANAGEMENT SYSTEMS INTERNET/INTRANET SOLUTIONS

                      http://www.jumpinternet.com
______________________________________________________________________

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word "unsubscribe" in
the SUBJECT of the message.

• Next message: Miguel A.L. Paraz: "[htdig] command line htsearch"
• Previous message: Gilles Detillieux: "Re: [htdig] Description-problem"

This archive was generated by hypermail 2.0b3 on Mon Mar 15 1999 - 08:57:46 PST