[htdig] limit urls

Charlie Romero (charlie@jumpinternet.com)
Wed, 10 Mar 1999 15:21:34 +0000

I am still in testing mode on a handful of servers. If you have a
/common/db.txt file for the start urls can you use the limit urls the same
way and reference the same start file for limits or duplicate the file for
limits (ie. /common/limitdb.txt)

Is this necessary to not crawl outside urls or is this overkill?

regarding the same, I saw in the archives that htdig would hang? with to
many start urls and limit urls. In 3.1.0b4 is this still an issue. I'm
having memory problems and I was wondering whether this was partly to blame.



Charlie Romero
Director of Corporate Development
(703) 299-3585
                       J U M P I N T E R N E T



To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word "unsubscribe" in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Mon Mar 15 1999 - 08:57:46 PST