[htdig] Redigging - Don't redig urls in DB


Tim Perdue (tim@phpbuilder.com)
Wed, 11 Aug 1999 09:18:55 -0700


It would be really nice, on an update dig, if htdig would not re-hit
pages that are already in its index. This creates a total nightmare on
large sites when you are trying to do an update dig.

Right now, I am trying to update dig the support forum on
PHPBuilder.com, but it is taking longer and longer every day because
htdig hits every single document (thousands of pages per day).

There's got to be a way to get around this without the hack that I use
on Geocrawler (dig the new pages, then merge the old and new document
databases)

Tim

-- 

PHPBuilder.com / Geocrawler.com

------------------------------------ To unsubscribe from the htdig mailing list, send a message to htdig@htdig.org containing the single word unsubscribe in the SUBJECT of the message.



This archive was generated by hypermail 2.0b3 on Wed Aug 11 1999 - 09:16:39 PDT