[htdig] dissapearing urls?

Subject: [htdig] dissapearing urls?
From: Sasa Mutic (sas@siteop.org)
Date: Wed Aug 23 2000 - 02:36:10 PDT


I have a big start.url file with around 11000 URL's. I have splited it into 30 smaller files and run "htdig -s" "htmerge" after each completed. Everything was fine until I finished somethign like 4000 url's. (the db.url file was showing over 10 million at the time, but that is with many duplicates). Then I noticed that I dont get some hits anymore that I was getting before. For example, my own homepage that was indexed at start was among hits before, now I get 0 hits for the same word. Also noticed that hits for some other words have lowered.

It seems like htsearch doesnt search through whole db file anymore. Or maybe file is limited in zise and it truncates the beggining after adding more at the end??

db.docdb: 829570K
db.docs.index: 4447232
db.urls: 924091K
db.wordlist: 879319K
db.words.db: 699739K
total: 3,333 GB

Please help me,

Sasa Mutic

This archive was generated by hypermail 2b28 : Wed Aug 23 2000 - 02:33:58 PDT