Re: [htdig] modification_time_is_now again

Subject: Re: [htdig] modification_time_is_now again
From: Gilles Detillieux (
Date: Fri Dec 03 1999 - 12:15:40 PST

According to Giancarlo Pinerolo:
> Gilles Detillieux wrote:
> > You lost me here. What's a 9999 dig? If I recall, there were still
> > some unresolved problems with hop counts during an update dig, but
> > as far as I can tell this is totally unrelated to modtime handling.
> > The modtime may affect whether a document is reindexed or not, but has
> > no effect on interpretation of hop counts.
> Please try the example. Quite spectacular to see the whole netscape site
> coming by (at least with 3.1.3)
> Giancarlo
> 1) initial dig:
> ---------htdig.conf
> database_dir: /home/htdig/db/test
> start_url:
> limit_urls_to: $start_url
> max_hop_count: 0
> modification_time_is_now: true
> ------ commands to execute
> /usr/sbin/htdig -v -s -t -i -h 0 -c htdig.conf>log
> /usr/sbin/htmerge -vv -s -c htdig.conf>>log
> 2) update dig
> --- htdig-u.conf
> common_dir: /home/htdig/common
> database_dir: /home/htdig/db/test
> start_url:
> limit_urls_to: $start_url
> max_hop_count: 0
> modification_time_is_now: false ### only difference
> ----command executed
> /usr/sbin/htdig -v -s -t -h 0 -c htdig-u.conf>>log

Yes, I was able to reproduce the error digging my own site. The only
effect that modification_time_is_now seems to have in all this is to
sometime trigger re-indexing of the document. The heart of the problem
seems to have something to do with hopcounts getting messed up, on not
properly set initially, in the db.docdb. I've been trying to get to the
bottom of this without much luck.

In the 3.2 development code, Geoff hacked it a bit so the initial hopcount
field is set to 0, instead of -1, when DocumentRef and URLRef objects are
first constructed. I don't know if that actually solves this problem or
not, but in any case it doesn't get to the root of the problem: what is
happening to those hopcounts in the first place?!

I have more testing to do with all of my recent changes to the 3.1.x code,
so I can't promise a quick answer, but I just wanted to let you know I've
reproduced the problem. I hope some of the other developers can help me
get to the bottom of this, and/or confirm whether Geoff's hack in 3.2 fixes
the problem.

Gilles R. Detillieux              E-mail: <>
Spinal Cord Research Centre       WWW:
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930

------------------------------------ To unsubscribe from the htdig mailing list, send a message to You will receive a message to confirm this.

This archive was generated by hypermail 2b28 : Fri Dec 03 1999 - 12:28:30 PST