[htdig] Duplicate URLs?

Jim Cole (greyleaf@yggdrasill.net)
Sun, 27 Jun 1999 16:37:32 -0600


I have been trying to run htdig on a site that consists of about 180 MB
counting everything in the entire directory tree, a significant portion
of which is not even linked through any URL. However, the combined size
of the word and document databases are ending up in the 500 to 600 MB
range; currently way too big for me to host anywhere. While trying to
understand why so much disk space was being used, I examined the verbose
output of htdig and noticed that the same HTML files were scrolling by
dozens of times, each time with a slightly different path(?). A sample
of the output is shown below.

-**********-******---------*-----***--- size = 17511
-**********-******---------*-----***--- size = 17511
-**********-******---------*-----***--- size = 17511
-**********-******---------*-----***--- size = 17511
-**********-******---------*-----***--- size = 17511
-**********-******---------*-----***--- size = 17511

Is this normal behavior? Might it have something to do with why my
databases are becoming so large?

Except for changing start_url, database_dir, and maintainer, I am using
the default config file.

Any help would be appreciated.


Jim Cole
To unsubscribe from the htdig mailing list, send a message to
htdig@htdig.org containing the single word "unsubscribe" in
the SUBJECT of the message.

This archive was generated by hypermail 2.0b3 on Sun Jun 27 1999 - 14:55:27 PDT