[freenet-dev] BDB uses far more memory than it's supposed to; ideas?

[Jano]

Jano wrote:

> Matthew Toseland wrote:

>> How to identify such misuse?
> 
> Memory profiling, I'd say. Though I have never done it with java.

I've started toying with jmap/jhat after upgrading to java6. This node is a
linux one. Here's a heap dump at node start:

Top 12 Class histogram:

Class                                   Instance Count  Total Size
class [B                                        379353  18473891
class [I                                        315048  12805068
class [C                                        56549   11786932
class [[B                                       6137    3190228
class [Lcom.sleepycat.je.tree.Node;             6135    3190200
class [Ljava.util.HashMap$Entry;                33158   2950160
class [J                                        2037    2091504
class freenet.client.async.SingleBlockInserter  30965   1795970
class [[I                                       20812   1415092
class java.util.HashMap$Entry                   66778   1068448
class java.util.HashMap                         33144   1060608
class java.lang.String                          56831   909296

Top 10 class instance counts (excluding platform):

31204 instances of class freenet.client.FailureCodeTracker
30965 instances of class freenet.client.async.SingleBlockInserter
20604 instances of class freenet.support.io.ReadOnlyFileSliceBucket
16213 instances of class com.sleepycat.je.tree.LN
10413 instances of class freenet.support.io.FileBucket
10403 instances of class freenet.crypt.ciphers.Rijndael
10367 instances of class freenet.support.io.PaddedEphemerallyEncryptedBucket
10355 instances of class freenet.support.io.DelayedFreeBucket
9947 instances of class freenet.keys.FreenetURI
6157 instances of class com.sleepycat.je.latch.Java5SharedLatchImpl 

Total instances pending finalization: 0

The strange short names are, so it seems, "platform classes", which I don't
know what means (apparently java.* and javax.* classes, but these strange
names go away too).

And here's the heap dump at the moment of node death (node configured with
128m max mem):

Class                                   Instance Count  Total Size
class [B                                        771728  34242096
class [C                                        113887  19922068
class [I                                        439531  18758092
class [[B                                       9527    4953028
class [Lcom.sleepycat.je.tree.Node;             9525    4953000
class [Ljava.util.HashMap$Entry;                50089   4642912
class [J                                        3030    3111184
class com.sleepycat.je.tree.LN                  173461  2254993
class [[I                                       28586   1943756
class java.lang.String                          115987  1855792
class freenet.client.async.SingleBlockInserter  31080   1802640
class java.util.HashMap$Entry                   108956  1743296

173461 instances of class com.sleepycat.je.tree.LN
31413 instances of class freenet.client.FailureCodeTracker
31080 instances of class freenet.client.async.SingleBlockInserter
20613 instances of class freenet.support.io.ReadOnlyFileSliceBucket
17564 instances of class freenet.keys.FreenetURI
14291 instances of class freenet.crypt.ciphers.Rijndael
14246 instances of class freenet.support.io.PaddedEphemerallyEncryptedBucket
11793 instances of class freenet.support.io.FileBucket
10358 instances of class freenet.support.io.DelayedFreeBucket
10065 instances of class freenet.support.LRUQueue$QItem

Total instances pending finalization: 0

>From these numbers it seems that sleepycat is notably leaking memory (the LN
objects). Someone familiar with DBD could perhaps pinpoint seeing this if
this is a bug in BDB or a mismanagement in freenet (unclosed whatevers?)

I'm running now with 256m to see if the differences are even more apparent.
I'll send later some graphs obtained with jconsole that pretty much back
that there's leaking going on (i.e. if we find the leak, freenet will run
comfortably with 128m or less).