From toad at amphibian.dyndns.org  Thu Feb  1 18:25:25 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Thu, 1 Feb 2007 18:25:25 +0000
Subject: [freenet-dev] FCP2 protocol incomplete for persistent requests
	handling
In-Reply-To: <d6149a790701301305s367480c7mc74d30749bf85935@mail.gmail.com>
References: <d6149a790701211248l1b905a72w353c5dbdcdc5c005@mail.gmail.com>
	<20070127193124.GA17168@amphibian.dyndns.org>
	<d6149a790701301305s367480c7mc74d30749bf85935@mail.gmail.com>
Message-ID: <20070201182525.GB28578@amphibian.dyndns.org>

On Tue, Jan 30, 2007 at 10:05:07PM +0100, bbackde at googlemail.com wrote:
> Found no existing bug to update.

#609 and #785.
> 
> Lets discuss this here in the public: after a WatchGlobal the client
> gets notified about the ongoing requests and their state. Then state
> changes of a request are sent (done, failed) or the progress of a
> request is sent. 

Yep.

> What I miss is a message if the request was removed
> from the global queue, or if the priority changed. This should also be
> sent by the node. Otherwise the clients have to send a
> ListPersistentRequest message periodically to keep informed about this
> kind of changes, which is some work and makes the whole concept of the
> auto-notify by the node incomplete. It should be not to hard to keep
> the client completely informed about each requests state without the
> need for ListPersistentRequest.

Sure, it's a bug. The practical effect of which is that clients are
rather more complex (and slower) than they need to be.
> 
> On 1/27/07, Matthew Toseland <toad at amphibian.dyndns.org> wrote:
> >On Sun, Jan 21, 2007 at 09:48:27PM +0100, bbackde at googlemail.com wrote:
> >> Is it a known issue that the FCP2 protocol is very incomplete
> >> regarding the handling of global queue items? Currently FCP2 clients
> >> are forced to periodically call ListPersistentRequests to keep up to
> >> date with the global queue items. This is very annoying and shows that
> >> the nodes implementation was only half done.
> >>
> >> Please provide full informations about the changes in the global
> >> queue, ideally no client have to use ListPersistentRequests after it
> >> sent a WatchGlobal message. Currently all the work is put onto the
> >> clients.
> >
> >Which changes are clients not notified about?
> >
> >I believe there are already some bugs related to this; please either
> >give me the bug numbers, comment on them, or otherwise file bugs and
> >indicate which ones are a priority for specific applications.
> >
> >
> >
> >-----BEGIN PGP SIGNATURE-----
> >Version: GnuPG v1.4.6 (GNU/Linux)
> >
> >iD8DBQFFu6iMA9rUluQ9pFARAjqJAJ9YIPHDFpkSVBU/a+2jR1l51rWliQCgnbco
> >fpbo67xA3FjiBQO5H5Y2zRY=
> >=28bC
> >-----END PGP SIGNATURE-----
> >
> >
> >
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070201/327bbb0e/attachment.pgp 

From toad at amphibian.dyndns.org  Thu Feb  1 18:37:08 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Thu, 1 Feb 2007 18:37:08 +0000
Subject: [freenet-dev] DDA autocheck
In-Reply-To: <d6149a790701301105u13f0d1d0w8fc833244d64041a@mail.gmail.com>
References: <d6149a790701162328x4bd40ca4i86de7d647725faa9@mail.gmail.com>
	<20070117073617.GA4061@freenetproject.org>
	<d6149a790701170016g4ad2f586ke8936c8c396b5736@mail.gmail.com>
	<20070130171834.GA15179@amphibian.dyndns.org>
	<d6149a790701301105u13f0d1d0w8fc833244d64041a@mail.gmail.com>
Message-ID: <20070201183708.GA11653@amphibian.dyndns.org>

On Tue, Jan 30, 2007 at 08:05:42PM +0100, bbackde at googlemail.com wrote:
> Frost uses DDA for all files, this saves some resources (transfer over
> socket to Frost). Is DDA to expensive in the node, should clients not
> use DDA for small files?

Well... DDA does use some resources (a file, a file handle, having to
remember to delete it etc; several sys calls)...

I would suggest using direct for message fetches.

For stuff that you want persistently on disk anyway, i.e. file
downloads, DDA is great; especially if you can queue a request to the
global queue, to download into the node's downloads/ dir (or Frost's
downloads dir?).
> 
> A TestDDA functionality in the node would be great.
> 
> Client to node:
> 
> TestDDA
> FileToWrite=c:\dir\file-123.tmp
> WriteContent =<unique token>
> FileToRead=c:\dir\file.456.tmp
> ReadContent=<unique token>
> EndMessage

For security reasons the actual written data must be generated by the
node. So drop WriteContent.
> 
> Node to client:
> 
> TestDDAResponse
> FileWritten=true
> FileRead=true
> ReadContent=<same unique read token>

Shouldn't that be WriteContent? It's the content that the node wrote to
the file...

> EndMessage

Okay, with the usual conditions - if the file already exists, the node
returns an error message.

How to deal with errors, in fact?

TestDDAResponse
FileRead=true
FileWritten=false
WriteFailureReason=Permission denied
WriteFailureCode=1
End

Meaning that the read test succeeded but the write test failed because
the node does not have permission to write the file.
> 
> This way the client can check if the file was really written (check
> unique token in file) and if the node was able to read the other
> unique token from the specified file. If something fails the client
> could still use DDA for PUTs (read allowed) or for GETs (write
> allowed).
> 
> Just an idea....

It's a good idea. I've filed a bug for it. If you want to implement it
yourself then assign it to yourself, otherwise I will get around to
it... eventually...

https://bugs.freenetproject.org/view.php?id=1086
> 
> On 1/30/07, Matthew Toseland <toad at amphibian.dyndns.org> wrote:
> > On Wed, Jan 17, 2007 at 09:16:56AM +0100, bbackde at googlemail.com wrote:
> > > I know there are many reasons why this could fail, but it would help
> > > for the most users.
> >
> > Yeah, it's something the node should provide IMHO.
> >
> > How? The client feeds the node a filename, the node generates a short
> > random hexadecimal string and writes it to the filename, then passes it
> > back to the client?
> >
> > What about exploiting the "downloads" directory? This is something we
> > know is accessible to the node, even if it isn't accessible to all
> > clients? Maybe there should be an FCP command to return the location of
> > the downloads directory, or even to create a subdirectory within it for
> > a specific client?
> > >
> > > On 1/17/07, Florent Daigni?re (NextGen$) <nextgens at freenetproject.org> wrote:
> > > > * bbackde at googlemail.com <bbackde at googlemail.com> [2007-01-17 08:28:41]:
> > > >
> > > > > Could the node provide a functionality to check DDA access to a given
> > > > > file? Currently Frost writes a file and tells the node to create the
> > > > > URI of this file. If the node can access the file then DDA is
> > > > > possible. But to check if the node can write to the file a GET would
> > > > > be needed, and this could take some time.
> > > > > I want a FCP2 command to let check the node read/write access to a
> > > > > given file to check if DDA is really possible.
> > > > >
> > > > > Thanks!
> > > >
> > > > Well, beeing able to access the file doesn't mean you are on the same
> > > > computer!
> > > >
> > > > It could be a mounted network filesystem or even worst ; a copy of the
> > > > files you are testing located in the same path.
> > > >
> > > > NextGen$
> > > >
> > > >
> > > > -----BEGIN PGP SIGNATURE-----
> > > > Version: GnuPG v1.4.6 (GNU/Linux)
> > > >
> > > > iD8DBQFFrdHxU/Z/dHFfxtcRAkvwAJ9E4pXrlHNXLk+ts3B15cdo/qhDpACfdbQs
> > > > o7tTz+m4dzIf9dXqNlOEDFQ=
> > > > =Bt2J
> > > > -----END PGP SIGNATURE-----
> > > >
> > > >
> > > > _______________________________________________
> > > > Devl mailing list
> > > > Devl at freenetproject.org
> > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> > > >
> > > >
> > > _______________________________________________
> > > Devl mailing list
> > > Devl at freenetproject.org
> > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> > >
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.6 (GNU/Linux)
> >
> > iD8DBQFFv33qA9rUluQ9pFARAiLeAJ9m4Rcng4Fq7AIwCrg8BiIlPeNfnwCgjhjJ
> > wsU2IPXx5++DbYYgZSPcc98=
> > =4uC5
> > -----END PGP SIGNATURE-----
> >
> > _______________________________________________
> > Devl mailing list
> > Devl at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> >
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070201/af9e3e25/attachment.pgp 

From toad at amphibian.dyndns.org  Thu Feb  1 18:44:17 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Thu, 1 Feb 2007 18:44:17 +0000
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
In-Reply-To: <1170193145.2232.449.camel@orchid.arb.net>
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
Message-ID: <20070201184417.GB11653@amphibian.dyndns.org>

On Tue, Jan 30, 2007 at 09:39:05PM +0000, Bob Ham wrote:
> On Tue, 2007-01-30 at 16:54 +0000, Matthew Toseland wrote:
> > After various reports of OOMs, and after high CPU usage which might have
> > been related to excessive GCing, I did some profiling.
> 
> I've done some profiling of my own:
> 
>   http://teasel.6gnip.net/~rah/freenet-threads-2007-01-30-21:23:18.png
> 
> The red line is the effective maximum memory that fred should use, as
> configured in wrapper.conf, in addition to the default DB memory usage.

The configured limit in wrapper.conf is absolute. The memory usage *as
reported by the JVM* will never exceed this.

I am not able to retrieve the png in the time it has taken to write this
email.

> The brown line is the actual memory usage.  The JVM in use is sun's,
> version 1.5.0, release 9.
> 
> It's clear to see that the configured limits are wholly ignored, and by
> significant percentages.  This is a major problem. 

On the contrary this is utterly irrelevant. It is Sun's problem, not our
problem; the JVM itself uses a significant amount of memory, and there
is literally nothing we can do about this. Also normally these reports
are usually the result of people taking the wrong memory measurement; in
linux there is RSS and VIRT; VIRT is more or less irrelevant.

> On my own machine,
> fred will get so large as to cause the kernel to start killing processes
> due to lack of memory.  These include named, cron, mysql and apache, in
> addition to fred itself.

What was the limit set to? What was the usage reported in the log file
(Memory in use: <blah> - only enabled with minor logging)? What was the
reported RSS? How much memory do you actually have (and swap)?
> 
> I only recently got my node back up after it refused to be able to start
> for some time.  This I've noted on the IRC, if you recall.  The problem
> would seem to be the size of the data store: after removing all of the
> data files, the node starts without any problems.  Needless to say, I've
> reduced the size now.
> 
> > Any ideas? We can:
> > - Just ignore it
> 
> Not an option in my opinion.

You have configured your memory limit too high for your actual RAM, or
something. This is irrelevant to the core issue, which is why does BDB
use so much memory.
> 
> > - Use another database. BDB has been fairly unreliable, hence the code
> >   in the store to reconstruct the database (store index) from the store
> >   file by deleting the database and parsing each key.
> > - Given the second item, if we had a reliable database we could store
> >   queued requests in it, thus limiting the overall memory usage
> >   regardless of the size of the request queue. But that would be a
> >   significant amount of work even with a database.
> 
> I think there are deeper issues to deal with. 

Please answer the questions I have posed above. It is likely that you
have configured your maximum memory to be greater than your total RAM or
something; this is absolutely not Fred's fault; it might conceivably be
a JVM problem.

> Even disregarding the
> database, fred's memory footprint is massive. 

Negative. I was running it with a maximum memory setting of 100MB, and
54MB+ of that was used by the database. If the database had respected
the configured setting of 20MB, that would be a grand total of 64MB.
Which IMHO is very reasonable.

> For the functionality
> that it provides, it seems to me to be excessive.  I can't understand
> why fred would need anything in excess of a few 10s of megabytes.  Where
> is this memory going?
> 
> Bob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070201/2fa9bd3c/attachment.pgp 

From toad at amphibian.dyndns.org  Thu Feb  1 18:45:29 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Thu, 1 Feb 2007 18:45:29 +0000
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
In-Reply-To: <epqrm4$1va$1@sea.gmane.org>
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
	<epqrm4$1va$1@sea.gmane.org>
Message-ID: <20070201184529.GC11653@amphibian.dyndns.org>

On Wed, Jan 31, 2007 at 08:46:14PM +0100, Jano wrote:
> Bob Ham wrote:
> 
> > I think there are deeper issues to deal with.  Even disregarding the
> > database, fred's memory footprint is massive.  For the functionality
> > that it provides, it seems to me to be excessive.  I can't understand
> > why fred would need anything in excess of a few 10s of megabytes.  Where
> > is this memory going?
> 
> I agree here. BDB seems a mature database; switching databases without
> really isolating the culprit seems premature.

Bob's "deeper issues" are illusory; local configuration problems.

BDB uses 54MB when I have told it to use 20MB. That is the issue here.
BDB is responsible for the majority of Fred's memory usage as of now as
far as I can see.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070201/0534c520/attachment.pgp 

From toad at amphibian.dyndns.org  Thu Feb  1 18:48:04 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Thu, 1 Feb 2007 18:48:04 +0000
Subject: [freenet-dev] [freenet-support] Houston, We have a problem
In-Reply-To: <45BF9587.7090601@WhenGendarmeSleeps.org>
References: <45B3BBF8.40407@WhenGendarmeSleeps.org>
	<005001c73db4$06504540$0a7b0001@deepspace3>
	<20070127194312.GA8459@amphibian.dyndns.org>
	<45BC8B80.4030105@WhenGendarmeSleeps.org>
	<20070130144551.GA13856@amphibian.dyndns.org>
	<45BF9587.7090601@WhenGendarmeSleeps.org>
Message-ID: <20070201184804.GD11653@amphibian.dyndns.org>

On Tue, Jan 30, 2007 at 06:59:19PM +0000, Volodya wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> > Possibly. But perhaps it's just "backoff sucks". Several of the simulations seemed to 
> > suggest this!
> 
> What do you mean here, please.

Many of the simulations showed that throttling without backoff was
superior to throttling with backoff. Backoff is intended to prevent
throttling from throttling the whole network down to nothing because of
a small number of very slow nodes causing lots of failures when too many
requests are routed to them.
> 
>                     - Volodya
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070201/1be1d12a/attachment.pgp 

From Volodya at WhenGendarmeSleeps.org  Thu Feb  1 21:59:39 2007
From: Volodya at WhenGendarmeSleeps.org (Volodya)
Date: Thu, 01 Feb 2007 21:59:39 +0000
Subject: [freenet-dev] [freenet-support] Houston, We have a problem
In-Reply-To: <20070201184804.GD11653@amphibian.dyndns.org>
References: <45B3BBF8.40407@WhenGendarmeSleeps.org>	<005001c73db4$06504540$0a7b0001@deepspace3>	<20070127194312.GA8459@amphibian.dyndns.org>	<45BC8B80.4030105@WhenGendarmeSleeps.org>	<20070130144551.GA13856@amphibian.dyndns.org>	<45BF9587.7090601@WhenGendarmeSleeps.org>
	<20070201184804.GD11653@amphibian.dyndns.org>
Message-ID: <45C262CB.1050206@WhenGendarmeSleeps.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> Many of the simulations showed that throttling without backoff was
> superior to throttling with backoff. Backoff is intended to prevent
> throttling from throttling the whole network down to nothing because of
> a small number of very slow nodes causing lots of failures when too many
> requests are routed to them.

Thanks for the answer.

I still am lost as to why all my connections are backed off. I have noticed that shutting
down everything but freenet makes the problem less. At least for a couple of hours after
start up. My CPU usage is at 100% all the time currently, so i think that it might be
because my node just doesn't comprehend what is being asked of it fast enough and
therefore it backs the peers off. This in turn creates the state where all the peers are
backed off and thus no place to rout through, something very weird begins to happen after
that, i am still looking for any correlation between what i've been doing and the node's
performance.

- --
http://freedom.libsyn.com/       Voice of Freedom, Radical Podcast
http://freeselfdefence.info/     Self-defence wiki
http://www.kingstonstudents.org/ Kingston University students' forum

 "None of us are free until all of us are free."    ~ Mihail Bakunin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

iD8DBQFFwmLKuWy2EFICg+0RAkgyAJ9giqDFKq3LLcl34bjlJTcCstiViACfcw9g
intqHjyRivswvqGiDgaTQj0=
=lNhO
-----END PGP SIGNATURE-----


From alejandro at mosteo.com  Thu Feb  1 23:05:20 2007
From: alejandro at mosteo.com (Jano)
Date: Fri, 02 Feb 2007 00:05:20 +0100
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
	<epqrm4$1va$1@sea.gmane.org>
	<20070201184529.GC11653@amphibian.dyndns.org>
Message-ID: <eptrne$c7r$1@sea.gmane.org>

Matthew Toseland wrote:

> On Wed, Jan 31, 2007 at 08:46:14PM +0100, Jano wrote:
>> Bob Ham wrote:
>> 
>> > I think there are deeper issues to deal with.  Even disregarding the
>> > database, fred's memory footprint is massive.  For the functionality
>> > that it provides, it seems to me to be excessive.  I can't understand
>> > why fred would need anything in excess of a few 10s of megabytes. 
>> > Where is this memory going?
>> 
>> I agree here. BDB seems a mature database; switching databases without
>> really isolating the culprit seems premature.
> 
> Bob's "deeper issues" are illusory; local configuration problems.
> 
> BDB uses 54MB when I have told it to use 20MB. That is the issue here.
> BDB is responsible for the majority of Fred's memory usage as of now as
> far as I can see.

But why? Can we be sure that the fault is not in some kind of misuse of BDB
instead of being intrinsic to BDB?



From rah at bash.sh  Fri Feb  2 01:29:54 2007
From: rah at bash.sh (Bob Ham)
Date: Fri, 02 Feb 2007 01:29:54 +0000
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
In-Reply-To: <20070201184417.GB11653@amphibian.dyndns.org>
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
	<20070201184417.GB11653@amphibian.dyndns.org>
Message-ID: <1170379794.2232.477.camel@orchid.arb.net>

On Thu, 2007-02-01 at 18:44 +0000, Matthew Toseland wrote:
> On Tue, Jan 30, 2007 at 09:39:05PM +0000, Bob Ham wrote:

> > It's clear to see that the configured limits are wholly ignored, and by
> > significant percentages.  This is a major problem. 
> 
> On the contrary this is utterly irrelevant.

Not when my named gets killed.

> It is Sun's problem, not our
> problem

Yes, but the limits are still being wholly ignored.  Which JVM would you
suggest running fred with that will respect configured memory limits?

> What was the limit set to?

196 MB

> What was the usage reported in the log file
> (Memory in use: <blah> - only enabled with minor logging)?

Which log file; wrapper log or fred log?  The statistics for the "java
memory allocated" and "java memory used" come straight off of the
statistics page of the web interface.

> What was the
> reported RSS?

What you see plotted.

> How much memory do you actually have (and swap)?

512 MB physical, 478 MB swap

> > Even disregarding the
> > database, fred's memory footprint is massive. 
> 
> Negative. I was running it with a maximum memory setting of 100MB, and
> 54MB+ of that was used by the database. If the database had respected
> the configured setting of 20MB, that would be a grand total of 64MB.
> Which IMHO is very reasonable.

Here is a plot with a maximum setting of 64MB, taken a couple of weeks
ago:

  http://teasel.6gnip.net/~rah/freenet-threads-2007-02-02-01:17:33.png

I've put it up here as well in case you have problems:

  http://pkl.net/~node/freenet-threads-2007-02-02-01:17:33.png

As you can see, there are quite a few "wrapper rstrt" labels; meaning
the wrapper has restarted the node.  The reason in these instances
appear to be "OOM" messages.  I'll set my memory limit to 100MB and see
what kind of stability results.

Bob

-- 
Bob Ham <rah at bash.sh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070202/f47d5462/attachment.pgp 

From rah at bash.sh  Fri Feb  2 02:50:53 2007
From: rah at bash.sh (Bob Ham)
Date: Fri, 02 Feb 2007 02:50:53 +0000
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
In-Reply-To: <20070201184417.GB11653@amphibian.dyndns.org>
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
	<20070201184417.GB11653@amphibian.dyndns.org>
Message-ID: <1170384654.2232.500.camel@orchid.arb.net>

On Thu, 2007-02-01 at 18:44 +0000, Matthew Toseland wrote:

> that would be a grand total of 64MB.
> Which IMHO is very reasonable.

As noted, that's pretty massive.  I don't think it's reasonable at all.
Here's some stats for comparable daemons that I quickly plucked out of a
"ps axu" on the box I'm running fred:

named	    3.17 MB
imapd	    2.85 MB
mysqld	   17.33 MB
apache2	    7.82 MB
postgres    6.47 MB
fred	  120.82 MB

Now, to me, fred is seeming a little out of place there.  Even if it was
64MB, that's still three times the size of mysql's footprint.

What, exactly, requires all this memory?

Bob

-- 
Bob Ham <rah at bash.sh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070202/a91e535e/attachment.pgp 

From bbackde at googlemail.com  Fri Feb  2 08:00:17 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Fri, 2 Feb 2007 09:00:17 +0100
Subject: [freenet-dev] Freenet compile failed
Message-ID: <d6149a790702020000y118cca51g9f508f8ac971f894@mail.gmail.com>

With the latest svn source, the ant compile failed, what is wrong?

compile:
    [javac] Compiling 362 source files to C:\Projects\fn07\build
    [javac] C:\Projects\fn07\src\freenet\node\fcp\ClientRequest.java:313:
reference to Bucket is ambiguous, both class freenet.support.io.Bucket
in freenet.support.io and class freenet.support.api.Bucket in
freenet.support.api match
    [javac]     protected void bucketToFS(SimpleFieldSet fs, String
name, boolean includeSize, Bucket data) {
    [javac]
                        ^
    [javac] 1 error

BUILD FAILED


From nextgens at freenetproject.org  Fri Feb  2 08:23:00 2007
From: nextgens at freenetproject.org (NextGen$)
Date: Fri, 2 Feb 2007 08:23:00 +0000
Subject: [freenet-dev] Freenet compile failed
In-Reply-To: <d6149a790702020000y118cca51g9f508f8ac971f894@mail.gmail.com>
References: <d6149a790702020000y118cca51g9f508f8ac971f894@mail.gmail.com>
Message-ID: <20070202082300.GH20976@freenetproject.org>

* bbackde at googlemail.com <bbackde at googlemail.com> [2007-02-02 09:00:17]:

> With the latest svn source, the ant compile failed, what is wrong?
> 
> compile:
>     [javac] Compiling 362 source files to C:\Projects\fn07\build
>     [javac] C:\Projects\fn07\src\freenet\node\fcp\ClientRequest.java:313:
> reference to Bucket is ambiguous, both class freenet.support.io.Bucket
> in freenet.support.io and class freenet.support.api.Bucket in
> freenet.support.api match
>     [javac]     protected void bucketToFS(SimpleFieldSet fs, String
> name, boolean includeSize, Bucket data) {
>     [javac]
>                         ^
>     [javac] 1 error
> 
> BUILD FAILED

You forgot to cleanup your working copy:

svn revert -R . && ant clean dist

NextGen$


From nextgens at freenetproject.org  Fri Feb  2 08:41:12 2007
From: nextgens at freenetproject.org (NextGen$)
Date: Fri, 2 Feb 2007 08:41:12 +0000
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
In-Reply-To: <1170384654.2232.500.camel@orchid.arb.net>
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
	<20070201184417.GB11653@amphibian.dyndns.org>
	<1170384654.2232.500.camel@orchid.arb.net>
Message-ID: <20070202084112.GI20976@freenetproject.org>

* Bob Ham <rah at bash.sh> [2007-02-02 02:50:53]:

> On Thu, 2007-02-01 at 18:44 +0000, Matthew Toseland wrote:
> 
> > that would be a grand total of 64MB.
> > Which IMHO is very reasonable.
> 
> As noted, that's pretty massive.  I don't think it's reasonable at all.
> Here's some stats for comparable daemons that I quickly plucked out of a
> "ps axu" on the box I'm running fred:
> 
> named	    3.17 MB
> imapd	    2.85 MB
> mysqld	   17.33 MB
> apache2	    7.82 MB
> postgres    6.47 MB
> fred	  120.82 MB
> 
> Now, to me, fred is seeming a little out of place there.  Even if it was
> 64MB, that's still three times the size of mysql's footprint.

How big is your mysql database ?

emu:~# du -hs /var/lib/mysql
117M    /var/lib/mysql

emu:~# ps auxf|grep mysql
mysql    17981  0.4  5.6 137184 25120 ?        Sl    2006 359:08  \_
/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql
--pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306
--socket=/var/run/mysqld/mysqld.sock

On the server we have ~100M of data and the memory footprint is ~25M. The
usual size for a datastore is 1G...

> What, exactly, requires all this memory?
> 
> Bob
> 
> -- 
> Bob Ham <rah at bash.sh>


From jflesch at nerim.net  Fri Feb  2 12:32:53 2007
From: jflesch at nerim.net (Jerome Flesch)
Date: Fri, 2 Feb 2007 13:32:53 +0100
Subject: [freenet-dev] r11648 : added ack of RemovePersistentRequest (new
	message PersistentRequestRemoved)
Message-ID: <a033de740702020432n367b5056k98464dea64245fb6@mail.gmail.com>

"new message PersistentRequestRemoved" ? I'm a little bit curious
about this new message:
Wasn't the node supposed to send a GetFailed ("cancelled by user")
when a request was removed ? If yes, what is the goal of this new
message ?


From bbackde at googlemail.com  Fri Feb  2 13:17:56 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Fri, 2 Feb 2007 14:17:56 +0100
Subject: [freenet-dev] r11648 : added ack of RemovePersistentRequest
	(new message PersistentRequestRemoved)
In-Reply-To: <d6149a790702020515i7e040d9as76c1273d766b5bba@mail.gmail.com>
References: <a033de740702020432n367b5056k98464dea64245fb6@mail.gmail.com>
	<d6149a790702020507w2ead20dx40956a0e0839573d@mail.gmail.com>
	<d6149a790702020515i7e040d9as76c1273d766b5bba@mail.gmail.com>
Message-ID: <d6149a790702020517x1f360c5bj606553bb7a6b6589@mail.gmail.com>

Forgot to reply to devl too...

On 2/2/07, bbackde at googlemail.com <bbackde at googlemail.com> wrote:
> Toad wrote:
> > GetFailed reason=Cancelled ? Admittedly this doesn't tell you for
> > certain that it has been removed... Hmm...
>
> So either a new message, or a new parameter reason=Removed would give
> a clear statement that the request was removed. Whereas the new
> message makes it easier for clients to handle the removed request,
> otherwise you have to do all inside the handler for GetFailed. My 2
> cents....
>
>
> On 2/2/07, bbackde at googlemail.com <bbackde at googlemail.com> wrote:
> > Did you follow the thread on tech yesterday? Thats why I added a new
> > msg....no problem to change it again.
> >
> > On 2/2/07, Jerome Flesch <jflesch at nerim.net> wrote:
> > > "new message PersistentRequestRemoved" ? I'm a little bit curious
> > > about this new message:
> > > Wasn't the node supposed to send a GetFailed ("cancelled by user")
> > > when a request was removed ? If yes, what is the goal of this new
> > > message ?
> > >
> >
>


From rah at bash.sh  Fri Feb  2 13:30:55 2007
From: rah at bash.sh (Bob Ham)
Date: Fri, 02 Feb 2007 13:30:55 +0000
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
In-Reply-To: <20070202084112.GI20976@freenetproject.org>
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
	<20070201184417.GB11653@amphibian.dyndns.org>
	<1170384654.2232.500.camel@orchid.arb.net>
	<20070202084112.GI20976@freenetproject.org>
Message-ID: <1170423056.2232.508.camel@orchid.arb.net>

On Fri, 2007-02-02 at 08:41 +0000, NextGen$ wrote:
> On the server we have ~100M of data and the memory footprint is ~25M. The
> usual size for a datastore is 1G...

Let me requote some mails here; you've obviously missed the point:

On Thu, 2007-02-01 at 18:44 +0000, Matthew Toseland wrote: 
> On Tue, Jan 30, 2007 at 09:39:05PM +0000, Bob Ham wrote:

> > Even disregarding the
> > database, fred's memory footprint is massive. 

> Negative. I was running it with a maximum memory setting of 100MB, and
> 54MB+ of that was used by the database. If the database had respected
> the configured setting of 20MB, that would be a grand total of 64MB.
> Which IMHO is very reasonable.


We're not talking about the memory requirements for the datastore.  They
are, as you can in fact see above, explicitly excluded from the
discussion.

I'll rephrase.  Even if fred (not including the datastore) uses 64 MB,
that's still 8 times the memory footprint of apache.  This is excessive.

Bob

-- 
Bob Ham <rah at bash.sh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070202/c80c22d5/attachment.pgp 

From jflesch at nerim.net  Fri Feb  2 13:39:53 2007
From: jflesch at nerim.net (Jerome Flesch)
Date: Fri, 2 Feb 2007 14:39:53 +0100
Subject: [freenet-dev] r11648 : added ack of RemovePersistentRequest
	(new message PersistentRequestRemoved)
In-Reply-To: <d6149a790702020533l165179e7od7a6337004ce1ac7@mail.gmail.com>
References: <a033de740702020432n367b5056k98464dea64245fb6@mail.gmail.com>
	<d6149a790702020507w2ead20dx40956a0e0839573d@mail.gmail.com>
	<d6149a790702020515i7e040d9as76c1273d766b5bba@mail.gmail.com>
	<a033de740702020525q181bd4a8q9635101eec19085c@mail.gmail.com>
	<d6149a790702020533l165179e7od7a6337004ce1ac7@mail.gmail.com>
Message-ID: <a033de740702020539j62e6cf2eo5a72467b9c1090fd@mail.gmail.com>

Ok, but why a new command ? Why not just another error code ? (xx =
cancelled by user ; yy = cancelled and removed by user)

Anyway, as long as the GetFailed command is still sent, it won't break
my implementation. I'm just asking that by curiosity.


2007/2/2, bbackde at googlemail.com <bbackde at googlemail.com>:
> imho cancel means both: request stopped for now, or request removed.
> Due to extendability of the protocol the parameters should be clear,
> so the answer for a RemovePersistentRequest should be something with
> the word "removed", same as requested. We should not start to
> introduce such ambiguities in the early state of 0.7.
>
> On 2/2/07, Jerome Flesch <jflesch at gmail.com> wrote:
> > Since there is no cancel command in FCP, I don't see what "cancelled
> > by user" can mean except that the user removed it ?
> >
> >
> > 2007/2/2, bbackde at googlemail.com <bbackde at googlemail.com>:
> > > Toad wrote:
> > > > GetFailed reason=Cancelled ? Admittedly this doesn't tell you for
> > > > certain that it has been removed... Hmm...
> > >
> > > So either a new message, or a new parameter reason=Removed would give
> > > a clear statement that the request was removed. Whereas the new
> > > message makes it easier for clients to handle the removed request,
> > > otherwise you have to do all inside the handler for GetFailed. My 2
> > > cents....
> > >
> > >
> > > On 2/2/07, bbackde at googlemail.com <bbackde at googlemail.com> wrote:
> > > > Did you follow the thread on tech yesterday? Thats why I added a new
> > > > msg....no problem to change it again.
> > > >
> > > > On 2/2/07, Jerome Flesch <jflesch at nerim.net> wrote:
> > > > > "new message PersistentRequestRemoved" ? I'm a little bit curious
> > > > > about this new message:
> > > > > Wasn't the node supposed to send a GetFailed ("cancelled by user")
> > > > > when a request was removed ? If yes, what is the goal of this new
> > > > > message ?
> > > > >
> > > >
> > >
> >
>


From bbackde at googlemail.com  Fri Feb  2 13:50:51 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Fri, 2 Feb 2007 14:50:51 +0100
Subject: [freenet-dev] r11648 : added ack of RemovePersistentRequest
	(new message PersistentRequestRemoved)
In-Reply-To: <a033de740702020539j62e6cf2eo5a72467b9c1090fd@mail.gmail.com>
References: <a033de740702020432n367b5056k98464dea64245fb6@mail.gmail.com>
	<d6149a790702020507w2ead20dx40956a0e0839573d@mail.gmail.com>
	<d6149a790702020515i7e040d9as76c1273d766b5bba@mail.gmail.com>
	<a033de740702020525q181bd4a8q9635101eec19085c@mail.gmail.com>
	<d6149a790702020533l165179e7od7a6337004ce1ac7@mail.gmail.com>
	<a033de740702020539j62e6cf2eo5a72467b9c1090fd@mail.gmail.com>
Message-ID: <d6149a790702020550m3cf488f2l664da1845a3082a5@mail.gmail.com>

I answered to this in my mail from 2:15pm. It was only my idea to add
a new message because _I_ thought it is easier for clients this way.
It is no problem to change the GetFailed message and to provide a
"reason=removed" to the client.

We just need a decision and I will make the changes. Lets wait until
some more people joined the discussion and voted for the one or the
other :)

PS: imho a GetFailed for removal of a request is'nt really correct,
because nothing failed. See my comments about ambiguities in my before
message. But to keep the overall number of messages low it could make
sense to do it this way.

On 2/2/07, Jerome Flesch <jflesch at nerim.net> wrote:
> Ok, but why a new command ? Why not just another error code ? (xx =
> cancelled by user ; yy = cancelled and removed by user)
>
> Anyway, as long as the GetFailed command is still sent, it won't break
> my implementation. I'm just asking that by curiosity.
>
>
> 2007/2/2, bbackde at googlemail.com <bbackde at googlemail.com>:
> > imho cancel means both: request stopped for now, or request removed.
> > Due to extendability of the protocol the parameters should be clear,
> > so the answer for a RemovePersistentRequest should be something with
> > the word "removed", same as requested. We should not start to
> > introduce such ambiguities in the early state of 0.7.
> >
> > On 2/2/07, Jerome Flesch <jflesch at gmail.com> wrote:
> > > Since there is no cancel command in FCP, I don't see what "cancelled
> > > by user" can mean except that the user removed it ?
> > >
> > >
> > > 2007/2/2, bbackde at googlemail.com <bbackde at googlemail.com>:
> > > > Toad wrote:
> > > > > GetFailed reason=Cancelled ? Admittedly this doesn't tell you for
> > > > > certain that it has been removed... Hmm...
> > > >
> > > > So either a new message, or a new parameter reason=Removed would give
> > > > a clear statement that the request was removed. Whereas the new
> > > > message makes it easier for clients to handle the removed request,
> > > > otherwise you have to do all inside the handler for GetFailed. My 2
> > > > cents....
> > > >
> > > >
> > > > On 2/2/07, bbackde at googlemail.com <bbackde at googlemail.com> wrote:
> > > > > Did you follow the thread on tech yesterday? Thats why I added a new
> > > > > msg....no problem to change it again.
> > > > >
> > > > > On 2/2/07, Jerome Flesch <jflesch at nerim.net> wrote:
> > > > > > "new message PersistentRequestRemoved" ? I'm a little bit curious
> > > > > > about this new message:
> > > > > > Wasn't the node supposed to send a GetFailed ("cancelled by user")
> > > > > > when a request was removed ? If yes, what is the goal of this new
> > > > > > message ?
> > > > > >
> > > > >
> > > >
> > >
> >
>


From toad at amphibian.dyndns.org  Fri Feb  2 19:32:41 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Fri, 2 Feb 2007 19:32:41 +0000
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
In-Reply-To: <eptrne$c7r$1@sea.gmane.org>
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
	<epqrm4$1va$1@sea.gmane.org>
	<20070201184529.GC11653@amphibian.dyndns.org>
	<eptrne$c7r$1@sea.gmane.org>
Message-ID: <20070202193241.GA29915@amphibian.dyndns.org>

On Fri, Feb 02, 2007 at 12:05:20AM +0100, Jano wrote:
> Matthew Toseland wrote:
> 
> > On Wed, Jan 31, 2007 at 08:46:14PM +0100, Jano wrote:
> >> Bob Ham wrote:
> >> 
> >> > I think there are deeper issues to deal with.  Even disregarding the
> >> > database, fred's memory footprint is massive.  For the functionality
> >> > that it provides, it seems to me to be excessive.  I can't understand
> >> > why fred would need anything in excess of a few 10s of megabytes. 
> >> > Where is this memory going?
> >> 
> >> I agree here. BDB seems a mature database; switching databases without
> >> really isolating the culprit seems premature.
> > 
> > Bob's "deeper issues" are illusory; local configuration problems.
> > 
> > BDB uses 54MB when I have told it to use 20MB. That is the issue here.
> > BDB is responsible for the majority of Fred's memory usage as of now as
> > far as I can see.
> 
> But why? Can we be sure that the fault is not in some kind of misuse of BDB
> instead of being intrinsic to BDB?

How to identify such misuse?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070202/9c7578d8/attachment.pgp 

From nextgens at freenetproject.org  Fri Feb  2 21:28:57 2007
From: nextgens at freenetproject.org (NextGen$)
Date: Fri, 2 Feb 2007 21:28:57 +0000
Subject: [freenet-dev] [freenet-cvs] r11648 -
	trunk/freenet/src/freenet/node/fcp
In-Reply-To: <20070202114242.0F3D99BC24@emu.freenetproject.org>
References: <20070202114242.0F3D99BC24@emu.freenetproject.org>
Message-ID: <20070202212857.GK20976@freenetproject.org>

* bback at freenetproject.org <bback at freenetproject.org> [2007-02-02 11:42:41]:

> Author: bback
> Date: 2007-02-02 11:42:40 +0000 (Fri, 02 Feb 2007)
> New Revision: 11648
> 
> Modified:
>    trunk/freenet/src/freenet/node/fcp/ClientGet.java
>    trunk/freenet/src/freenet/node/fcp/ClientPutBase.java
>    trunk/freenet/src/freenet/node/fcp/ClientRequest.java
>    trunk/freenet/src/freenet/node/fcp/ModifyPersistentRequest.java
>    trunk/freenet/src/freenet/node/fcp/RemovePersistentRequest.java
> Log:
> - added ack of ModifyPersistentRequest
> - !!! added ack of RemovePersistentRequest (new message PersistentRequestRemoved)
> 

[snip.]

> Modified: trunk/freenet/src/freenet/node/fcp/ClientRequest.java
> ===================================================================
> --- trunk/freenet/src/freenet/node/fcp/ClientRequest.java	2007-01-31 19:33:47 UTC (rev 11647)
> +++ trunk/freenet/src/freenet/node/fcp/ClientRequest.java	2007-02-02 11:42:40 UTC (rev 11648)
> @@ -1,20 +1,13 @@
>  package freenet.node.fcp;
>  
> -import java.io.BufferedReader;
> -import java.io.BufferedWriter;
> -import java.io.IOException;
> -import java.net.MalformedURLException;
> +import java.io.*;
> +import java.net.*;
>  
> -import freenet.client.async.ClientRequester;
> -import freenet.keys.FreenetURI;
> -import freenet.support.Fields;
> -import freenet.support.HexUtil;
> -import freenet.support.Logger;
> -import freenet.support.SimpleFieldSet;
> -import freenet.support.api.Bucket;
> -import freenet.support.io.FileBucket;
> -import freenet.support.io.PaddedEphemerallyEncryptedBucket;
> -import freenet.support.io.SerializableToFieldSetBucket;
> +import freenet.client.async.*;
> +import freenet.keys.*;
> +import freenet.support.*;
> +import freenet.support.api.*;
> +import freenet.support.io.*;
>  
[snip.]


Is there any ground for doing things like that ? generalizing imports.
I would say it's against the current coding standards used by the
team^weclipse^wtoad ;)

NextGen$


From bbackde at googlemail.com  Fri Feb  2 21:33:32 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Fri, 2 Feb 2007 22:33:32 +0100
Subject: [freenet-dev] [freenet-cvs] r11648 -
	trunk/freenet/src/freenet/node/fcp
In-Reply-To: <20070202212857.GK20976@freenetproject.org>
References: <20070202114242.0F3D99BC24@emu.freenetproject.org>
	<20070202212857.GK20976@freenetproject.org>
Message-ID: <d6149a790702021333o68267784g663697539fc2daa2@mail.gmail.com>

There is no ground and no reason. It was just a misuse of eclipse I
didn't even noticed. Please change it back, I will take care of this
in the future.

On 2/2/07, NextGen$ <nextgens at freenetproject.org> wrote:
> * bback at freenetproject.org <bback at freenetproject.org> [2007-02-02 11:42:41]:
>
> > Author: bback
> > Date: 2007-02-02 11:42:40 +0000 (Fri, 02 Feb 2007)
> > New Revision: 11648
> >
> > Modified:
> >    trunk/freenet/src/freenet/node/fcp/ClientGet.java
> >    trunk/freenet/src/freenet/node/fcp/ClientPutBase.java
> >    trunk/freenet/src/freenet/node/fcp/ClientRequest.java
> >    trunk/freenet/src/freenet/node/fcp/ModifyPersistentRequest.java
> >    trunk/freenet/src/freenet/node/fcp/RemovePersistentRequest.java
> > Log:
> > - added ack of ModifyPersistentRequest
> > - !!! added ack of RemovePersistentRequest (new message PersistentRequestRemoved)
> >
>
> [snip.]
>
> > Modified: trunk/freenet/src/freenet/node/fcp/ClientRequest.java
> > ===================================================================
> > --- trunk/freenet/src/freenet/node/fcp/ClientRequest.java     2007-01-31 19:33:47 UTC (rev 11647)
> > +++ trunk/freenet/src/freenet/node/fcp/ClientRequest.java     2007-02-02 11:42:40 UTC (rev 11648)
> > @@ -1,20 +1,13 @@
> >  package freenet.node.fcp;
> >
> > -import java.io.BufferedReader;
> > -import java.io.BufferedWriter;
> > -import java.io.IOException;
> > -import java.net.MalformedURLException;
> > +import java.io.*;
> > +import java.net.*;
> >
> > -import freenet.client.async.ClientRequester;
> > -import freenet.keys.FreenetURI;
> > -import freenet.support.Fields;
> > -import freenet.support.HexUtil;
> > -import freenet.support.Logger;
> > -import freenet.support.SimpleFieldSet;
> > -import freenet.support.api.Bucket;
> > -import freenet.support.io.FileBucket;
> > -import freenet.support.io.PaddedEphemerallyEncryptedBucket;
> > -import freenet.support.io.SerializableToFieldSetBucket;
> > +import freenet.client.async.*;
> > +import freenet.keys.*;
> > +import freenet.support.*;
> > +import freenet.support.api.*;
> > +import freenet.support.io.*;
> >
> [snip.]
>
>
> Is there any ground for doing things like that ? generalizing imports.
> I would say it's against the current coding standards used by the
> team^weclipse^wtoad ;)
>
> NextGen$
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>


From bbackde at googlemail.com  Sat Feb  3 15:21:15 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Sat, 3 Feb 2007 16:21:15 +0100
Subject: [freenet-dev] Fwd: r11648 : added ack of RemovePersistentRequest
	(new message PersistentRequestRemoved)
In-Reply-To: <d6149a790702030313g67d81ac2t3e976eb108b88d98@mail.gmail.com>
References: <a033de740702020432n367b5056k98464dea64245fb6@mail.gmail.com>
	<d6149a790702020507w2ead20dx40956a0e0839573d@mail.gmail.com>
	<d6149a790702020515i7e040d9as76c1273d766b5bba@mail.gmail.com>
	<a033de740702020525q181bd4a8q9635101eec19085c@mail.gmail.com>
	<d6149a790702020533l165179e7od7a6337004ce1ac7@mail.gmail.com>
	<a033de740702020539j62e6cf2eo5a72467b9c1090fd@mail.gmail.com>
	<d6149a790702020550m3cf488f2l664da1845a3082a5@mail.gmail.com>
	<d6149a790702030313g67d81ac2t3e976eb108b88d98@mail.gmail.com>
Message-ID: <d6149a790702030721s7a23fd80sf1af2bedec6ac3ff@mail.gmail.com>

Any more comments from devl? Toad, nextgens? I want to complete the
coding on this this weekend.

On 2/2/07, bbackde at googlemail.com <bbackde at googlemail.com> wrote:
> I answered to this in my mail from 2:15pm. It was only my idea to add
> a new message because _I_ thought it is easier for clients this way.
> It is no problem to change the GetFailed message and to provide a
> "reason=removed" to the client.
>
> We just need a decision and I will make the changes. Lets wait until
> some more people joined the discussion and voted for the one or the
> other :)
>
> PS: imho a GetFailed for removal of a request is'nt really correct,
> because nothing failed. See my comments about ambiguities in my before
> message. But to keep the overall number of messages low it could make
> sense to do it this way.
>
> On 2/2/07, Jerome Flesch <jflesch at nerim.net> wrote:
> > Ok, but why a new command ? Why not just another error code ? (xx =
> > cancelled by user ; yy = cancelled and removed by user)
> >
> > Anyway, as long as the GetFailed command is still sent, it won't break
> > my implementation. I'm just asking that by curiosity.
> >
> >
> > 2007/2/2, bbackde at googlemail.com <bbackde at googlemail.com>:
> > > imho cancel means both: request stopped for now, or request removed.
> > > Due to extendability of the protocol the parameters should be clear,
> > > so the answer for a RemovePersistentRequest should be something with
> > > the word "removed", same as requested. We should not start to
> > > introduce such ambiguities in the early state of 0.7.
> > >
> > > On 2/2/07, Jerome Flesch <jflesch at gmail.com> wrote:
> > > > Since there is no cancel command in FCP, I don't see what "cancelled
> > > > by user" can mean except that the user removed it ?
> > > >
> > > >
> > > > 2007/2/2, bbackde at googlemail.com <bbackde at googlemail.com>:
> > > > > Toad wrote:
> > > > > > GetFailed reason=Cancelled ? Admittedly this doesn't tell you for
> > > > > > certain that it has been removed... Hmm...
> > > > >
> > > > > So either a new message, or a new parameter reason=Removed would give
> > > > > a clear statement that the request was removed. Whereas the new
> > > > > message makes it easier for clients to handle the removed request,
> > > > > otherwise you have to do all inside the handler for GetFailed. My 2
> > > > > cents....
> > > > >
> > > > >
> > > > > On 2/2/07, bbackde at googlemail.com <bbackde at googlemail.com> wrote:
> > > > > > Did you follow the thread on tech yesterday? Thats why I added a new
> > > > > > msg....no problem to change it again.
> > > > > >
> > > > > > On 2/2/07, Jerome Flesch <jflesch at nerim.net> wrote:
> > > > > > > "new message PersistentRequestRemoved" ? I'm a little bit curious
> > > > > > > about this new message:
> > > > > > > Wasn't the node supposed to send a GetFailed ("cancelled by user")
> > > > > > > when a request was removed ? If yes, what is the goal of this new
> > > > > > > message ?
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>


From toad at amphibian.dyndns.org  Sat Feb  3 16:11:38 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 3 Feb 2007 16:11:38 +0000
Subject: [freenet-dev] AnotherIndex
Message-ID: <20070203161137.GA19739@amphibian.dyndns.org>

Should we include Another Index? It doesn't have site descriptions - but
Entry Point isn't categorised, and the others don't seem to be
maintained.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070203/2331f12e/attachment.pgp 

From toad at amphibian.dyndns.org  Sat Feb  3 17:33:13 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 3 Feb 2007 17:33:13 +0000
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
In-Reply-To: <1170379794.2232.477.camel@orchid.arb.net>
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
	<20070201184417.GB11653@amphibian.dyndns.org>
	<1170379794.2232.477.camel@orchid.arb.net>
Message-ID: <20070203173313.GA13731@amphibian.dyndns.org>

On Fri, Feb 02, 2007 at 01:29:54AM +0000, Bob Ham wrote:
> On Thu, 2007-02-01 at 18:44 +0000, Matthew Toseland wrote:
> > On Tue, Jan 30, 2007 at 09:39:05PM +0000, Bob Ham wrote:
> 
> > > It's clear to see that the configured limits are wholly ignored, and by
> > > significant percentages.  This is a major problem. 
> > 
> > On the contrary this is utterly irrelevant.
> 
> Not when my named gets killed.
> 
> > It is Sun's problem, not our
> > problem
> 
> Yes, but the limits are still being wholly ignored.  Which JVM would you
> suggest running fred with that will respect configured memory limits?
> 
> > What was the limit set to?
> 
> 196 MB
> 
> > What was the usage reported in the log file
> > (Memory in use: <blah> - only enabled with minor logging)?
> 
> Which log file; wrapper log or fred log?  The statistics for the "java
> memory allocated" and "java memory used" come straight off of the
> statistics page of the web interface.

And they exceeded 192MB? I have never seen this; what JVM?
> 
> > What was the
> > reported RSS?
> 
> What you see plotted.
> 
> > How much memory do you actually have (and swap)?
> 
> 512 MB physical, 478 MB swap
> 
> > > Even disregarding the
> > > database, fred's memory footprint is massive. 
> > 
> > Negative. I was running it with a maximum memory setting of 100MB, and
> > 54MB+ of that was used by the database. If the database had respected
> > the configured setting of 20MB, that would be a grand total of 64MB.
> > Which IMHO is very reasonable.
> 
> Here is a plot with a maximum setting of 64MB, taken a couple of weeks
> ago:
> 
>   http://teasel.6gnip.net/~rah/freenet-threads-2007-02-02-01:17:33.png
> 
> I've put it up here as well in case you have problems:
> 
>   http://pkl.net/~node/freenet-threads-2007-02-02-01:17:33.png

This is all taken from the web interface? Or are you using some external
tool? I assume "Count" is in megabytes?
> 
> As you can see, there are quite a few "wrapper rstrt" labels; meaning
> the wrapper has restarted the node.  The reason in these instances
> appear to be "OOM" messages.  I'll set my memory limit to 100MB and see
> what kind of stability results.
> 
> Bob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070203/0c61d704/attachment.pgp 

From toad at amphibian.dyndns.org  Sat Feb  3 17:34:04 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 3 Feb 2007 17:34:04 +0000
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
In-Reply-To: <1170384654.2232.500.camel@orchid.arb.net>
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
	<20070201184417.GB11653@amphibian.dyndns.org>
	<1170384654.2232.500.camel@orchid.arb.net>
Message-ID: <20070203173404.GB13731@amphibian.dyndns.org>

On Fri, Feb 02, 2007 at 02:50:53AM +0000, Bob Ham wrote:
> On Thu, 2007-02-01 at 18:44 +0000, Matthew Toseland wrote:
> 
> > that would be a grand total of 64MB.
> > Which IMHO is very reasonable.
> 
> As noted, that's pretty massive.  I don't think it's reasonable at all.
> Here's some stats for comparable daemons that I quickly plucked out of a
> "ps axu" on the box I'm running fred:
> 
> named	    3.17 MB
> imapd	    2.85 MB
> mysqld	   17.33 MB
> apache2	    7.82 MB
> postgres    6.47 MB
> fred	  120.82 MB

Squid can use tons. So can a well-configured high load apache.
> 
> Now, to me, fred is seeming a little out of place there.  Even if it was
> 64MB, that's still three times the size of mysql's footprint.

So what? The typical low end new PC has 512MB of RAM.
> 
> What, exactly, requires all this memory?
> 
> Bob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070203/5ae48c30/attachment.pgp 

From toad at amphibian.dyndns.org  Sat Feb  3 17:36:25 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 3 Feb 2007 17:36:25 +0000
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
In-Reply-To: <1170423056.2232.508.camel@orchid.arb.net>
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
	<20070201184417.GB11653@amphibian.dyndns.org>
	<1170384654.2232.500.camel@orchid.arb.net>
	<20070202084112.GI20976@freenetproject.org>
	<1170423056.2232.508.camel@orchid.arb.net>
Message-ID: <20070203173625.GC13731@amphibian.dyndns.org>

On Fri, Feb 02, 2007 at 01:30:55PM +0000, Bob Ham wrote:
> On Fri, 2007-02-02 at 08:41 +0000, NextGen$ wrote:
> > On the server we have ~100M of data and the memory footprint is ~25M. The
> > usual size for a datastore is 1G...
> 
> Let me requote some mails here; you've obviously missed the point:
> 
> On Thu, 2007-02-01 at 18:44 +0000, Matthew Toseland wrote: 
> > On Tue, Jan 30, 2007 at 09:39:05PM +0000, Bob Ham wrote:
> 
> > > Even disregarding the
> > > database, fred's memory footprint is massive. 
> 
> > Negative. I was running it with a maximum memory setting of 100MB, and
> > 54MB+ of that was used by the database. If the database had respected
> > the configured setting of 20MB, that would be a grand total of 64MB.
> > Which IMHO is very reasonable.
> 
> We're not talking about the memory requirements for the datastore.  They
> are, as you can in fact see above, explicitly excluded from the
> discussion.
> 
> I'll rephrase.  Even if fred (not including the datastore) uses 64 MB,
> that's still 8 times the memory footprint of apache.  This is excessive.

No, if you exclude the store, that would be 44MB. Which appears to be
reasonable to me. Obviously I'd prefer that it run in 10MB, but at some
point we have to settle for a reasonable level of usage and get on with
all the other things we need to do with Freenet.
> 
> Bob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070203/6668b7d3/attachment.pgp 

From toad at amphibian.dyndns.org  Sat Feb  3 17:38:10 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 3 Feb 2007 17:38:10 +0000
Subject: [freenet-dev] [freenet-support] Houston, We have a problem
In-Reply-To: <45C262CB.1050206@WhenGendarmeSleeps.org>
References: <45B3BBF8.40407@WhenGendarmeSleeps.org>
	<005001c73db4$06504540$0a7b0001@deepspace3>
	<20070127194312.GA8459@amphibian.dyndns.org>
	<45BC8B80.4030105@WhenGendarmeSleeps.org>
	<20070130144551.GA13856@amphibian.dyndns.org>
	<45BF9587.7090601@WhenGendarmeSleeps.org>
	<20070201184804.GD11653@amphibian.dyndns.org>
	<45C262CB.1050206@WhenGendarmeSleeps.org>
Message-ID: <20070203173810.GD13731@amphibian.dyndns.org>

On Thu, Feb 01, 2007 at 09:59:39PM +0000, Volodya wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> > Many of the simulations showed that throttling without backoff was
> > superior to throttling with backoff. Backoff is intended to prevent
> > throttling from throttling the whole network down to nothing because of
> > a small number of very slow nodes causing lots of failures when too many
> > requests are routed to them.
> 
> Thanks for the answer.
> 
> I still am lost as to why all my connections are backed off. I have noticed that shutting
> down everything but freenet makes the problem less. At least for a couple of hours after
> start up. My CPU usage is at 100% all the time currently, so i think that it might be
> because my node just doesn't comprehend what is being asked of it fast enough and
> therefore it backs the peers off. This in turn creates the state where all the peers are
> backed off and thus no place to rout through, something very weird begins to happen after
> that, i am still looking for any correlation between what i've been doing and the node's
> performance.

So the 100% cpu usage, which appears to be caused by lots of GCing,
causes the backoffs?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070203/c113dea9/attachment.pgp 

From toad at amphibian.dyndns.org  Sat Feb  3 17:40:03 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 3 Feb 2007 17:40:03 +0000
Subject: [freenet-dev] r11648 : added ack of RemovePersistentRequest
	(new message PersistentRequestRemoved)
In-Reply-To: <a033de740702020539j62e6cf2eo5a72467b9c1090fd@mail.gmail.com>
References: <a033de740702020432n367b5056k98464dea64245fb6@mail.gmail.com>
	<d6149a790702020507w2ead20dx40956a0e0839573d@mail.gmail.com>
	<d6149a790702020515i7e040d9as76c1273d766b5bba@mail.gmail.com>
	<a033de740702020525q181bd4a8q9635101eec19085c@mail.gmail.com>
	<d6149a790702020533l165179e7od7a6337004ce1ac7@mail.gmail.com>
	<a033de740702020539j62e6cf2eo5a72467b9c1090fd@mail.gmail.com>
Message-ID: <20070203174003.GE13731@amphibian.dyndns.org>

On Fri, Feb 02, 2007 at 02:39:53PM +0100, Jerome Flesch wrote:
> Ok, but why a new command ? Why not just another error code ? (xx =
> cancelled by user ; yy = cancelled and removed by user)
> 
> Anyway, as long as the GetFailed command is still sent, it won't break
> my implementation. I'm just asking that by curiosity.

Well, just because it's cancelled doesn't necessarily mean it's removed.
> 
> 
> 2007/2/2, bbackde at googlemail.com <bbackde at googlemail.com>:
> > imho cancel means both: request stopped for now, or request removed.
> > Due to extendability of the protocol the parameters should be clear,
> > so the answer for a RemovePersistentRequest should be something with
> > the word "removed", same as requested. We should not start to
> > introduce such ambiguities in the early state of 0.7.
> >
> > On 2/2/07, Jerome Flesch <jflesch at gmail.com> wrote:
> > > Since there is no cancel command in FCP, I don't see what "cancelled
> > > by user" can mean except that the user removed it ?
> > >
> > >
> > > 2007/2/2, bbackde at googlemail.com <bbackde at googlemail.com>:
> > > > Toad wrote:
> > > > > GetFailed reason=Cancelled ? Admittedly this doesn't tell you for
> > > > > certain that it has been removed... Hmm...
> > > >
> > > > So either a new message, or a new parameter reason=Removed would give
> > > > a clear statement that the request was removed. Whereas the new
> > > > message makes it easier for clients to handle the removed request,
> > > > otherwise you have to do all inside the handler for GetFailed. My 2
> > > > cents....
> > > >
> > > >
> > > > On 2/2/07, bbackde at googlemail.com <bbackde at googlemail.com> wrote:
> > > > > Did you follow the thread on tech yesterday? Thats why I added a new
> > > > > msg....no problem to change it again.
> > > > >
> > > > > On 2/2/07, Jerome Flesch <jflesch at nerim.net> wrote:
> > > > > > "new message PersistentRequestRemoved" ? I'm a little bit curious
> > > > > > about this new message:
> > > > > > Wasn't the node supposed to send a GetFailed ("cancelled by user")
> > > > > > when a request was removed ? If yes, what is the goal of this new
> > > > > > message ?
> > > > > >
> > > > >
> > > >
> > >
> >
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070203/52a7427e/attachment.pgp 

From toad at amphibian.dyndns.org  Sat Feb  3 17:40:40 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 3 Feb 2007 17:40:40 +0000
Subject: [freenet-dev] r11648 : added ack of RemovePersistentRequest
	(new message PersistentRequestRemoved)
In-Reply-To: <d6149a790702020550m3cf488f2l664da1845a3082a5@mail.gmail.com>
References: <a033de740702020432n367b5056k98464dea64245fb6@mail.gmail.com>
	<d6149a790702020507w2ead20dx40956a0e0839573d@mail.gmail.com>
	<d6149a790702020515i7e040d9as76c1273d766b5bba@mail.gmail.com>
	<a033de740702020525q181bd4a8q9635101eec19085c@mail.gmail.com>
	<d6149a790702020533l165179e7od7a6337004ce1ac7@mail.gmail.com>
	<a033de740702020539j62e6cf2eo5a72467b9c1090fd@mail.gmail.com>
	<d6149a790702020550m3cf488f2l664da1845a3082a5@mail.gmail.com>
Message-ID: <20070203174040.GF13731@amphibian.dyndns.org>

On Fri, Feb 02, 2007 at 02:50:51PM +0100, bbackde at googlemail.com wrote:
> I answered to this in my mail from 2:15pm. It was only my idea to add
> a new message because _I_ thought it is easier for clients this way.
> It is no problem to change the GetFailed message and to provide a
> "reason=removed" to the client.

Or a Removed=true|false flag.
> 
> We just need a decision and I will make the changes. Lets wait until
> some more people joined the discussion and voted for the one or the
> other :)
> 
> PS: imho a GetFailed for removal of a request is'nt really correct,
> because nothing failed. See my comments about ambiguities in my before
> message. But to keep the overall number of messages low it could make
> sense to do it this way.
> 
> On 2/2/07, Jerome Flesch <jflesch at nerim.net> wrote:
> > Ok, but why a new command ? Why not just another error code ? (xx =
> > cancelled by user ; yy = cancelled and removed by user)
> >
> > Anyway, as long as the GetFailed command is still sent, it won't break
> > my implementation. I'm just asking that by curiosity.
> >
> >
> > 2007/2/2, bbackde at googlemail.com <bbackde at googlemail.com>:
> > > imho cancel means both: request stopped for now, or request removed.
> > > Due to extendability of the protocol the parameters should be clear,
> > > so the answer for a RemovePersistentRequest should be something with
> > > the word "removed", same as requested. We should not start to
> > > introduce such ambiguities in the early state of 0.7.
> > >
> > > On 2/2/07, Jerome Flesch <jflesch at gmail.com> wrote:
> > > > Since there is no cancel command in FCP, I don't see what "cancelled
> > > > by user" can mean except that the user removed it ?
> > > >
> > > >
> > > > 2007/2/2, bbackde at googlemail.com <bbackde at googlemail.com>:
> > > > > Toad wrote:
> > > > > > GetFailed reason=Cancelled ? Admittedly this doesn't tell you for
> > > > > > certain that it has been removed... Hmm...
> > > > >
> > > > > So either a new message, or a new parameter reason=Removed would give
> > > > > a clear statement that the request was removed. Whereas the new
> > > > > message makes it easier for clients to handle the removed request,
> > > > > otherwise you have to do all inside the handler for GetFailed. My 2
> > > > > cents....
> > > > >
> > > > >
> > > > > On 2/2/07, bbackde at googlemail.com <bbackde at googlemail.com> wrote:
> > > > > > Did you follow the thread on tech yesterday? Thats why I added a new
> > > > > > msg....no problem to change it again.
> > > > > >
> > > > > > On 2/2/07, Jerome Flesch <jflesch at nerim.net> wrote:
> > > > > > > "new message PersistentRequestRemoved" ? I'm a little bit curious
> > > > > > > about this new message:
> > > > > > > Wasn't the node supposed to send a GetFailed ("cancelled by user")
> > > > > > > when a request was removed ? If yes, what is the goal of this new
> > > > > > > message ?
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070203/ad10b2da/attachment.pgp 

From bbackde at googlemail.com  Sat Feb  3 18:23:17 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Sat, 3 Feb 2007 19:23:17 +0100
Subject: [freenet-dev] r11648 : added ack of RemovePersistentRequest
	(new message PersistentRequestRemoved)
In-Reply-To: <20070203174040.GF13731@amphibian.dyndns.org>
References: <a033de740702020432n367b5056k98464dea64245fb6@mail.gmail.com>
	<d6149a790702020507w2ead20dx40956a0e0839573d@mail.gmail.com>
	<d6149a790702020515i7e040d9as76c1273d766b5bba@mail.gmail.com>
	<a033de740702020525q181bd4a8q9635101eec19085c@mail.gmail.com>
	<d6149a790702020533l165179e7od7a6337004ce1ac7@mail.gmail.com>
	<a033de740702020539j62e6cf2eo5a72467b9c1090fd@mail.gmail.com>
	<d6149a790702020550m3cf488f2l664da1845a3082a5@mail.gmail.com>
	<20070203174040.GF13731@amphibian.dyndns.org>
Message-ID: <d6149a790702031023y55a33922n266d87e7f9a7c3af@mail.gmail.com>

Yeah this is the 3rd possibility *g*. Who will decide this now?


On 2/3/07, Matthew Toseland <toad at amphibian.dyndns.org> wrote:
> On Fri, Feb 02, 2007 at 02:50:51PM +0100, bbackde at googlemail.com wrote:
> > I answered to this in my mail from 2:15pm. It was only my idea to add
> > a new message because _I_ thought it is easier for clients this way.
> > It is no problem to change the GetFailed message and to provide a
> > "reason=removed" to the client.
>
> Or a Removed=true|false flag.
> >
> > We just need a decision and I will make the changes. Lets wait until
> > some more people joined the discussion and voted for the one or the
> > other :)
> >
> > PS: imho a GetFailed for removal of a request is'nt really correct,
> > because nothing failed. See my comments about ambiguities in my before
> > message. But to keep the overall number of messages low it could make
> > sense to do it this way.
> >
> > On 2/2/07, Jerome Flesch <jflesch at nerim.net> wrote:
> > > Ok, but why a new command ? Why not just another error code ? (xx =
> > > cancelled by user ; yy = cancelled and removed by user)
> > >
> > > Anyway, as long as the GetFailed command is still sent, it won't break
> > > my implementation. I'm just asking that by curiosity.
> > >
> > >
> > > 2007/2/2, bbackde at googlemail.com <bbackde at googlemail.com>:
> > > > imho cancel means both: request stopped for now, or request removed.
> > > > Due to extendability of the protocol the parameters should be clear,
> > > > so the answer for a RemovePersistentRequest should be something with
> > > > the word "removed", same as requested. We should not start to
> > > > introduce such ambiguities in the early state of 0.7.
> > > >
> > > > On 2/2/07, Jerome Flesch <jflesch at gmail.com> wrote:
> > > > > Since there is no cancel command in FCP, I don't see what "cancelled
> > > > > by user" can mean except that the user removed it ?
> > > > >
> > > > >
> > > > > 2007/2/2, bbackde at googlemail.com <bbackde at googlemail.com>:
> > > > > > Toad wrote:
> > > > > > > GetFailed reason=Cancelled ? Admittedly this doesn't tell you for
> > > > > > > certain that it has been removed... Hmm...
> > > > > >
> > > > > > So either a new message, or a new parameter reason=Removed would give
> > > > > > a clear statement that the request was removed. Whereas the new
> > > > > > message makes it easier for clients to handle the removed request,
> > > > > > otherwise you have to do all inside the handler for GetFailed. My 2
> > > > > > cents....
> > > > > >
> > > > > >
> > > > > > On 2/2/07, bbackde at googlemail.com <bbackde at googlemail.com> wrote:
> > > > > > > Did you follow the thread on tech yesterday? Thats why I added a new
> > > > > > > msg....no problem to change it again.
> > > > > > >
> > > > > > > On 2/2/07, Jerome Flesch <jflesch at nerim.net> wrote:
> > > > > > > > "new message PersistentRequestRemoved" ? I'm a little bit curious
> > > > > > > > about this new message:
> > > > > > > > Wasn't the node supposed to send a GetFailed ("cancelled by user")
> > > > > > > > when a request was removed ? If yes, what is the goal of this new
> > > > > > > > message ?
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > _______________________________________________
> > Devl mailing list
> > Devl at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> >
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFFxMkYA9rUluQ9pFARAqVuAKCR8vmy/KBzx6PiyKsZqrRSl4iRTACgq+6W
> EM1XVus05bRXge5S/2PJo5g=
> =ZMbE
> -----END PGP SIGNATURE-----
>
>
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>
>


From nextgens at freenetproject.org  Sat Feb  3 21:41:54 2007
From: nextgens at freenetproject.org (NextGen$)
Date: Sat, 3 Feb 2007 21:41:54 +0000
Subject: [freenet-dev] r11648 : added ack of RemovePersistentRequest
	(new message PersistentRequestRemoved)
In-Reply-To: <d6149a790702031023y55a33922n266d87e7f9a7c3af@mail.gmail.com>
References: <a033de740702020432n367b5056k98464dea64245fb6@mail.gmail.com>
	<d6149a790702020507w2ead20dx40956a0e0839573d@mail.gmail.com>
	<d6149a790702020515i7e040d9as76c1273d766b5bba@mail.gmail.com>
	<a033de740702020525q181bd4a8q9635101eec19085c@mail.gmail.com>
	<d6149a790702020533l165179e7od7a6337004ce1ac7@mail.gmail.com>
	<a033de740702020539j62e6cf2eo5a72467b9c1090fd@mail.gmail.com>
	<d6149a790702020550m3cf488f2l664da1845a3082a5@mail.gmail.com>
	<20070203174040.GF13731@amphibian.dyndns.org>
	<d6149a790702031023y55a33922n266d87e7f9a7c3af@mail.gmail.com>
Message-ID: <20070203214154.GN20976@freenetproject.org>

* bbackde at googlemail.com <bbackde at googlemail.com> [2007-02-03 19:23:17]:

> Yeah this is the 3rd possibility *g*. Who will decide this now?
> 
> 
> On 2/3/07, Matthew Toseland <toad at amphibian.dyndns.org> wrote:
> > On Fri, Feb 02, 2007 at 02:50:51PM +0100, bbackde at googlemail.com wrote:
> > > I answered to this in my mail from 2:15pm. It was only my idea to add
> > > a new message because _I_ thought it is easier for clients this way.
> > > It is no problem to change the GetFailed message and to provide a
> > > "reason=removed" to the client.
> >
> > Or a Removed=true|false flag.

I would be in favor of the flag solution.


From alejandro at mosteo.com  Sat Feb  3 22:22:00 2007
From: alejandro at mosteo.com (Jano)
Date: Sat, 03 Feb 2007 23:22:00 +0100
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
	<epqrm4$1va$1@sea.gmane.org>
	<20070201184529.GC11653@amphibian.dyndns.org>
	<eptrne$c7r$1@sea.gmane.org>
	<20070202193241.GA29915@amphibian.dyndns.org>
Message-ID: <eq31u5$cag$1@sea.gmane.org>

Matthew Toseland wrote:

> On Fri, Feb 02, 2007 at 12:05:20AM +0100, Jano wrote:
>> Matthew Toseland wrote:
>> 
>> > On Wed, Jan 31, 2007 at 08:46:14PM +0100, Jano wrote:
>> >> Bob Ham wrote:
>> >> 
>> >> > I think there are deeper issues to deal with.  Even disregarding the
>> >> > database, fred's memory footprint is massive.  For the functionality
>> >> > that it provides, it seems to me to be excessive.  I can't
>> >> > understand why fred would need anything in excess of a few 10s of
>> >> > megabytes. Where is this memory going?
>> >> 
>> >> I agree here. BDB seems a mature database; switching databases without
>> >> really isolating the culprit seems premature.
>> > 
>> > Bob's "deeper issues" are illusory; local configuration problems.
>> > 
>> > BDB uses 54MB when I have told it to use 20MB. That is the issue here.
>> > BDB is responsible for the majority of Fred's memory usage as of now as
>> > far as I can see.
>> 
>> But why? Can we be sure that the fault is not in some kind of misuse of
>> BDB instead of being intrinsic to BDB?
> 
> How to identify such misuse?

Memory profiling, I'd say. Though I have never done it with java.



From rah at bash.sh  Sat Feb  3 22:52:33 2007
From: rah at bash.sh (Bob Ham)
Date: Sat, 03 Feb 2007 22:52:33 +0000
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
In-Reply-To: <20070203173313.GA13731@amphibian.dyndns.org>
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
	<20070201184417.GB11653@amphibian.dyndns.org>
	<1170379794.2232.477.camel@orchid.arb.net>
	<20070203173313.GA13731@amphibian.dyndns.org>
Message-ID: <1170543153.2232.540.camel@orchid.arb.net>

On Sat, 2007-02-03 at 17:33 +0000, Matthew Toseland wrote:
> On Fri, Feb 02, 2007 at 01:29:54AM +0000, Bob Ham wrote:

> > Yes, but the limits are still being wholly ignored.  Which JVM would you
> > suggest running fred with that will respect configured memory limits?
> > 
> > > What was the limit set to?
> > 
> > 196 MB
> > 
> > > What was the usage reported in the log file
> > > (Memory in use: <blah> - only enabled with minor logging)?
> > 
> > Which log file; wrapper log or fred log?  The statistics for the "java
> > memory allocated" and "java memory used" come straight off of the
> > statistics page of the web interface.
> 
> And they exceeded 192MB? I have never seen this; what JVM?

The statistics page didn't report greater than 196MB but greater than
196MB was used.


> This is all taken from the web interface? Or are you using some external
> tool? I assume "Count" is in megabytes?

Different places; the java memory allocation and usage, and the store
usage are all taken from the statistics page.  I've attached the script
used to collect the stats.  I've also put up the data collection scripts
and the plotting script here:

  http://teasel.6gnip.net/~rah/freenet-threads-2007-01-21-23-37.tar.bz2

There shouldn't be any problems with access now.

Bob

-- 
Bob Ham <rah at bash.sh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freenet-threads
Type: application/x-shellscript
Size: 2011 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070203/dce469c1/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070203/dce469c1/attachment.pgp 

From bbackde at googlemail.com  Sat Feb  3 22:52:37 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Sat, 3 Feb 2007 23:52:37 +0100
Subject: [freenet-dev] r11648 : added ack of RemovePersistentRequest
	(new message PersistentRequestRemoved)
In-Reply-To: <20070203214154.GN20976@freenetproject.org>
References: <a033de740702020432n367b5056k98464dea64245fb6@mail.gmail.com>
	<d6149a790702020507w2ead20dx40956a0e0839573d@mail.gmail.com>
	<d6149a790702020515i7e040d9as76c1273d766b5bba@mail.gmail.com>
	<a033de740702020525q181bd4a8q9635101eec19085c@mail.gmail.com>
	<d6149a790702020533l165179e7od7a6337004ce1ac7@mail.gmail.com>
	<a033de740702020539j62e6cf2eo5a72467b9c1090fd@mail.gmail.com>
	<d6149a790702020550m3cf488f2l664da1845a3082a5@mail.gmail.com>
	<20070203174040.GF13731@amphibian.dyndns.org>
	<d6149a790702031023y55a33922n266d87e7f9a7c3af@mail.gmail.com>
	<20070203214154.GN20976@freenetproject.org>
Message-ID: <d6149a790702031452h36914e62ucd5999a02ff16c81@mail.gmail.com>

This new flag introduces more ambiguousness for clients.

I think it should either be a GetFailed with reason=Removed (I don't
like this one because the get didn't actually fail, but this depends
on the definition of 'fail'), or a new message
PersistentRequestRemoved. With a new message existing clients also
have no problem because they must ignore the message and the behavior
of fcp2 doesn't change at all for them.

On 2/3/07, NextGen$ <nextgens at freenetproject.org> wrote:
> * bbackde at googlemail.com <bbackde at googlemail.com> [2007-02-03 19:23:17]:
>
> > Yeah this is the 3rd possibility *g*. Who will decide this now?
> >
> >
> > On 2/3/07, Matthew Toseland <toad at amphibian.dyndns.org> wrote:
> > > On Fri, Feb 02, 2007 at 02:50:51PM +0100, bbackde at googlemail.com wrote:
> > > > I answered to this in my mail from 2:15pm. It was only my idea to add
> > > > a new message because _I_ thought it is easier for clients this way.
> > > > It is no problem to change the GetFailed message and to provide a
> > > > "reason=removed" to the client.
> > >
> > > Or a Removed=true|false flag.
>
> I would be in favor of the flag solution.
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>


From dbkr at freenetproject.org  Sun Feb  4 11:45:38 2007
From: dbkr at freenetproject.org (Dave Baker)
Date: Sun, 4 Feb 2007 11:45:38 +0000
Subject: [freenet-dev] Emu again
Message-ID: <200702041145.38992.dbkr@freenetproject.org>

Hi,

Emu's HTTPS seems to be down again. Same problem as last time - web server is 
fine on port 80 but port 443 is closed.


Dave


From nextgens at freenetproject.org  Sun Feb  4 12:12:03 2007
From: nextgens at freenetproject.org (NextGen$)
Date: Sun, 4 Feb 2007 12:12:03 +0000
Subject: [freenet-dev] Emu again
In-Reply-To: <200702041145.38992.dbkr@freenetproject.org>
References: <200702041145.38992.dbkr@freenetproject.org>
Message-ID: <20070204121203.GO20976@freenetproject.org>

* Dave Baker <dbkr at freenetproject.org> [2007-02-04 11:45:38]:

> Hi,
> 
> Emu's HTTPS seems to be down again. Same problem as last time - web server is 
> fine on port 80 but port 443 is closed.
> 
> 
> Dave

I've restarted it ... No time to investigate now :$


From alejandro at mosteo.com  Sun Feb  4 17:29:04 2007
From: alejandro at mosteo.com (Jano)
Date: Sun, 04 Feb 2007 18:29:04 +0100
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
	<epqrm4$1va$1@sea.gmane.org>
	<20070201184529.GC11653@amphibian.dyndns.org>
	<eptrne$c7r$1@sea.gmane.org>
	<20070202193241.GA29915@amphibian.dyndns.org>
	<eq31u5$cag$1@sea.gmane.org>
Message-ID: <eq555o$ipq$1@sea.gmane.org>

Jano wrote:

> Matthew Toseland wrote:

>> How to identify such misuse?
> 
> Memory profiling, I'd say. Though I have never done it with java.

I've started toying with jmap/jhat after upgrading to java6. This node is a
linux one. Here's a heap dump at node start:

Top 12 Class histogram:

Class                                   Instance Count  Total Size
class [B                                        379353  18473891
class [I                                        315048  12805068
class [C                                        56549   11786932
class [[B                                       6137    3190228
class [Lcom.sleepycat.je.tree.Node;             6135    3190200
class [Ljava.util.HashMap$Entry;                33158   2950160
class [J                                        2037    2091504
class freenet.client.async.SingleBlockInserter  30965   1795970
class [[I                                       20812   1415092
class java.util.HashMap$Entry                   66778   1068448
class java.util.HashMap                         33144   1060608
class java.lang.String                          56831   909296

Top 10 class instance counts (excluding platform):

31204 instances of class freenet.client.FailureCodeTracker
30965 instances of class freenet.client.async.SingleBlockInserter
20604 instances of class freenet.support.io.ReadOnlyFileSliceBucket
16213 instances of class com.sleepycat.je.tree.LN
10413 instances of class freenet.support.io.FileBucket
10403 instances of class freenet.crypt.ciphers.Rijndael
10367 instances of class freenet.support.io.PaddedEphemerallyEncryptedBucket
10355 instances of class freenet.support.io.DelayedFreeBucket
9947 instances of class freenet.keys.FreenetURI
6157 instances of class com.sleepycat.je.latch.Java5SharedLatchImpl 

Total instances pending finalization: 0

The strange short names are, so it seems, "platform classes", which I don't
know what means (apparently java.* and javax.* classes, but these strange
names go away too).

And here's the heap dump at the moment of node death (node configured with
128m max mem):

Class                                   Instance Count  Total Size
class [B                                        771728  34242096
class [C                                        113887  19922068
class [I                                        439531  18758092
class [[B                                       9527    4953028
class [Lcom.sleepycat.je.tree.Node;             9525    4953000
class [Ljava.util.HashMap$Entry;                50089   4642912
class [J                                        3030    3111184
class com.sleepycat.je.tree.LN                  173461  2254993
class [[I                                       28586   1943756
class java.lang.String                          115987  1855792
class freenet.client.async.SingleBlockInserter  31080   1802640
class java.util.HashMap$Entry                   108956  1743296

173461 instances of class com.sleepycat.je.tree.LN
31413 instances of class freenet.client.FailureCodeTracker
31080 instances of class freenet.client.async.SingleBlockInserter
20613 instances of class freenet.support.io.ReadOnlyFileSliceBucket
17564 instances of class freenet.keys.FreenetURI
14291 instances of class freenet.crypt.ciphers.Rijndael
14246 instances of class freenet.support.io.PaddedEphemerallyEncryptedBucket
11793 instances of class freenet.support.io.FileBucket
10358 instances of class freenet.support.io.DelayedFreeBucket
10065 instances of class freenet.support.LRUQueue$QItem

Total instances pending finalization: 0

>From these numbers it seems that sleepycat is notably leaking memory (the LN
objects). Someone familiar with DBD could perhaps pinpoint seeing this if
this is a bug in BDB or a mismanagement in freenet (unclosed whatevers?)

I'm running now with 256m to see if the differences are even more apparent.
I'll send later some graphs obtained with jconsole that pretty much back
that there's leaking going on (i.e. if we find the leak, freenet will run
comfortably with 128m or less).



From bbackde at googlemail.com  Sun Feb  4 20:38:47 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Sun, 4 Feb 2007 21:38:47 +0100
Subject: [freenet-dev] New FCP2 messages
Message-ID: <d6149a790702041238l7f1a0eadhb86022332aac18d1@mail.gmail.com>

If FCP2 want to provide clients an easy access, then the following
changes are needed:

Answer to a ModifyPersistentRequest should not be a new
Persistent<blah> with all values, but a ModifiedPersistentRequest
message with only the changed values. Otherwise the client has to do
all syncing and searching for really changed values.

Answer to a RemovePersistentRequest should be a
RemovedPersistentRequest, this is a clear answer to the request and
clients don't need to do much investigation here.

Its easy to implement this in the node and I offer to do the changes
and to document them in the wiki. Existing clients are not affected.

Please discuss this, and with your answer provide an explanation WHY
you recommendation is better then the given ones, thanks.

rgds, bback.


From jflesch at nerim.net  Sun Feb  4 21:32:43 2007
From: jflesch at nerim.net (Jerome Flesch)
Date: Sun, 4 Feb 2007 22:32:43 +0100
Subject: [freenet-dev] New FCP2 messages
In-Reply-To: <d6149a790702041238l7f1a0eadhb86022332aac18d1@mail.gmail.com>
References: <d6149a790702041238l7f1a0eadhb86022332aac18d1@mail.gmail.com>
Message-ID: <a033de740702041332o5405d578hc609fb0d0e918ee1@mail.gmail.com>

> If FCP2 want to provide clients an easy access, then the following
> changes are needed:
>
> Answer to a ModifyPersistentRequest should not be a new
> Persistent<blah> with all values, but a ModifiedPersistentRequest
> message with only the changed values. Otherwise the client has to do
> all syncing and searching for really changed values.
>
> Answer to a RemovePersistentRequest should be a
> RemovedPersistentRequest, this is a clear answer to the request and
> clients don't need to do much investigation here.
>
Does that mean you will remove the GetFailed message ?
Don't forget that currently, this message is broadcasted to all the
FCP clients with WatchGlobal set to 1 (Thaw, for example).


> Its easy to implement this in the node and I offer to do the changes
> and to document them in the wiki. Existing clients are not affected.
>
> Please discuss this, and with your answer provide an explanation WHY
> you recommendation is better then the given ones, thanks.
>
> rgds, bback.
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>


From bbackde at googlemail.com  Sun Feb  4 21:46:15 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Sun, 4 Feb 2007 22:46:15 +0100
Subject: [freenet-dev] New FCP2 messages
In-Reply-To: <a033de740702041332o5405d578hc609fb0d0e918ee1@mail.gmail.com>
References: <d6149a790702041238l7f1a0eadhb86022332aac18d1@mail.gmail.com>
	<a033de740702041332o5405d578hc609fb0d0e918ee1@mail.gmail.com>
Message-ID: <d6149a790702041346g3b6d35e0x456ab426a9891879@mail.gmail.com>

On 2/4/07, Jerome Flesch <jflesch at nerim.net> wrote:
> > If FCP2 want to provide clients an easy access, then the following
> > changes are needed:
> >
> > Answer to a ModifyPersistentRequest should not be a new
> > Persistent<blah> with all values, but a ModifiedPersistentRequest
> > message with only the changed values. Otherwise the client has to do
> > all syncing and searching for really changed values.
> >
> > Answer to a RemovePersistentRequest should be a
> > RemovedPersistentRequest, this is a clear answer to the request and
> > clients don't need to do much investigation here.
> >
> Does that mean you will remove the GetFailed message ?
> Don't forget that currently, this message is broadcasted to all the
> FCP clients with WatchGlobal set to 1 (Thaw, for example).
>
Of course GetFailed will not be removed, but it would not be sent on a
RemovePersistentRequest. And it was never sent before after a
RemovePersistentRequest (was not implemented in node), so nothing
would change.
>
> > Its easy to implement this in the node and I offer to do the changes
> > and to document them in the wiki. Existing clients are not affected.
> >
> > Please discuss this, and with your answer provide an explanation WHY
> > you recommendation is better then the given ones, thanks.
> >
> > rgds, bback.
> > _______________________________________________
> > Devl mailing list
> > Devl at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> >
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>


From jflesch at nerim.net  Sun Feb  4 22:09:13 2007
From: jflesch at nerim.net (Jerome Flesch)
Date: Sun, 4 Feb 2007 23:09:13 +0100
Subject: [freenet-dev] New FCP2 messages
In-Reply-To: <d6149a790702041346g3b6d35e0x456ab426a9891879@mail.gmail.com>
References: <d6149a790702041238l7f1a0eadhb86022332aac18d1@mail.gmail.com>
	<a033de740702041332o5405d578hc609fb0d0e918ee1@mail.gmail.com>
	<d6149a790702041346g3b6d35e0x456ab426a9891879@mail.gmail.com>
Message-ID: <a033de740702041409v6d5acce3t191bce9163fa0ee@mail.gmail.com>

2007/2/4, bbackde at googlemail.com <bbackde at googlemail.com>:
> On 2/4/07, Jerome Flesch <jflesch at nerim.net> wrote:
> > > If FCP2 want to provide clients an easy access, then the following
> > > changes are needed:
> > >
> > > Answer to a ModifyPersistentRequest should not be a new
> > > Persistent<blah> with all values, but a ModifiedPersistentRequest
> > > message with only the changed values. Otherwise the client has to do
> > > all syncing and searching for really changed values.
> > >
> > > Answer to a RemovePersistentRequest should be a
> > > RemovedPersistentRequest, this is a clear answer to the request and
> > > clients don't need to do much investigation here.
> > >
> > Does that mean you will remove the GetFailed message ?
> > Don't forget that currently, this message is broadcasted to all the
> > FCP clients with WatchGlobal set to 1 (Thaw, for example).
> >
> Of course GetFailed will not be removed, but it would not be sent on a
> RemovePersistentRequest. And it was never sent before after a
> RemovePersistentRequest (was not implemented in node), so nothing
> would change.
>
Then, that's ok for me.


> >
> > > Its easy to implement this in the node and I offer to do the changes
> > > and to document them in the wiki. Existing clients are not affected.
> > >
> > > Please discuss this, and with your answer provide an explanation WHY
> > > you recommendation is better then the given ones, thanks.
> > >
> > > rgds, bback.
> > > _______________________________________________
> > > Devl mailing list
> > > Devl at freenetproject.org
> > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> > >
> > _______________________________________________
> > Devl mailing list
> > Devl at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> >
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>


From toad at amphibian.dyndns.org  Mon Feb  5 12:12:26 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 5 Feb 2007 12:12:26 +0000
Subject: [freenet-dev] Immediate priorities
Message-ID: <20070205121226.GA31958@amphibian.dyndns.org>

At the moment the priority is the CPU/memory performance issues which
have been reported fairly widely. A big part of this comes down to
object churn, and I have made significant progress on this on Saturday.
But I seem to have broken something fairly major in the process. I will
therefore fix this, and do some more work on this. Testers would be
helpful; I know there are issues with the build I committed on Saturday
(1ms ping time, backoff, ...), but even there I'd like to know more
about how it fails.

Once this is sorted I will put out a new build to the auto-update (there
are reports of this not working, but I think that's just due to 1010 USK
breakage). bback seems to have taken on the FCP changes that need to be
done, so that leaves me free to get on with the remaining major issues:
- Aftermath of 1010: Auto-update keys, ARK keys etc will need to be
  changed; if I have any peers still on 1009, I need to start up a node
  and insert the auto-update to them (although the network may be too
  fragmented for this to be of any use, it's worth trying).
- Backoff / load limiting. Once the current issues have been sorted
  hopefully this will improve.
- Location clustering / routing issues. May very well be caused by
  lack of opennet, but further investigation is required.
- Opennet.

PS Some people have complained about my absence during most of January.
A big chunk of this was (unpaid) sick-leave; the rest was (also unpaid)
holiday time. I'm back now!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070205/65396cbd/attachment.pgp 

From toad at amphibian.dyndns.org  Mon Feb  5 12:18:29 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 5 Feb 2007 12:18:29 +0000
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
In-Reply-To: <eq555o$ipq$1@sea.gmane.org>
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
	<epqrm4$1va$1@sea.gmane.org>
	<20070201184529.GC11653@amphibian.dyndns.org>
	<eptrne$c7r$1@sea.gmane.org>
	<20070202193241.GA29915@amphibian.dyndns.org>
	<eq31u5$cag$1@sea.gmane.org> <eq555o$ipq$1@sea.gmane.org>
Message-ID: <20070205121829.GA23223@amphibian.dyndns.org>

[B [I [C [[B almost certainly mean byte[], int[], char[], byte[][].

I didn't get jmap to work (maybe because I was using java 5), but I did
do some invasive profiling (with stack traces), and used that to
identify and eliminate some high-churn objects; if the current problem is
that too much garbage collection is occurring (causing 100% cpu usage),
this is most likely caused by too many objects being allocated per second.

Anyway my original trace is here:
http://amphibian.dyndns.org/java.hprof.multi-day.no-logging.1011.txt

This is produced by options:
wrapper.java.additional.3=-Xloggc:freenet.loggc
wrapper.java.additional.4=-Xrunhprof:heap=all,format=a,depth=12,lineno=y,doe=y,
gc_okay=y

(Set logging to NORMAL if you haven't already; heavy logging with
profiling makes the node break)

On Sun, Feb 04, 2007 at 06:29:04PM +0100, Jano wrote:
> Jano wrote:
> 
> > Matthew Toseland wrote:
> 
> >> How to identify such misuse?
> > 
> > Memory profiling, I'd say. Though I have never done it with java.
> 
> I've started toying with jmap/jhat after upgrading to java6. This node is a
> linux one. Here's a heap dump at node start:
> 
> Top 12 Class histogram:
> 
> Class                                   Instance Count  Total Size
> class [B                                        379353  18473891
> class [I                                        315048  12805068
> class [C                                        56549   11786932
> class [[B                                       6137    3190228
> class [Lcom.sleepycat.je.tree.Node;             6135    3190200
> class [Ljava.util.HashMap$Entry;                33158   2950160
> class [J                                        2037    2091504
> class freenet.client.async.SingleBlockInserter  30965   1795970
> class [[I                                       20812   1415092
> class java.util.HashMap$Entry                   66778   1068448
> class java.util.HashMap                         33144   1060608
> class java.lang.String                          56831   909296
> 
> Top 10 class instance counts (excluding platform):
> 
> 31204 instances of class freenet.client.FailureCodeTracker
> 30965 instances of class freenet.client.async.SingleBlockInserter
> 20604 instances of class freenet.support.io.ReadOnlyFileSliceBucket
> 16213 instances of class com.sleepycat.je.tree.LN
> 10413 instances of class freenet.support.io.FileBucket
> 10403 instances of class freenet.crypt.ciphers.Rijndael
> 10367 instances of class freenet.support.io.PaddedEphemerallyEncryptedBucket
> 10355 instances of class freenet.support.io.DelayedFreeBucket
> 9947 instances of class freenet.keys.FreenetURI
> 6157 instances of class com.sleepycat.je.latch.Java5SharedLatchImpl 
> 
> Total instances pending finalization: 0
> 
> The strange short names are, so it seems, "platform classes", which I don't
> know what means (apparently java.* and javax.* classes, but these strange
> names go away too).
> 
> And here's the heap dump at the moment of node death (node configured with
> 128m max mem):
> 
> Class                                   Instance Count  Total Size
> class [B                                        771728  34242096
> class [C                                        113887  19922068
> class [I                                        439531  18758092
> class [[B                                       9527    4953028
> class [Lcom.sleepycat.je.tree.Node;             9525    4953000
> class [Ljava.util.HashMap$Entry;                50089   4642912
> class [J                                        3030    3111184
> class com.sleepycat.je.tree.LN                  173461  2254993
> class [[I                                       28586   1943756
> class java.lang.String                          115987  1855792
> class freenet.client.async.SingleBlockInserter  31080   1802640
> class java.util.HashMap$Entry                   108956  1743296
> 
> 173461 instances of class com.sleepycat.je.tree.LN
> 31413 instances of class freenet.client.FailureCodeTracker
> 31080 instances of class freenet.client.async.SingleBlockInserter
> 20613 instances of class freenet.support.io.ReadOnlyFileSliceBucket
> 17564 instances of class freenet.keys.FreenetURI
> 14291 instances of class freenet.crypt.ciphers.Rijndael
> 14246 instances of class freenet.support.io.PaddedEphemerallyEncryptedBucket
> 11793 instances of class freenet.support.io.FileBucket
> 10358 instances of class freenet.support.io.DelayedFreeBucket
> 10065 instances of class freenet.support.LRUQueue$QItem
> 
> Total instances pending finalization: 0
> 
> From these numbers it seems that sleepycat is notably leaking memory (the LN
> objects). Someone familiar with DBD could perhaps pinpoint seeing this if
> this is a bug in BDB or a mismanagement in freenet (unclosed whatevers?)
> 
> I'm running now with 256m to see if the differences are even more apparent.
> I'll send later some graphs obtained with jconsole that pretty much back
> that there's leaking going on (i.e. if we find the leak, freenet will run
> comfortably with 128m or less).
> 
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070205/5c44ae68/attachment.pgp 

From toad at amphibian.dyndns.org  Mon Feb  5 12:20:00 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 5 Feb 2007 12:20:00 +0000
Subject: [freenet-dev] r11648 : added ack of RemovePersistentRequest
	(new message PersistentRequestRemoved)
In-Reply-To: <d6149a790702031452h36914e62ucd5999a02ff16c81@mail.gmail.com>
References: <d6149a790702020507w2ead20dx40956a0e0839573d@mail.gmail.com>
	<d6149a790702020515i7e040d9as76c1273d766b5bba@mail.gmail.com>
	<a033de740702020525q181bd4a8q9635101eec19085c@mail.gmail.com>
	<d6149a790702020533l165179e7od7a6337004ce1ac7@mail.gmail.com>
	<a033de740702020539j62e6cf2eo5a72467b9c1090fd@mail.gmail.com>
	<d6149a790702020550m3cf488f2l664da1845a3082a5@mail.gmail.com>
	<20070203174040.GF13731@amphibian.dyndns.org>
	<d6149a790702031023y55a33922n266d87e7f9a7c3af@mail.gmail.com>
	<20070203214154.GN20976@freenetproject.org>
	<d6149a790702031452h36914e62ucd5999a02ff16c81@mail.gmail.com>
Message-ID: <20070205122000.GB23223@amphibian.dyndns.org>

On Sat, Feb 03, 2007 at 11:52:37PM +0100, bbackde at googlemail.com wrote:
> This new flag introduces more ambiguousness for clients.
> 
> I think it should either be a GetFailed with reason=Removed (I don't
> like this one because the get didn't actually fail, but this depends
> on the definition of 'fail'), or a new message
> PersistentRequestRemoved. With a new message existing clients also
> have no problem because they must ignore the message and the behavior
> of fcp2 doesn't change at all for them.

Well internally it's because it was cancelled; that's the failure code.
There is just a side-effect which is that it is no longer on the queue.
So I think a flag would be good. Existing clients should ignore any
fields within a message that they don't recognize.
> 
> On 2/3/07, NextGen$ <nextgens at freenetproject.org> wrote:
> > * bbackde at googlemail.com <bbackde at googlemail.com> [2007-02-03 19:23:17]:
> >
> > > Yeah this is the 3rd possibility *g*. Who will decide this now?
> > >
> > >
> > > On 2/3/07, Matthew Toseland <toad at amphibian.dyndns.org> wrote:
> > > > On Fri, Feb 02, 2007 at 02:50:51PM +0100, bbackde at googlemail.com wrote:
> > > > > I answered to this in my mail from 2:15pm. It was only my idea to add
> > > > > a new message because _I_ thought it is easier for clients this way.
> > > > > It is no problem to change the GetFailed message and to provide a
> > > > > "reason=removed" to the client.
> > > >
> > > > Or a Removed=true|false flag.
> >
> > I would be in favor of the flag solution.
> > _______________________________________________
> > Devl mailing list
> > Devl at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> >
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070205/e946aa00/attachment.pgp 

From toad at amphibian.dyndns.org  Mon Feb  5 12:21:05 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 5 Feb 2007 12:21:05 +0000
Subject: [freenet-dev] New FCP2 messages
In-Reply-To: <d6149a790702041238l7f1a0eadhb86022332aac18d1@mail.gmail.com>
References: <d6149a790702041238l7f1a0eadhb86022332aac18d1@mail.gmail.com>
Message-ID: <20070205122105.GC23223@amphibian.dyndns.org>

On Sun, Feb 04, 2007 at 09:38:47PM +0100, bbackde at googlemail.com wrote:
> If FCP2 want to provide clients an easy access, then the following
> changes are needed:
> 
> Answer to a ModifyPersistentRequest should not be a new
> Persistent<blah> with all values, but a ModifiedPersistentRequest
> message with only the changed values. Otherwise the client has to do
> all syncing and searching for really changed values.

I don't get this. Why not just have a Persistent<blah> ? Surely less
messages is the simpler solution?
> 
> Answer to a RemovePersistentRequest should be a
> RemovedPersistentRequest, this is a clear answer to the request and
> clients don't need to do much investigation here.

Or a GetFailed with Removed=true.
> 
> Its easy to implement this in the node and I offer to do the changes
> and to document them in the wiki. Existing clients are not affected.

By all means.
> 
> Please discuss this, and with your answer provide an explanation WHY
> you recommendation is better then the given ones, thanks.
> 
> rgds, bback.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070205/249ebae3/attachment.pgp 

From toad at amphibian.dyndns.org  Mon Feb  5 12:22:50 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 5 Feb 2007 12:22:50 +0000
Subject: [freenet-dev] New FCP2 messages
In-Reply-To: <d6149a790702041346g3b6d35e0x456ab426a9891879@mail.gmail.com>
References: <d6149a790702041238l7f1a0eadhb86022332aac18d1@mail.gmail.com>
	<a033de740702041332o5405d578hc609fb0d0e918ee1@mail.gmail.com>
	<d6149a790702041346g3b6d35e0x456ab426a9891879@mail.gmail.com>
Message-ID: <20070205122250.GD23223@amphibian.dyndns.org>

On Sun, Feb 04, 2007 at 10:46:15PM +0100, bbackde at googlemail.com wrote:
> On 2/4/07, Jerome Flesch <jflesch at nerim.net> wrote:
> > > If FCP2 want to provide clients an easy access, then the following
> > > changes are needed:
> > >
> > > Answer to a ModifyPersistentRequest should not be a new
> > > Persistent<blah> with all values, but a ModifiedPersistentRequest
> > > message with only the changed values. Otherwise the client has to do
> > > all syncing and searching for really changed values.
> > >
> > > Answer to a RemovePersistentRequest should be a
> > > RemovedPersistentRequest, this is a clear answer to the request and
> > > clients don't need to do much investigation here.
> > >
> > Does that mean you will remove the GetFailed message ?
> > Don't forget that currently, this message is broadcasted to all the
> > FCP clients with WatchGlobal set to 1 (Thaw, for example).
> >
> Of course GetFailed will not be removed, but it would not be sent on a
> RemovePersistentRequest. And it was never sent before after a
> RemovePersistentRequest (was not implemented in node), so nothing
> would change.

It was certainly intended that it be sent. In practice, if you send a
RemovePersistentRequest this will cause the request to be cancelled,
which will then cause it to produce a GetFailed with reason=Cancelled.
The only problem with the current behaviour is that there is no
indication as to whether the request is still in the queue. We can fix
that by including an explicit indicator - a flag.
> >
> > > Its easy to implement this in the node and I offer to do the changes
> > > and to document them in the wiki. Existing clients are not affected.
> > >
> > > Please discuss this, and with your answer provide an explanation WHY
> > > you recommendation is better then the given ones, thanks.
> > >
> > > rgds, bback.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070205/3f8ffdd1/attachment.pgp 

From toad at amphibian.dyndns.org  Mon Feb  5 12:28:16 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 5 Feb 2007 12:28:16 +0000
Subject: [freenet-dev] [freenet-cvs] r11668 -
	trunk/freenet/src/freenet/crypt
In-Reply-To: <20070204144135.E09619BBE4@emu.freenetproject.org>
References: <20070204144135.E09619BBE4@emu.freenetproject.org>
Message-ID: <20070205122816.GB17176@amphibian.dyndns.org>

Nice catch! Thanks!

On Sun, Feb 04, 2007 at 02:41:35PM +0000, dbkr at freenetproject.org wrote:
> Author: dbkr
> Date: 2007-02-04 14:41:35 +0000 (Sun, 04 Feb 2007)
> New Revision: 11668
> 
> Modified:
>    trunk/freenet/src/freenet/crypt/RijndaelPCFBMode.java
> Log:
> Fix to r.11663 (RijndaelPCFBMode subclass constructor was missing a paramter to the superconstructer call).
> 
> 
> Modified: trunk/freenet/src/freenet/crypt/RijndaelPCFBMode.java
> ===================================================================
> --- trunk/freenet/src/freenet/crypt/RijndaelPCFBMode.java	2007-02-04 14:39:54 UTC (rev 11667)
> +++ trunk/freenet/src/freenet/crypt/RijndaelPCFBMode.java	2007-02-04 14:41:35 UTC (rev 11668)
> @@ -29,7 +29,7 @@
>      }
>  
>      public RijndaelPCFBMode(Rijndael c, byte[] iv) {
> -    	super(c);
> +    	super(c, iv);
>      	int tempSize = c.getTempArraySize();
>      	a = new int[tempSize];
>      	t = new int[tempSize];
> 
> _______________________________________________
> cvs mailing list
> cvs at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/cvs
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070205/49665a54/attachment.pgp 

From bbackde at googlemail.com  Mon Feb  5 12:29:22 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Mon, 5 Feb 2007 13:29:22 +0100
Subject: [freenet-dev] New FCP2 messages
In-Reply-To: <20070205122105.GC23223@amphibian.dyndns.org>
References: <d6149a790702041238l7f1a0eadhb86022332aac18d1@mail.gmail.com>
	<20070205122105.GC23223@amphibian.dyndns.org>
Message-ID: <d6149a790702050429w2db529ev22531b35996289a0@mail.gmail.com>

On 2/5/07, Matthew Toseland <toad at amphibian.dyndns.org> wrote:
> On Sun, Feb 04, 2007 at 09:38:47PM +0100, bbackde at googlemail.com wrote:
> > If FCP2 want to provide clients an easy access, then the following
> > changes are needed:
> >
> > Answer to a ModifyPersistentRequest should not be a new
> > Persistent<blah> with all values, but a ModifiedPersistentRequest
> > message with only the changed values. Otherwise the client has to do
> > all syncing and searching for really changed values.
>
> I don't get this. Why not just have a Persistent<blah> ? Surely less
> messages is the simpler solution?

It makes the processing for clients easier. And I think this should be
the goal. Consider the code of a client: the client gets a new
PersistentBlah message, and if he already have the request in its
table/list, then he now have to compare all values (knowing which ones
could change) against the new values.
A ModifiedPersistentRequest message would contain only the changed
values, making life for clients easier. Also its more simple for
clients to get a concrete answer to a request, not a common answer
like a PersistentBlah.

This is just my point of view, as client developer ;)  feel free to
vote me down.

> >
> > Answer to a RemovePersistentRequest should be a
> > RemovedPersistentRequest, this is a clear answer to the request and
> > clients don't need to do much investigation here.
>
> Or a GetFailed with Removed=true.

See above. Of course the client could figure out what happened with a
flag but its easier with an own message.

> >
> > Its easy to implement this in the node and I offer to do the changes
> > and to document them in the wiki. Existing clients are not affected.
>
> By all means.
> >
> > Please discuss this, and with your answer provide an explanation WHY
> > you recommendation is better then the given ones, thanks.
> >
> > rgds, bback.
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFFxyExA9rUluQ9pFARAhHgAJ9tha5al/ze6VRgMwoMH+sUW+KkqgCZAaQd
> 70Hj0vKr491RTTnuJROEoEc=
> =7v5n
> -----END PGP SIGNATURE-----
>
>
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>
>


From alejandro at mosteo.com  Mon Feb  5 12:42:00 2007
From: alejandro at mosteo.com (Jano)
Date: Mon, 05 Feb 2007 13:42:00 +0100
Subject: [freenet-dev] BDB uses far more memory than it's supposed to;
	ideas?
References: <20070130165441.GA24058@amphibian.dyndns.org>
	<1170193145.2232.449.camel@orchid.arb.net>
	<epqrm4$1va$1@sea.gmane.org>
	<20070201184529.GC11653@amphibian.dyndns.org>
	<eptrne$c7r$1@sea.gmane.org>
	<20070202193241.GA29915@amphibian.dyndns.org>
	<eq31u5$cag$1@sea.gmane.org> <eq555o$ipq$1@sea.gmane.org>
	<20070205121829.GA23223@amphibian.dyndns.org>
Message-ID: <eq78ne$u9k$1@sea.gmane.org>

Matthew Toseland wrote:

> [B [I [C [[B almost certainly mean byte[], int[], char[], byte[][].

Good to know.

> I didn't get jmap to work (maybe because I was using java 5), but I did
> do some invasive profiling (with stack traces), and used that to
> identify and eliminate some high-churn objects; if the current problem is
> that too much garbage collection is occurring (causing 100% cpu usage),
> this is most likely caused by too many objects being allocated per second.

FWIW, I have the OoM problem but not high CPU problems (see my graphs
pending moderation).

Though, as the point of OoM gets closer, the JVM will attempt a full OoM
each time it would run out of memory, so I'd say that in the final moments
of a node, CPU churn due to GC will be very high (I'll try to capture these
moments with jconsole, very nice tool).

> Anyway my original trace is here:
> http://amphibian.dyndns.org/java.hprof.multi-day.no-logging.1011.txt
> 
> This is produced by options:
> wrapper.java.additional.3=-Xloggc:freenet.loggc
>
wrapper.java.additional.4=-Xrunhprof:heap=all,format=a,depth=12,lineno=y,doe=y,
> gc_okay=y
> 
> (Set logging to NORMAL if you haven't already; heavy logging with
> profiling makes the node break)
> 
> On Sun, Feb 04, 2007 at 06:29:04PM +0100, Jano wrote:
>> Jano wrote:
>> 
>> > Matthew Toseland wrote:
>> 
>> >> How to identify such misuse?
>> > 
>> > Memory profiling, I'd say. Though I have never done it with java.
>> 
>> I've started toying with jmap/jhat after upgrading to java6. This node is
>> a linux one. Here's a heap dump at node start:
>> 
>> Top 12 Class histogram:
>> 
>> Class                                   Instance Count  Total Size
>> class [B                                        379353  18473891
>> class [I                                        315048  12805068
>> class [C                                        56549   11786932
>> class [[B                                       6137    3190228
>> class [Lcom.sleepycat.je.tree.Node;             6135    3190200
>> class [Ljava.util.HashMap$Entry;                33158   2950160
>> class [J                                        2037    2091504
>> class freenet.client.async.SingleBlockInserter  30965   1795970
>> class [[I                                       20812   1415092
>> class java.util.HashMap$Entry                   66778   1068448
>> class java.util.HashMap                         33144   1060608
>> class java.lang.String                          56831   909296
>> 
>> Top 10 class instance counts (excluding platform):
>> 
>> 31204 instances of class freenet.client.FailureCodeTracker
>> 30965 instances of class freenet.client.async.SingleBlockInserter
>> 20604 instances of class freenet.support.io.ReadOnlyFileSliceBucket
>> 16213 instances of class com.sleepycat.je.tree.LN
>> 10413 instances of class freenet.support.io.FileBucket
>> 10403 instances of class freenet.crypt.ciphers.Rijndael
>> 10367 instances of class
>> freenet.support.io.PaddedEphemerallyEncryptedBucket 10355 instances of
>> class freenet.support.io.DelayedFreeBucket 9947 instances of class
>> freenet.keys.FreenetURI 6157 instances of class
>> com.sleepycat.je.latch.Java5SharedLatchImpl
>> 
>> Total instances pending finalization: 0
>> 
>> The strange short names are, so it seems, "platform classes", which I
>> don't know what means (apparently java.* and javax.* classes, but these
>> strange names go away too).
>> 
>> And here's the heap dump at the moment of node death (node configured
>> with 128m max mem):
>> 
>> Class                                   Instance Count  Total Size
>> class [B                                        771728  34242096
>> class [C                                        113887  19922068
>> class [I                                        439531  18758092
>> class [[B                                       9527    4953028
>> class [Lcom.sleepycat.je.tree.Node;             9525    4953000
>> class [Ljava.util.HashMap$Entry;                50089   4642912
>> class [J                                        3030    3111184
>> class com.sleepycat.je.tree.LN                  173461  2254993
>> class [[I                                       28586   1943756
>> class java.lang.String                          115987  1855792
>> class freenet.client.async.SingleBlockInserter  31080   1802640
>> class java.util.HashMap$Entry                   108956  1743296
>> 
>> 173461 instances of class com.sleepycat.je.tree.LN
>> 31413 instances of class freenet.client.FailureCodeTracker
>> 31080 instances of class freenet.client.async.SingleBlockInserter
>> 20613 instances of class freenet.support.io.ReadOnlyFileSliceBucket
>> 17564 instances of class freenet.keys.FreenetURI
>> 14291 instances of class freenet.crypt.ciphers.Rijndael
>> 14246 instances of class
>> freenet.support.io.PaddedEphemerallyEncryptedBucket 11793 instances of
>> class freenet.support.io.FileBucket 10358 instances of class
>> freenet.support.io.DelayedFreeBucket 10065 instances of class
>> freenet.support.LRUQueue$QItem
>> 
>> Total instances pending finalization: 0
>> 
>> From these numbers it seems that sleepycat is notably leaking memory (the
>> LN objects). Someone familiar with DBD could perhaps pinpoint seeing this
>> if this is a bug in BDB or a mismanagement in freenet (unclosed
>> whatevers?)
>> 
>> I'm running now with 256m to see if the differences are even more
>> apparent. I'll send later some graphs obtained with jconsole that pretty
>> much back that there's leaking going on (i.e. if we find the leak,
>> freenet will run comfortably with 128m or less).
>> 
>> _______________________________________________
>> Devl mailing list
>> Devl at freenetproject.org
>> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>>




From toad at amphibian.dyndns.org  Mon Feb  5 12:45:46 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 5 Feb 2007 12:45:46 +0000
Subject: [freenet-dev] [freenet-cvs] r11679 - in
	trunk/apps/load-balancing-sims/phase7/sim: . clients messages
In-Reply-To: <20070205115216.B20789BB8F@emu.freenetproject.org>
References: <20070205115216.B20789BB8F@emu.freenetproject.org>
Message-ID: <20070205124546.GD17176@amphibian.dyndns.org>

What's going on with the sendAck(int) -> send(int)?

On Mon, Feb 05, 2007 at 11:52:16AM +0000, mrogers at freenetproject.org wrote:
> Author: mrogers
> Date: 2007-02-05 11:52:15 +0000 (Mon, 05 Feb 2007)
> New Revision: 11679
> 
> Modified:
>    trunk/apps/load-balancing-sims/phase7/sim/Node.java
>    trunk/apps/load-balancing-sims/phase7/sim/Packet.java
>    trunk/apps/load-balancing-sims/phase7/sim/Peer.java
>    trunk/apps/load-balancing-sims/phase7/sim/Sim.java
>    trunk/apps/load-balancing-sims/phase7/sim/clients/Client.java
>    trunk/apps/load-balancing-sims/phase7/sim/clients/SimplePublisher.java
>    trunk/apps/load-balancing-sims/phase7/sim/messages/ChkInsert.java
>    trunk/apps/load-balancing-sims/phase7/sim/messages/Search.java
>    trunk/apps/load-balancing-sims/phase7/sim/messages/SskInsert.java
> Log:
> Disabled ack coalescing in preparation for improved RTO/FRTO calculation
> 
> Modified: trunk/apps/load-balancing-sims/phase7/sim/Node.java
> ===================================================================
> --- trunk/apps/load-balancing-sims/phase7/sim/Node.java	2007-02-05 00:36:43 UTC (rev 11678)
> +++ trunk/apps/load-balancing-sims/phase7/sim/Node.java	2007-02-05 11:52:15 UTC (rev 11679)
> @@ -1,5 +1,5 @@
>  package sim;
> -import sim.generators.Client;
> +import sim.clients.Client;
>  import sim.handlers.*;
>  import sim.messages.*;
>  import java.util.HashMap;
> 
> Modified: trunk/apps/load-balancing-sims/phase7/sim/Packet.java
> ===================================================================
> --- trunk/apps/load-balancing-sims/phase7/sim/Packet.java	2007-02-05 00:36:43 UTC (rev 11678)
> +++ trunk/apps/load-balancing-sims/phase7/sim/Packet.java	2007-02-05 11:52:15 UTC (rev 11679)
> @@ -1,40 +1,32 @@
>  // A low-level packet (as opposed to a high-level message)
>  
>  package sim;
> -import sim.messages.Ack;
>  import sim.messages.Message;
>  import java.util.ArrayList;
>  
>  class Packet
>  {
> -	public final static int HEADER_SIZE = 70; // Including IP & UDP headers
> -	public final static int ACK_SIZE = 4; // Size of an ack in bytes
> +	public final static int HEADER_SIZE = 60; // Including IP & UDP headers
>  	public final static int MAX_SIZE = 1450; // MTU including headers
>  	public final static int SENSIBLE_PAYLOAD = 1000; // Coalescing
>  	
>  	public final int src, dest; // Network addresses
>  	public int size = HEADER_SIZE; // Size in bytes, including headers
>  	public int seq = -1; // Data sequence number (-1 if no data)
> -	public ArrayList<Ack> acks = null;
> +	public int ack = -1; // Ack sequence number (-1 if no ack)
>  	public ArrayList<Message> messages = null;
>  	
>  	public double sent; // Time at which the packet was (re) transmitted
>  	public double latency; // Link latency, stored here for convenience
>  	
> -	public Packet (int src, int dest, double latency)
> +	public Packet (int src, int dest, double latency, int ack)
>  	{
>  		this.src = src;
>  		this.dest = dest;
>  		this.latency = latency;
> +		this.ack = ack;
>  	}
>  	
> -	public void addAck (Ack a)
> -	{
> -		if (acks == null) acks = new ArrayList<Ack>();
> -		acks.add (a);
> -		size += a.size();
> -	}
> -	
>  	public void addMessage (Message m)
>  	{
>  		if (messages == null) messages = new ArrayList<Message>();
> 
> Modified: trunk/apps/load-balancing-sims/phase7/sim/Peer.java
> ===================================================================
> --- trunk/apps/load-balancing-sims/phase7/sim/Peer.java	2007-02-05 00:36:43 UTC (rev 11678)
> +++ trunk/apps/load-balancing-sims/phase7/sim/Peer.java	2007-02-05 11:52:15 UTC (rev 11679)
> @@ -36,7 +36,6 @@
>  	private int txSeq = 0; // Sequence number of next outgoing data packet
>  	private int txMaxSeq = SEQ_RANGE - 1; // Highest sequence number
>  	private LinkedList<Packet> txBuffer; // Retransmission buffer
> -	private DeadlineQueue<Ack> ackQueue; // Outgoing acks
>  	private DeadlineQueue<Message> searchQueue; // Outgoing search messages
>  	private DeadlineQueue<Message> transferQueue; // Outgoing transfers
>  	private CongestionWindow window; // AIMD congestion window
> @@ -61,7 +60,6 @@
>  		this.location = location;
>  		this.latency = latency;
>  		txBuffer = new LinkedList<Packet>();
> -		ackQueue = new DeadlineQueue<Ack>();
>  		searchQueue = new DeadlineQueue<Message>();
>  		transferQueue = new DeadlineQueue<Message>();
>  		window = new CongestionWindow (this);
> @@ -83,20 +81,9 @@
>  		// Start the coalescing timer
>  		startTimer();
>  		// Send as many packets as possible
> -		while (send());
> +		while (send (-1));
>  	}
>  	
> -	// Queue an ack for transmission
> -	private void sendAck (int seq)
> -	{
> -		if (LOG) log ("ack " + seq + " added to ack queue");
> -		ackQueue.add (new Ack (seq, Event.time() + MAX_DELAY));
> -		// Start the coalescing timer
> -		startTimer();
> -		// Send as many packets as possible
> -		while (send());
> -	}
> -	
>  	// Start the coalescing timer
>  	private void startTimer()
>  	{
> @@ -107,11 +94,11 @@
>  	}
>  	
>  	// Try to send a packet, return true if a packet was sent
> -	private boolean send()
> +	private boolean send (int ack)
>  	{
> -		int waiting = ackQueue.size+searchQueue.size+transferQueue.size;
> +		int waiting = searchQueue.size + transferQueue.size;
>  		if (LOG) log (waiting + " bytes waiting");
> -		if (waiting == 0) return false;
> +		if (ack == -1 && waiting == 0) return false;
>  		
>  		// Return to slow start when the link is idle
>  		double now = Event.time();
> @@ -123,29 +110,30 @@
>  		size = Math.min (size, node.bandwidth.available());
>  		if (LOG) log (size + " bytes available for packet");
>  		
> -		// Urgent acks to send?
> -		if (ackQueue.deadline() <= now) return sendPacket (size);
> +		// Ack to send?
> +		if (ack != -1) return sendPacket (ack, size);
>  		// Urgent searches and room to send them?
>  		if (searchQueue.deadline() <= now
> -		&& searchQueue.headSize() <= size) return sendPacket (size);
> +		&& searchQueue.headSize() <= size)
> +			return sendPacket (ack, size);
>  		// Urgent transfers and room to send them?
>  		if (transferQueue.deadline() <= now
> -		&& transferQueue.headSize() <= size) return sendPacket (size);
> +		&& transferQueue.headSize() <= size)
> +			return sendPacket (ack, size);
>  		// Enough non-urgent messages for a large packet, and room?
>  		if (waiting >= Packet.SENSIBLE_PAYLOAD
> -		&& size >= Packet.SENSIBLE_PAYLOAD) return sendPacket (size);
> +		&& size >= Packet.SENSIBLE_PAYLOAD)
> +			return sendPacket (ack, size);
>  		
>  		if (LOG) log ("not sending a packet");
>  		return false;
>  	}
>  	
>  	// Try to send a packet up to the specified size, return true if sent
> -	private boolean sendPacket (int maxSize)
> +	private boolean sendPacket (int ack, int maxSize)
>  	{
>  		// Construct a packet
> -		Packet p = new Packet (node.net.address, address, latency);
> -		// Add all waiting acks to the packet
> -		while (ackQueue.size > 0) p.addAck (ackQueue.pop());
> +		Packet p = new Packet (node.net.address, address, latency, ack);
>  		if (LOG) log ((maxSize - p.size) + " bytes for messages");
>  		// Don't allow more than SEQ_RANGE payloads to be in flight
>  		if (txSeq <= txMaxSeq) {
> @@ -169,7 +157,7 @@
>  			log ("waiting for ack " + (txMaxSeq - SEQ_RANGE + 1));
>  		}
>  		// Don't send empty packets
> -		if (p.acks == null && p.messages == null) return false;
> +		if (p.ack == -1 && p.messages == null) return false;
>  		// Transmit the packet
>  		if (LOG) log ("sending packet " +p.seq+ ", " +p.size+ " bytes");
>  		node.sendPacket (p);
> @@ -186,7 +174,7 @@
>  	// Called by Node when a packet arrives
>  	public void handlePacket (Packet p)
>  	{
> -		if (p.acks != null) for (Ack a : p.acks) handleAck (a);
> +		if (p.ack != -1) handleAck (p.ack);
>  		if (p.messages != null) handleData (p);
>  	}
>  	
> @@ -195,7 +183,7 @@
>  		if (LOG) log ("received packet " +p.seq+ ", expected " +rxSeq);
>  		if (p.seq < rxSeq || rxDupe.contains (p.seq)) {
>  			if (LOG) log ("duplicate packet");
> -			sendAck (p.seq); // Original ack may have been lost
> +			send (p.seq); // Original ack may have been lost
>  		}
>  		else if (p.seq == rxSeq) {
>  			// Find the sequence number of the next missing packet
> @@ -204,7 +192,7 @@
>  			// Deliver the messages to the node
>  			for (Message m : p.messages)
>  				node.handleMessage (m, this);
> -			sendAck (p.seq);
> +			send (p.seq);
>  		}
>  		else if (p.seq < rxSeq + SEQ_RANGE) {
>  			if (LOG) log ("packet out of order");
> @@ -212,37 +200,36 @@
>  			// Deliver the messages to the node
>  			for (Message m : p.messages)
>  				node.handleMessage (m, this);
> -			sendAck (p.seq);
> +			send (p.seq);
>  		}
>  		// This indicates a misbehaving sender - discard the packet
>  		else if (LOG) log ("WARNING: sequence number out of range");
>  	}
>  	
> -	private void handleAck (Ack a)
> +	private void handleAck (int ack)
>  	{
> -		int seq = a.id;
> -		if (LOG) log ("received ack " + seq);
> +		if (LOG) log ("received ack " + ack);
>  		double now = Event.time();
>  		Iterator<Packet> i = txBuffer.iterator();
>  		while (i.hasNext()) {
>  			Packet p = i.next();
>  			double age = now - p.sent;
>  			// Explicit ack
> -			if (p.seq == seq) {
> +			if (p.seq == ack) {
>  				i.remove();
>  				// Update the congestion window
>  				window.bytesAcked (p.size);
>  				// Update the average round-trip time
>  				rtt = rtt * RTT_DECAY + age * (1.0 - RTT_DECAY);
>  				if (LOG) {
> -					log ("packet " +p.seq+ " acknowledged");
> +					log ("packet " + ack + " acknowledged");
>  					log ("round-trip time " + age);
>  					log ("average round-trip time " + rtt);
>  				}
>  				break;
>  			}
>  			// Fast retransmission
> -			if (p.seq < seq && age > FRTO * rtt + MAX_DELAY) {
> +			if (p.seq < ack && age > FRTO * rtt) {
>  				p.sent = now;
>  				if (LOG) log ("fast retransmitting " + p.seq);
>  				node.resendPacket (p);
> @@ -253,8 +240,8 @@
>  		if (txBuffer.isEmpty()) txMaxSeq = txSeq + SEQ_RANGE - 1;
>  		else txMaxSeq = txBuffer.peek().seq + SEQ_RANGE - 1;
>  		if (LOG) log ("maximum sequence number " + txMaxSeq);
> -		// Send as many packets a possible
> -		if (timerRunning) while (send());
> +		// Send as many packets as possible
> +		if (timerRunning) while (send (-1));
>  		else checkDeadlines();
>  	}
>  	
> @@ -326,7 +313,7 @@
>  		
>  		double now = Event.time();
>  		for (Packet p : txBuffer) {
> -			if (now - p.sent > RTO * rtt + MAX_DELAY) {
> +			if (now - p.sent > RTO * rtt) {
>  				// Retransmission timeout
>  				if (LOG) log ("retransmitting " + p.seq);
>  				p.sent = now;
> @@ -341,11 +328,11 @@
>  	private void checkDeadlines()
>  	{
>  		// Send as many packets as possible
> -		while (send());
> +		while (send (-1));
>  		// Find the next coalescing deadline - ignore message deadlines
>  		// if there isn't room in the congestion window to send them
> -		double dl = ackQueue.deadline();
> -		int win = window.available() -Packet.HEADER_SIZE -ackQueue.size;
> +		double dl = Double.POSITIVE_INFINITY;
> +		int win = window.available() - Packet.HEADER_SIZE;
>  		if (searchQueue.headSize() <= win)
>  			dl = Math.min (dl, searchQueue.deadline());
>  		if (transferQueue.headSize() <= win)
> @@ -370,12 +357,9 @@
>  	// Are we waiting for the bandwidth limiter?
>  	private boolean shouldPoll()
>  	{
> -		double now = Event.time();
> -		// Will we need to send an ack before the next polling interval?
> -		if (ackQueue.deadline() < now + node.bandwidth.poll)
> -			return false;
>  		double bw = node.bandwidth.available();
>  		double win = window.available();
> +		double now = Event.time();
>  		// Is there an overdue search that's waiting for bandwidth?
>  		if (searchQueue.headSize() > bw
>  		&& searchQueue.headSize() <= win
> 
> Modified: trunk/apps/load-balancing-sims/phase7/sim/Sim.java
> ===================================================================
> --- trunk/apps/load-balancing-sims/phase7/sim/Sim.java	2007-02-05 00:36:43 UTC (rev 11678)
> +++ trunk/apps/load-balancing-sims/phase7/sim/Sim.java	2007-02-05 11:52:15 UTC (rev 11679)
> @@ -1,5 +1,5 @@
>  package sim;
> -import sim.generators.SimplePublisher;
> +import sim.clients.SimplePublisher;
>  
>  class Sim implements EventTarget
>  {
> 
> Modified: trunk/apps/load-balancing-sims/phase7/sim/clients/Clie