From nextgens at freenetproject.org  Sat Dec  1 00:08:40 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=)
Date: Sat, 1 Dec 2007 01:08:40 +0100
Subject: [freenet-dev] Another update to german language
In-Reply-To: <4750A3C4.5020009@gmx.de>
References: <20071130224522.26781390874@freenetproject.org>
	<4750A3C4.5020009@gmx.de>
Message-ID: <20071201000840.GC10184@freenetproject.org>

* Thomas <tommy100 at gmx.de> [2007-12-01 00:59:00]:

> Added/changed some more strings and reverted 2 not intended changes from last commit

Commited in r16172
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071201/18513a85/attachment.pgp 

From toad at amphibian.dyndns.org  Sat Dec  1 16:18:22 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 1 Dec 2007 16:18:22 +0000
Subject: [freenet-dev] Why Freenet is so SLOW! Proposal for new load
	management
In-Reply-To: <47509E27.6090303@cs.ucl.ac.uk>
References: <200711300126.45810.toad@amphibian.dyndns.org>
	<200711301859.58824.toad@amphibian.dyndns.org>
	<47509E27.6090303@cs.ucl.ac.uk>
Message-ID: <200712011618.30874.toad@amphibian.dyndns.org>

On Friday 30 November 2007 23:35, Michael Rogers wrote:
> Matthew Toseland wrote:
> > TCP/IP is end-to-end. It loses packets on each hop.
> 
> There's really no comparison between Freenet requests and IP packets.
> Packets don't backtrack. Routers don't set local timeouts for every
> packet. There's no flow control or retransmission between routers, only
> between endpoints. TCP is *not* a good model for what happens end-to-end
> across Freenet. (In fact the term end-to-end doesn't even make sense - a
> single insert can reach multiple endpoints!)

Agreed, it's not a good analogy. But it's the basis of the current 
backoff-plus-AIMD load management system. Which is why we need to get rid of 
it.
> 
> *However* TCP is a good model for what happens between neighbouring
> nodes. In fact it's such a good model that I think we should copy it.

We do. The link layer uses a protocol which behaves more or less the same as 
TCP with regard to congestion control.
> 
> > One request and one ack use the same number of bytes in immediate terms of 
> > just that message, but a request will use a LOT more bytes overall.
> > 
> > Therefore, IMHO it would be inappropriate to limit messages rather than 
> > requests. We may need to limit messages as well, but requests are the main 
> > concern.
> 
> You're right, token passing isn't redundant after all. We need a
> transport layer that limits messages, and a load balancing layer that
> limits requests. They should be separate layers.
> 
> > How exactly do you decide which requests to accept then?
> 
> You don't decide, because there's no rejection any more. You just hand
> out tokens and try to forward whatever requests your peers in their
> infinite wisdom decide to send you.

IMHO we would still need rejection, for several reasons:
1. Reject due to loop.
2. Reject due to overload if something unexpected happens.
3. Reject due to overload as a normal part of the operation of the node 
because if we just send one token to one node it will not always send a 
request, so IMHO we have to send tokens to several nodes and then reject some 
if we get more requests than we'd expected.
> 
> Cheers,
> Michael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071201/9d72e59b/attachment.pgp 

From toad at amphibian.dyndns.org  Sat Dec  1 17:51:35 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 1 Dec 2007 17:51:35 +0000
Subject: [freenet-dev] [freenet-cvs] r16168 -
	trunk/freenet/src/freenet/node/fcp
In-Reply-To: <20071130234406.7F295479EF5@freenetproject.org>
References: <20071130234406.7F295479EF5@freenetproject.org>
Message-ID: <200712011751.41083.toad@amphibian.dyndns.org>

IMHO we should allow DDA's to have an Identifier, which if not specified is 
the directory name given. Then we can easily implement #1753.

On Friday 30 November 2007 23:44, you wrote:
> Author: nextgens
> Date: 2007-11-30 23:44:06 +0000 (Fri, 30 Nov 2007)
> New Revision: 16168
> 
> Modified:
>    trunk/freenet/src/freenet/node/fcp/FCPConnectionHandler.java
> Log:
> Resolve #1856: Nested TestDDA fails
> 
> Modified: trunk/freenet/src/freenet/node/fcp/FCPConnectionHandler.java
> ===================================================================
> --- trunk/freenet/src/freenet/node/fcp/FCPConnectionHandler.java	2007-11-30 
23:34:04 UTC (rev 16167)
> +++ trunk/freenet/src/freenet/node/fcp/FCPConnectionHandler.java	2007-11-30 
23:44:06 UTC (rev 16168)
> @@ -372,8 +372,15 @@
>  		if(!directory.exists() || !directory.isDirectory())
>  			throw new IllegalArgumentException("The specified path isn't a 
directory! or doesn't exist or the node doesn't have access to it!");
>  		
> +		// See #1856
> +		DDACheckJob job = null;
> +		synchronized (inTestDirectories) {
> +			job = (DDACheckJob) inTestDirectories.get(directory);
> +		}
> +		if(job != null)
> +			throw new IllegalArgumentException("There is already a TestDDA going on 
for that directory!");
> +		
>  		File writeFile = (write ? new File(path, "DDACheck-" + new 
Random().nextInt() + ".tmp") : null);
> -		
>  		File readFile = null;
>  		if(read) {
>  			try {
> 
> _______________________________________________
> cvs mailing list
> cvs at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/cvs
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071201/f1c714d0/attachment.pgp 

From toad at amphibian.dyndns.org  Sat Dec  1 18:07:57 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 1 Dec 2007 18:07:57 +0000
Subject: [freenet-dev] [freenet-cvs] r16182 -
	trunk/freenet/src/freenet/node
In-Reply-To: <20071201151436.632FC47AE7D@freenetproject.org>
References: <20071201151436.632FC47AE7D@freenetproject.org>
Message-ID: <200712011807.57341.toad@amphibian.dyndns.org>

Wrong. The below will not disconnect if we rekeyed due to transferred data 
rather than due to time.

On Saturday 01 December 2007 15:14, nextgens at freenetproject.org wrote:
> Author: nextgens
> Date: 2007-12-01 15:14:36 +0000 (Sat, 01 Dec 2007)
> New Revision: 16182
> 
> Modified:
>    trunk/freenet/src/freenet/node/PeerNode.java
> Log:
> JFK: simplify|fix the logic of rekeying
> 
> Modified: trunk/freenet/src/freenet/node/PeerNode.java
> ===================================================================
> --- trunk/freenet/src/freenet/node/PeerNode.java	2007-12-01 14:50:20 UTC 
(rev 16181)
> +++ trunk/freenet/src/freenet/node/PeerNode.java	2007-12-01 15:14:36 UTC 
(rev 16182)
> @@ -899,30 +899,33 @@
>  
>  	protected void maybeRekey() {
>  		long now = System.currentTimeMillis();
> -		boolean hasRekeyed = false;
> -		if(hasLiveHandshake(now))
> -			return;
> -
> +		boolean shouldDisconnect = false;
> +		boolean shouldReturn = false;
> +		boolean shouldRekey = false;
>  		long timeWhenRekeyingShouldOccur = 0;
> -		synchronized(this) {
> -			if(isRekeying || !isConnected)
> -				return;
> +		
> +		synchronized (this) {
>  			timeWhenRekeyingShouldOccur = timeLastRekeyed + 
FNPPacketMangler.SESSION_KEY_REKEYING_INTERVAL;
> -			if((timeWhenRekeyingShouldOccur < now) || 
(totalBytesExchangedWithCurrentTracker > 
FNPPacketMangler.AMOUNT_OF_BYTES_ALLOWED_BEFORE_WE_REKEY)) {
> -				hasRekeyed = true;
> +			shouldDisconnect = (timeWhenRekeyingShouldOccur + 
FNPPacketMangler.MAX_SESSION_KEY_REKEYING_DELAY < now) && isRekeying;
> +			shouldReturn = isRekeying || !isConnected;
> +			shouldRekey = (timeWhenRekeyingShouldOccur < now) || 
(totalBytesExchangedWithCurrentTracker > 
FNPPacketMangler.AMOUNT_OF_BYTES_ALLOWED_BEFORE_WE_REKEY);
> +		}
> +		
> +		if(shouldDisconnect) {
> +			String time = 
TimeUtil.formatTime(FNPPacketMangler.MAX_SESSION_KEY_REKEYING_DELAY);
> +			System.err.println("The peer (" + this + ") has been asked to rekey " + 
time + " ago... force disconnect.");
> +			Logger.error(this, "The peer (" + this + ") has been asked to rekey " + 
time + " ago... force disconnect.");
> +			forceDisconnect();
> +		} else if (shouldReturn || hasLiveHandshake(now)) {
> +			return;
> +		} else if(shouldRekey) {
> +			synchronized(this) {
>  				isRekeying = true;
>  				sendHandshakeTime = now; // Immediately
>  				ctx = null;
>  			}
> +			Logger.normal(this, "We are asking for the key to be renewed (" + 
this.detectedPeer + ')');
>  		}
> -
> -		if(timeWhenRekeyingShouldOccur + 
FNPPacketMangler.MAX_SESSION_KEY_REKEYING_DELAY < now) {
> -			Logger.error(this, "The peer (" + this + ") has been asked to rekey " + 
TimeUtil.formatTime(FNPPacketMangler.MAX_SESSION_KEY_REKEYING_DELAY) + " 
ago... force disconnect.");
> -			forceDisconnect();
> -		}
> -
> -		if(hasRekeyed)
> -			Logger.normal(this, "We are asking for the key to be renewed (" + 
this.detectedPeer + ')');
>  	}
>  
>  	/**
> 
> _______________________________________________
> cvs mailing list
> cvs at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/cvs
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071201/8f3325ab/attachment.pgp 

From toad at amphibian.dyndns.org  Sat Dec  1 18:17:09 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 1 Dec 2007 18:17:09 +0000
Subject: [freenet-dev] [freenet-cvs] r16125 -
	trunk/freenet/src/freenet/node
In-Reply-To: <20071130131032.8590E47962C@freenetproject.org>
References: <20071130131032.8590E47962C@freenetproject.org>
Message-ID: <200712011817.10452.toad@amphibian.dyndns.org>

Is there any way that this can be verified automatically? If not, I'd like you 
to revert it. Indenting as in changing whitespace is one thing, indenting as 
in eliminating the {}'s in { one statement } is quite another thing...

On Friday 30 November 2007 13:10, you wrote:
> Author: nextgens
> Date: 2007-11-30 13:10:32 +0000 (Fri, 30 Nov 2007)
> New Revision: 16125
> 
> Modified:
>    trunk/freenet/src/freenet/node/PeerNode.java
> Log:
> Indent properly
> 
> Modified: trunk/freenet/src/freenet/node/PeerNode.java
> ===================================================================

[ snip almost the entire file ]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071201/cf97d6e3/attachment.pgp 

From nextgens at freenetproject.org  Sat Dec  1 19:27:21 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=)
Date: Sat, 1 Dec 2007 20:27:21 +0100
Subject: [freenet-dev] [freenet-cvs] r16168
	-	trunk/freenet/src/freenet/node/fcp
In-Reply-To: <200712011751.41083.toad@amphibian.dyndns.org>
References: <20071130234406.7F295479EF5@freenetproject.org>
	<200712011751.41083.toad@amphibian.dyndns.org>
Message-ID: <20071201192721.GC4351@freenetproject.org>

* Matthew Toseland <toad at amphibian.dyndns.org> [2007-12-01 17:51:35]:

> IMHO we should allow DDA's to have an Identifier, which if not specified is 
> the directory name given. Then we can easily implement #1753.
> 

And what would be the point ? how do we decide which one rules if
results are different ?

> On Friday 30 November 2007 23:44, you wrote:
> > Author: nextgens
> > Date: 2007-11-30 23:44:06 +0000 (Fri, 30 Nov 2007)
> > New Revision: 16168
> > 
> > Modified:
> >    trunk/freenet/src/freenet/node/fcp/FCPConnectionHandler.java
> > Log:
> > Resolve #1856: Nested TestDDA fails
> > 
> > Modified: trunk/freenet/src/freenet/node/fcp/FCPConnectionHandler.java
> > ===================================================================
> > --- trunk/freenet/src/freenet/node/fcp/FCPConnectionHandler.java	2007-11-30 
> 23:34:04 UTC (rev 16167)
> > +++ trunk/freenet/src/freenet/node/fcp/FCPConnectionHandler.java	2007-11-30 
> 23:44:06 UTC (rev 16168)
> > @@ -372,8 +372,15 @@
> >  		if(!directory.exists() || !directory.isDirectory())
> >  			throw new IllegalArgumentException("The specified path isn't a 
> directory! or doesn't exist or the node doesn't have access to it!");
> >  		
> > +		// See #1856
> > +		DDACheckJob job = null;
> > +		synchronized (inTestDirectories) {
> > +			job = (DDACheckJob) inTestDirectories.get(directory);
> > +		}
> > +		if(job != null)
> > +			throw new IllegalArgumentException("There is already a TestDDA going on 
> for that directory!");
> > +		
> >  		File writeFile = (write ? new File(path, "DDACheck-" + new 
> Random().nextInt() + ".tmp") : null);
> > -		
> >  		File readFile = null;
> >  		if(read) {
> >  			try {
> > 
> > _______________________________________________
> > cvs mailing list
> > cvs at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/cvs
> > 
> > 



> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071201/1c62d709/attachment.pgp 

From nextgens at freenetproject.org  Sat Dec  1 19:30:33 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=)
Date: Sat, 1 Dec 2007 20:30:33 +0100
Subject: [freenet-dev] [freenet-cvs] r16125
	-	trunk/freenet/src/freenet/node
In-Reply-To: <200712011817.10452.toad@amphibian.dyndns.org>
References: <20071130131032.8590E47962C@freenetproject.org>
	<200712011817.10452.toad@amphibian.dyndns.org>
Message-ID: <20071201193033.GF4351@freenetproject.org>

* Matthew Toseland <toad at amphibian.dyndns.org> [2007-12-01 18:17:09]:

> Is there any way that this can be verified automatically? If not, I'd like you 
> to revert it. Indenting as in changing whitespace is one thing, indenting as 
> in eliminating the {}'s in { one statement } is quite another thing...

Hmm, sorry about that... doe anyone know how to tell netbeans not to do
that when it indents ?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071201/b83b8f91/attachment.pgp 

From nextgens at freenetproject.org  Sat Dec  1 19:49:17 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=)
Date: Sat, 1 Dec 2007 20:49:17 +0100
Subject: [freenet-dev] [freenet-cvs] r16125
	-	trunk/freenet/src/freenet/node
In-Reply-To: <200712011817.10452.toad@amphibian.dyndns.org>
References: <20071130131032.8590E47962C@freenetproject.org>
	<200712011817.10452.toad@amphibian.dyndns.org>
Message-ID: <20071201194917.GG4351@freenetproject.org>

* Matthew Toseland <toad at amphibian.dyndns.org> [2007-12-01 18:17:09]:

> Is there any way that this can be verified automatically?

I've found http://andrei.gmxhome.de/bytecode/index.html ... maybe you
can use it to compare the produced bytecode.

> If not, I'd like you to revert it.

Well, tell me; if that doesn't work I'll revert it.

> Indenting as in changing whitespace is one thing, indenting as 
> in eliminating the {}'s in { one statement } is quite another thing...

Agreed; I haven't done it on purpose : I naively trusted netbeans to
"only indent" the code.

NextGen$
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071201/ddccbe5e/attachment.pgp 

From toad at amphibian.dyndns.org  Sat Dec  1 22:54:30 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 1 Dec 2007 22:54:30 +0000
Subject: [freenet-dev] Freenet 0.7 build 1084
Message-ID: <200712012254.35799.toad@amphibian.dyndns.org>

Freenet 0.7 build 1084 is now available. Please upgrade; hopefully your node 
will do this for you automatically. This build features some significant bug 
fixes for recent connectivity problems, some FCP fixes, we drop some obsolete 
code, and the beginnings of opennet announcement (which isn't used yet).

Please report any bugs you find to the bug tracker 
(https://bugs.freenetproject.org ), via IRC, Frost (the freenet board) or 
support at freenetproject.org. Thanks for helping build Freenet.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071201/7653285c/attachment.pgp 

From toad at amphibian.dyndns.org  Sat Dec  1 23:13:40 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 1 Dec 2007 23:13:40 +0000
Subject: [freenet-dev]
	=?utf-8?q?=5Bfreenet-cvs=5D_r16168_-=09trunk/freene?=
	=?utf-8?q?t/src/freenet/node/fcp?=
In-Reply-To: <20071201192721.GC4351@freenetproject.org>
References: <20071130234406.7F295479EF5@freenetproject.org>
	<200712011751.41083.toad@amphibian.dyndns.org>
	<20071201192721.GC4351@freenetproject.org>
Message-ID: <200712012313.45234.toad@amphibian.dyndns.org>

On Saturday 01 December 2007 19:27, Florent Daigni?re wrote:
> * Matthew Toseland <toad at amphibian.dyndns.org> [2007-12-01 17:51:35]:
> 
> > IMHO we should allow DDA's to have an Identifier, which if not specified 
is 
> > the directory name given. Then we can easily implement #1753.
> 
> And what would be the point ? how do we decide which one rules if
> results are different ?

The point is it can be tied to a request. A request can then block on the 
result of the testDDA.
> 
> > On Friday 30 November 2007 23:44, you wrote:
> > > Author: nextgens
> > > Date: 2007-11-30 23:44:06 +0000 (Fri, 30 Nov 2007)
> > > New Revision: 16168
> > > 
> > > Modified:
> > >    trunk/freenet/src/freenet/node/fcp/FCPConnectionHandler.java
> > > Log:
> > > Resolve #1856: Nested TestDDA fails
> > > 
> > > Modified: trunk/freenet/src/freenet/node/fcp/FCPConnectionHandler.java
> > > ===================================================================
> > > --- trunk/freenet/src/freenet/node/fcp/FCPConnectionHandler.java	
2007-11-30 
> > 23:34:04 UTC (rev 16167)
> > > +++ trunk/freenet/src/freenet/node/fcp/FCPConnectionHandler.java	
2007-11-30 
> > 23:44:06 UTC (rev 16168)
> > > @@ -372,8 +372,15 @@
> > >  		if(!directory.exists() || !directory.isDirectory())
> > >  			throw new IllegalArgumentException("The specified path isn't a 
> > directory! or doesn't exist or the node doesn't have access to it!");
> > >  		
> > > +		// See #1856
> > > +		DDACheckJob job = null;
> > > +		synchronized (inTestDirectories) {
> > > +			job = (DDACheckJob) inTestDirectories.get(directory);
> > > +		}
> > > +		if(job != null)
> > > +			throw new IllegalArgumentException("There is already a TestDDA going 
on 
> > for that directory!");
> > > +		
> > >  		File writeFile = (write ? new File(path, "DDACheck-" + new 
> > Random().nextInt() + ".tmp") : null);
> > > -		
> > >  		File readFile = null;
> > >  		if(read) {
> > >  			try {
> > > 
> > > _______________________________________________
> > > cvs mailing list
> > > cvs at freenetproject.org
> > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/cvs
> > > 
> > > 
> 
> 
> 
> > _______________________________________________
> > Devl mailing list
> > Devl at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071201/f0759707/attachment.pgp 

From toad at amphibian.dyndns.org  Sat Dec  1 23:26:50 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 1 Dec 2007 23:26:50 +0000
Subject: [freenet-dev] Why Freenet is so SLOW! Proposal for new load
	management
In-Reply-To: <A36817BE-8DCF-45B0-B320-F2EA96E0BF61@osndok.com>
References: <200711300126.45810.toad@amphibian.dyndns.org>
	<200711301732.30968.toad@amphibian.dyndns.org>
	<A36817BE-8DCF-45B0-B320-F2EA96E0BF61@osndok.com>
Message-ID: <200712012326.51301.toad@amphibian.dyndns.org>

On Friday 30 November 2007 19:04, Robert Hailey wrote:
> 
> On Nov 30, 2007, at 11:32 AM, Matthew Toseland wrote:
> > Not true either, most rejects are due to bandwidth liability
> 
> I'm not disagreeing with you, but at least from my node:
> 
> Preemptive Rejection Reasons
> 	28543	>threadLimit
> 	655	>SUB_MAX_PING_TIME
> 
> Local Preemptive Rejection Reasons
> 	718	>threadLimit
> 	33	>SUB_MAX_PING_TIME
> 
> These, the only two listed values, I interpret as meaning that my node  
> refuses requests primarily because it is waiting on everyone else (the  
> thread; remembering the state information of requests while they are  
> processed).

That's only your node. And is it still doing that, post 1084? There were some 
wierd bugs going around...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071201/4691dcda/attachment.pgp 

From toad at amphibian.dyndns.org  Sat Dec  1 23:30:01 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 1 Dec 2007 23:30:01 +0000
Subject: [freenet-dev] Why Freenet is so SLOW! Proposal for new load
	management
In-Reply-To: <EA219B65-8C33-49C7-A3F0-475E1B3A35AA@osndok.com>
References: <200711300126.45810.toad@amphibian.dyndns.org>
	<200711301901.22336.toad@amphibian.dyndns.org>
	<EA219B65-8C33-49C7-A3F0-475E1B3A35AA@osndok.com>
Message-ID: <200712012330.01276.toad@amphibian.dyndns.org>

On Friday 30 November 2007 19:35, Robert Hailey wrote:
> 
> On Nov 30, 2007, at 1:01 PM, Matthew Toseland wrote:
> 
> > It really isn't THAT slow. It's not a significant factor in the  
> > network's
> > overall performance.
> >
> > On Friday 30 November 2007 18:49, Robert Hailey wrote:
> >>
> >> [...] that an SSK signature is verified at every node[...] would  
> >> yield the same
> >> result 100% of the time.[...]
> 
> On Nov 30, 2007, at 1:04 PM, Robert Hailey wrote:
> > [...] I interpret as meaning that my node refuses requests primarily  
> > because it is waiting on everyone else [...]
> 
> To use an analogy from the recent Houston/Katrina-Rita evacuation, in  
> relating to bandwidth. A single stoplight at the end of a major  
> freeway is not a significant factor, unless of course there are  
> 100,000 cars waiting at it. Even worse, make it two distantly-spaced  
> stoplights, and no effective progress will be made.
> 
> If it is was the case that all rejects are from bandwidth limitations,  
> it would follow that the network was running at optimal capacity, only  
> to be made better by re-arranging the utilization of the bandwidth. Or  
> also if the network was mostly-idle, I suppose. Besides i/o (network/ 
> disk), the other major latency is processing (crypto).

Perhaps. Or perhaps it isn't being used because bandwidth liability is too 
conservative. Or perhaps it isn't being used because we're not matching 
demand to supply, as I originally suggested: when there is a gap, we don't 
get a request, when there isn't a gap, we do.
> 
> My point is that while node X is verifying my request (which I'm going  
> to be doing anyway), I must wait. And in waiting, requests are being  
> dropped. I see the major problem at hand is that the time in-handling  
> of packets must be minimized, as if we are all waiting on the slowest  
> node, and for a slow-enough node, the cryptographic verification in a  
> virtual machine would be slow indeed. This might even be an effective  
> DDoS attack (accept-and-wait --> overload threads).

If every node on the network was waiting for one really slow node then yes 
we'd have a problem. :)
> 
> Or else, the current system in-place may be like ethernet: beyond a  
> point of congestion, it just doesn't work.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071201/ed35968e/attachment.pgp 

From toad at amphibian.dyndns.org  Sun Dec  2 00:02:02 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sun, 2 Dec 2007 00:02:02 +0000
Subject: [freenet-dev] Why Freenet is so SLOW!
In-Reply-To: <8EFC03C4-519E-48A0-8463-EE77E717B9BB@osndok.com>
References: <200711300126.45810.toad@amphibian.dyndns.org>
	<EA219B65-8C33-49C7-A3F0-475E1B3A35AA@osndok.com>
	<8EFC03C4-519E-48A0-8463-EE77E717B9BB@osndok.com>
Message-ID: <200712020002.10322.toad@amphibian.dyndns.org>

On Friday 30 November 2007 23:03, Robert Hailey wrote:
> 
> On Nov 30, 2007, at 1:35 PM, Robert Hailey wrote:
> > [...] I see the major problem at hand is that the time in-handling
> > of packets must be minimized[...]
> 
> To illustrate my point the following patch shows an approximate 100%  
> speedup for freenet, by my crude profiling.

Speedup measured how? More comments below.
> 
> Index: freenet/node/RequestHandler.java
> ===================================================================
> --- freenet/node/RequestHandler.java	(revision 16160)
> +++ freenet/node/RequestHandler.java	(working copy)
> @@ -121,7 +121,7 @@
>           	htl = source.decrementHTL(htl);
> 
>           Message accepted = DMT.createFNPAccepted(uid);
> -        source.sendSync(accepted, null);
> +        source.sendAsync(accepted, null, 0, null);
> 
>           Object o = node.makeRequestSender(key, htl, uid, source,  
> closestLoc, resetClosestLoc, false, true, false);
>           if(o instanceof KeyBlock) {
> 
> A much-more aggressive patch (which makes all RequestHandler sends  
> Async except the 'last') shows could be 12x; but am concerned over the  
> byte-counter side effects with which I am not familiar.

Hmmm. Okay, here's how it works:

*Sender is responsible for sending a request out, talking to the nodes doing 
the request.
*Handler is responsible for receiving a request, and keeping the requestor up 
to date.
One Sender may serve many Handlers, or it may serve a local request. One 
Handler may have no Sender if it is e.g. a request satisfied from the store, 
or if it ran out of HTL.

There are two main reasons for using sendSync():
- To make sure the connection to the requestor is still open, and responsive 
(won't timeout us and then reroute the request) and kill the request if it 
isn't. Once a request has started, it is not possible to stop it (on 
downstream nodes), so opportunities are quite limited; the fact that one 
Sender can serve many Handlers or none makes it difficult to push the stop 
button.
- To get the correct byte counts.
This is really important, because it underlies the major load limiting 
systems. However it is only important for terminal messages: we can 
reasonably assume the rest will be piggybacked if necessary.

In RequestHandler, most of the sendSync()'s are terminal, and therefore must 
remain. The one that is the most interesting is the one you mentioned above: 
sending the Accepted message. 

One reason to keep it as-is is to deal with nodes connecting, sending a load 
of requests, and then disconnecting, as a DoS attack. This is implausible as 
a sensible the load limiting system won't let them start enough requests to 
be a problem given the relative difficulty of getting new connections. Of 
course, getting new connections is likely to be rather easy on opennet...

Another reason is if a node or a link is hopelessly overloaded, we don't want 
to run a request which will not serve any purpose because the node will 
timeout on us and route it to another node. This isn't really dealt with by 
the current code either.

So it looks like your patch is worthwhile...

You can use the same logic on all the Handler's and Sender's. Either send me a 
new patch, or send me your preferred username, password and email address so 
I can give you an SVN account and you can commit it yourself.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071202/a95cea10/attachment.pgp 

From m.rogers at cs.ucl.ac.uk  Sun Dec  2 00:25:57 2007
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Sun, 02 Dec 2007 00:25:57 +0000
Subject: [freenet-dev] Why Freenet is so SLOW! Proposal for new load
 management
In-Reply-To: <200712011618.30874.toad@amphibian.dyndns.org>
References: <200711300126.45810.toad@amphibian.dyndns.org>
	<200711301859.58824.toad@amphibian.dyndns.org>
	<47509E27.6090303@cs.ucl.ac.uk>
	<200712011618.30874.toad@amphibian.dyndns.org>
Message-ID: <4751FB95.2030802@cs.ucl.ac.uk>

Matthew Toseland wrote:
> We do. The link layer uses a protocol which behaves more or less the same as 
> TCP with regard to congestion control.

Maybe this has been fixed in 1084, but a couple of days ago my node was
trying to send 107 KB/s through a 16 KB/s pipe for long enough to kill
all the TCP connections sharing the pipe. I understand that was due to a
bug at the request layer, but it shows that congestion control at the
message layer currently isn't working: we're depending on the request
layer to do both jobs.

> IMHO we would still need rejection, for several reasons:
> 1. Reject due to loop.

Yeah, of course. :-)

> 2. Reject due to overload if something unexpected happens.

I'm not sure what you have in mind...

> 3. Reject due to overload as a normal part of the operation of the node 
> because if we just send one token to one node it will not always send a 
> request, so IMHO we have to send tokens to several nodes and then reject some 
> if we get more requests than we'd expected.

If we still need to reject requests pre-emptively then what's the
advantage of tokens over the current system? But on the other hand I see
your point, it's possible that several peers will spend their tokens at
once.

How about this: instead of tokens, we give each peer an enforced delay
between requests. If the load is light, we gradually decrease the delay.
If the load is heavy, we increase the delay. Like token buckets, we can
enforce fairness or other policies by giving different peers different
delays.

Unlike stop/start signals or tokens it smooths out bursts, because a
peer can't save up delay: if there's a long delay between requests A and
B, you still have to wait the full period between B and C.

What happens if a request arrives early? Maybe the peer's ignoring the
delay or maybe it was network jitter. Keep the peer's requests in a
queue and process them at the proper time... if the queue gets more than
about one second long, something's wrong - drop the connection and warn
the user.

What do you reckon?

Cheers,
Michael


From nextgens at freenetproject.org  Sun Dec  2 14:52:51 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=)
Date: Sun, 2 Dec 2007 15:52:51 +0100
Subject: [freenet-dev] Freenet build 1085
Message-ID: <20071202145250.GC4369@freenetproject.org>

Freenet build 1085 is out. It fixes an important bug, hence it is
self-mandatory tonight at midnight.

Your node should auto-update soon, if it doesn't before tomorrow,
force a manual update.

NextGen$
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071202/54d0b22b/attachment.pgp 

From batosai at batosai.net  Sun Dec  2 17:26:35 2007
From: batosai at batosai.net (Julien Cornuwel aka batosai)
Date: Sun, 02 Dec 2007 18:26:35 +0100
Subject: [freenet-dev] Updated french translation for 1085
Message-ID: <4752EACB.1060004@batosai.net>


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: freenet.l10n.fr.override.properties
Url: http://emu.freenetproject.org/pipermail/devl/attachments/20071202/f129b444/attachment.txt 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071202/f129b444/attachment.pgp 

From liquido at mail.com  Sun Dec  2 18:55:57 2007
From: liquido at mail.com (Cooo heter)
Date: Sun, 2 Dec 2007 13:55:57 -0500
Subject: [freenet-dev] Swedish translation V.7 - Cooo
Message-ID: <20071202185557.9F9C916427D@ws1-4.us4.outblaze.com>

 Swedish translation V.7 - Cooo

Another update to the swedish translation.
Mostly new translations, some of them might be updates to old ones.

HTMLFilter.unknownTag=ok?nd tagg ${tag}
QueueToadlet.progress=Utveckling
QueueToadlet.insertFileInsertFileLabel=Ladda upp fil
QueueToadlet.insertFileLabel=Fil
QueueToadlet.insertFileBrowseLabel=Bl?ddra
QueueToadlet.insertFileCompressLabel=Komprimera
Toadlet.nodeHomepage=Startsidan
IPDetectorPluginManager.direct=Du verkar vara direkt ansluten till
internet. Anslutningar till andra Freenet noder b?r ske utan n?gra
problem.
IPDetectorPluginManager.noConnectivity=Din internet anslutning verkar
inte st?dja UPD.Om inte detta ?r en felbed?mmning s? ?r det inte troligt
att du kan anv?nda Freenet.
StatisticsToadlet.fullTitle=Statistik f?r ${name}
StatisticsToadlet.statisticGatheringTitle=Statistik samling
PluginManager.pluginLoadingFailedWithMessage=${name} kunde inte laddas:
${message}
PluginManager.pluginLoadingFailedTitle=Kunde inte ladda plugin!
PluginMana ger.pluginLoadingFailed=${name} kunde inte laddas
FirstTimeWizardToadlet.connectToStrangers=Anslut till fr?mlingar?
FirstTimeWizardToadlet.continue=Forts?tt
FetchException.shortError.29=Felaktig MIME typ
UserAlertManager.alertsOnHomepage=| Mer information finns p?
${link}startsidan${/link}.
N2NTMUserAlert.title=Nod till Nod Text meddelande nummer ${number} fr?n
${peername} (${peer})
N2NTMUserAlert.header=Avs?ndare : ${from} | Skrivet : ${composed} |
Skickat : ${sent} | Mottaget : ${received}
ConnectivityToadlet.noreply=INGET SVAR
ConnectivityToadlet.addressTitle=Adress
ConnectivityToadlet.sentReceivedTitle=Skickade / Mottagna paket
ConnectivityToadlet.local=LOKAL
IPUndetectedUserAlert.suggestForwardTwoPorts=Det ?r ocks? en god ide att
vidarebefodra portarna ${port1} och ${port2} (UDP) i din router f?r att
g?ra det l?tt att ansluta till din nod.
IPUndetectedUserAlert.suggestForwardPort=Det ?r ocks? en god ide att
vidarebefodra porten ${port} (UDP) i din router f?r att g?ra det l?tt att
ansluta till din nod.
NodeUpdateManager.updateCatastropheTitle=Katastrofalt fel vid
uppdatering!
BookmarkItem.deleteBookmarkUpdateNotification=Radera meddelande
BookmarkItem.unnamedBookmark=Namnl?st bokm?rke
ConfigToadlet.pluginmanager=Pluginhanterare
ConfigToadlet.contributeTranslation=Bidra till ?vers?ttningen
ConfigToadlet.pluginmanager2=pluginhanterare2
ConfigToadlet.logger=loggning
ConfigToadlet.false=nej
ConfigToadlet.true=ja
FileOffer.succeededReceiveTitle=Filen togs emot utan problem
FileOffer.rejectTransferButton=Avvisa ?verf?ring
FProxyToadlet.backToFProxy=${link}Klicka h?r${/link} f?r att g? till
FProxys startsida.
FProxyToadlet.expectedMimeType=F?rv?ntad MIME typ: ${mime}
FProxyToadlet.opennetTitle=Fr?mlingar
FProxyToadlet.welcomeTitle=Start
FProxyToadlet.welcome=Startsida
FProxyToadlet.abortToHomepage=Avbryt och ?terv?nd till FProxys startsida
PproxyToadlet.refreshOnStartup=Ladda om fr?n servern vid uppstart
PproxyToadlet.startingPluginsTitle=Startar Plugin
PproxyToadlet.startingPluginStatus=Status
PproxyToadlet.Load=Starta
PproxyToadlet.changeReloadOnStartup=?ndra
PproxyToadlet.reloadOnStartupShort=Starta varje g?ng Freenet startas?
PproxyToadlet.reloadPluginTitle=Ladda om Plugin
PproxyToadlet.pluginStopping=Pluginen stoppas
PproxyToadlet.startingPluginName=Plugin namn
PproxyToadlet.unauthorized=Du har inte beh?righet att bes?ka den h?r
sidan.
PproxyToadlet.loadOfficialPlugin=Starta en officiell plugin
PproxyToadlet.Error=Fel
PproxyToadlet.startingPluginStatus.starting=startar
PproxyToadlet.startingPluginStatus.downloading=Laddar ner
PeerManagerUserAlert.connErrorTitle=Vissa noder kan inte ansluta
SimpleToadletServer.cannotChangePortOnTheFly=Kan inte ?ndra FProxys port
nummer n?r noden ?r ig?ng
Node.maxOpennetPeersMustBeTwentyOrLess=F?r ej vara ?ver 20
Node.maxOpennetPeers=Maximalt antal Opennet noder
Node.maxOpennetPeersLong=Maximalt antal Opennet noder (ska vara mellan 0
till och med 20)
Node.oneConnectionPerIP=Begr?nsa till en anslutning per adress?
BookmarkManager.malformedBookmark=Felaktigt bokm?rke
ConnectionsToadlet.nodeStatus.LISTENING=LYSSNAR
ConnectionsToadlet.nodeStatus.CLOCK PROBLEM=KLOCK PROBLEM
ConnectionsToadlet.nodeStatus.DISCONNECTED=ICKE ANSLUTEN
ConnectionsToadlet.nodeStatus.TOO OLD=F?R GAMMAL
ConnectionsToadlet.nodeStatus.UNKNOWN STATUS=OK?ND STATUS
ConnectionsToadlet.nodeStatus.CONNECTED=ANSLUTEN
ConnectionsToadlet.nodeStatus.NEVER CONNECTED=ALDRIG ANSLUTEN
ConnectionsToadlet.nodeStatus.CONNECTION ERROR=ANSLUTNINGS FEL
ConnectionsToadlet.nodeStatus.LISTEN ONLY=ENDAST LYSSNA
ConnectionsToadlet.nodeStatus.BUSY=UPPTAGEN
ConnectionsToadlet.nodeStatus.BACKED OFF=UPPTAGEN
ConnectionsToadlet.nodeStatus.TOO NEW=F?R NY
UpdatedVersionAvailableUserAlert.title=En ny stabil version av Freenet
finns tillg?nglig
UpdatedVersionAvailableUserAlert.fetchingNewNode=Noden laddar nu ner en
ny version av Freenet (nod version ${nodeVersion}).
UpdatedVersionAvailableUserAlert.alsoDownloadedNewExtJar=Noden har ocks?
laddat ner en ny version av Freenet extra jar, version ${version}
WelcomeToadlet.update=Uppdatera
WelcomeToadlet.shutdownConfirm=?r det s?kert att du vill st?nga av noden?
WelcomeToadlet.privateKeyHeader=Privat nyckel
WelcomeToadlet.shutdownNode=St?ng av noden
WelcomeToadlet.subjectHeader=?mne
WelcomeToadlet.homepageFullTitleWithName=Freenet FProxys startsida f?r
${name}
WelcomeToadlet.thanks=Tack f?r att du anv?nder Freenet.
WelcomeToadlet.restarting=V?nligen v?nta medans Freenet startas om, detta
kan ta upp till 3 minuter.Tack f?r att du anv?nder Freenet.
WelcomeToadlet.shutdownConfirmTitle=Nod avst?ngning
WelcomeToadlet.keyRequestLabel=Nyckel:
WelcomeToadlet.publicKeyHeader=Publik nyckel
WelcomeToadlet.versionHeader=Versions information & nod kontrol
WelcomeToadlet.shutdownDone=Freenet noden st?ngs ner.
WelcomeToadlet.shutdown=St?ng av
WelcomeToadlet.restart=Starta om
WelcomeToadlet.testnetWarning=K?r i testnet l?ge. Detta kommer allvarligt
att ?ventyra din anonymitet!
WelcomeToadlet.updatingTitle=Noden uppdateras
WelcomeToadlet.restartingTitle=Freenet noden startas om.
WelcomeToadlet.restartNode=Starta om noden
WelcomeToadlet.restartConfirm=?r det s?kert att du vill starta om noden?
WelcomeToadlet.restartConfirmTitle=Nod omstart
WelcomeToadlet.nodeUpdateConfirmTitle=Godk?nn nod uppdatering
WelcomeToadlet.nodeUpdateConfirm=Vill du uppdatera Freenet noden?
WelcomeToadlet.messageHeader=Meddelande
WelcomeToadlet.threadDumpTitle=Skapa en tr?d dump
PluginToadlet.pluginListTitle=Startade plugins
BookmarkEditorToadlet.descLabel=Beskrivning:
BookmarkEditorToadlet.invalidKey=Freenet nyckeln ?r ogiltig.
End

-- 
Over 2 Million Holiday Gift Ideas - Take a Look!
mail.com shopping at  http://mail.shopping.com/?linkin_id=8033174

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://emu.freenetproject.org/pipermail/devl/attachments/20071202/d2a1b815/attachment.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freenet.l10n.se.override.properties.0.7-Cooo
Type: application/octet-stream
Size: 6552 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071202/d2a1b815/attachment.obj 

From nextgens at freenetproject.org  Sun Dec  2 19:58:00 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=)
Date: Sun, 2 Dec 2007 20:58:00 +0100
Subject: [freenet-dev] Swedish translation V.7 - Cooo
In-Reply-To: <20071202185557.9F9C916427D@ws1-4.us4.outblaze.com>
References: <20071202185557.9F9C916427D@ws1-4.us4.outblaze.com>
Message-ID: <20071202195759.GD4369@freenetproject.org>

* Cooo heter <liquido at mail.com> [2007-12-02 13:55:57]:

>  Swedish translation V.7 - Cooo
> 
> Another update to the swedish translation.
> Mostly new translations, some of them might be updates to old ones.
> 

Commited in r16208
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071202/cc5e8d86/attachment.pgp 

From luke771 at gmail.com  Mon Dec  3 11:27:26 2007
From: luke771 at gmail.com (luke771)
Date: Mon, 03 Dec 2007 12:27:26 +0100
Subject: [freenet-dev] ITA l10n 071203_ph00
Message-ID: <4753E81E.4050102@gmail.com>

- Some rewordings to complete and make more accurate some translations, 
and also make things easier to understand, expecially in long strings 
with technical explanations.

=> One mess sorted out: part of the First Time Wizard section was "out 
of synch" (some strings would show each other's translation in the IT 
version)

- Several typos corrected


_ph00

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: freenet.l10n.it.override.properties_071203_ph00
Url: http://emu.freenetproject.org/pipermail/devl/attachments/20071203/0b9e2b2f/attachment.txt 

From toad at amphibian.dyndns.org  Mon Dec  3 12:17:11 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 3 Dec 2007 12:17:11 +0000
Subject: [freenet-dev] [freenet-cvs] r16205 -
	trunk/freenet/src/freenet/node
In-Reply-To: <20071202141659.AB0494797DF@freenetproject.org>
References: <20071202141659.AB0494797DF@freenetproject.org>
Message-ID: <200712031217.17983.toad@amphibian.dyndns.org>

That's one of nextgens' netbeans auto-indents, are you sure it's real? If it 
is, we may have a lot more similar problems...

On Sunday 02 December 2007 14:16, nextgens at freenetproject.org wrote:
> Author: nextgens
> Date: 2007-12-02 14:16:59 +0000 (Sun, 02 Dec 2007)
> New Revision: 16205
> 
> Modified:
>    trunk/freenet/src/freenet/node/PeerManager.java
>    trunk/freenet/src/freenet/node/PeerNode.java
> Log:
> Fix a variable shadowing in PeerManager and a typo preventing version 
informations from beeing parsed.
> 
> Modified: trunk/freenet/src/freenet/node/PeerNode.java
> ===================================================================
> --- trunk/freenet/src/freenet/node/PeerNode.java	2007-12-02 11:40:07 UTC 
(rev 16204)
> +++ trunk/freenet/src/freenet/node/PeerNode.java	2007-12-02 14:16:59 UTC 
(rev 16205)
> @@ -1556,7 +1556,7 @@
>  			node.peers.addConnectedPeer(this);
>  			onConnect();
>  		}
> -
> +		
>  		setPeerNodeStatus(now);
>  		return true;
>  	}
> @@ -1798,18 +1798,16 @@
>  			throw new FSParseException(err);
>  		}
>  		String newVersion = fs.get("version");
> -		if(newVersion == null)
> +		if(newVersion == null) {
>  			// Version may be ommitted for an ARK.
> -
>  			if(!forARK)
>  				throw new FSParseException("No version");
> -		else {
> +		} else {
>  			if(!newVersion.equals(version))
>  				changedAnything = true;
>  			version = newVersion;
>  			Version.seenVersion(newVersion);
>  		}
> -
>  		lastGoodVersion = fs.get("lastGoodVersion");
>  
>  		updateShouldDisconnectNow();
> 
> _______________________________________________
> cvs mailing list
> cvs at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/cvs
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071203/06d61553/attachment.pgp 

From toad at amphibian.dyndns.org  Mon Dec  3 12:23:13 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 3 Dec 2007 12:23:13 +0000
Subject: [freenet-dev] Why Freenet is so SLOW! Proposal for new load
	management
In-Reply-To: <4751FB95.2030802@cs.ucl.ac.uk>
References: <200711300126.45810.toad@amphibian.dyndns.org>
	<200712011618.30874.toad@amphibian.dyndns.org>
	<4751FB95.2030802@cs.ucl.ac.uk>
Message-ID: <200712031223.14490.toad@amphibian.dyndns.org>

On Sunday 02 December 2007 00:25, Michael Rogers wrote:
> Matthew Toseland wrote:
> > We do. The link layer uses a protocol which behaves more or less the same 
as 
> > TCP with regard to congestion control.
> 
> Maybe this has been fixed in 1084, but a couple of days ago my node was
> trying to send 107 KB/s through a 16 KB/s pipe for long enough to kill
> all the TCP connections sharing the pipe. I understand that was due to a
> bug at the request layer, but it shows that congestion control at the
> message layer currently isn't working: we're depending on the request
> layer to do both jobs.

Well... yes and no. It was a bug at the LINK layer iirc. Remember the 
pathetically low payload percentages?
> 
> > IMHO we would still need rejection, for several reasons:
> > 1. Reject due to loop.
> 
> Yeah, of course. :-)
> 
> > 2. Reject due to overload if something unexpected happens.
> 
> I'm not sure what you have in mind...
> 
> > 3. Reject due to overload as a normal part of the operation of the node 
> > because if we just send one token to one node it will not always send a 
> > request, so IMHO we have to send tokens to several nodes and then reject 
some 
> > if we get more requests than we'd expected.
> 
> If we still need to reject requests pre-emptively then what's the
> advantage of tokens over the current system? But on the other hand I see
> your point, it's possible that several peers will spend their tokens at
> once.

The point is we can't have large numbers of tokens accumulate on a peer: this 
is very difficult to manage. Therefore, we must send short-lived tokens to 
multiple peers whenever there's an opportunity to do a request.
> 
> How about this: instead of tokens, we give each peer an enforced delay
> between requests. If the load is light, we gradually decrease the delay.
> If the load is heavy, we increase the delay. Like token buckets, we can
> enforce fairness or other policies by giving different peers different
> delays.

We tried this in 0.5...
> 
> Unlike stop/start signals or tokens it smooths out bursts, because a
> peer can't save up delay: if there's a long delay between requests A and
> B, you still have to wait the full period between B and C.

I'm not talking about stop/start signals, nor am I talking about tokens in the 
sense that you use the word.
> 
> What happens if a request arrives early? 

If it arrives early, if we've already accepted a different request, etc etc, 
we simply reject it with overload. Then it remains queued on the sender.

> Maybe the peer's ignoring the 
> delay or maybe it was network jitter. Keep the peer's requests in a
> queue and process them at the proper time... if the queue gets more than
> about one second long, something's wrong - drop the connection and warn
> the user.

The proposal already queues the requests, but we need to control the rate at 
which requests enter the queue.
> 
> What do you reckon?
> 
> Cheers,
> Michael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071203/1ef54260/attachment.pgp 

From nextgens at freenetproject.org  Mon Dec  3 13:01:31 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=)
Date: Mon, 3 Dec 2007 14:01:31 +0100
Subject: [freenet-dev] ITA l10n 071203_ph00
In-Reply-To: <4753E81E.4050102@gmail.com>
References: <4753E81E.4050102@gmail.com>
Message-ID: <20071203130130.GD4285@freenetproject.org>

* luke771 <luke771 at gmail.com> [2007-12-03 12:27:26]:

> - Some rewordings to complete and make more accurate some translations, and 
> also make things easier to understand, expecially in long strings with 
> technical explanations.
>
> => One mess sorted out: part of the First Time Wizard section was "out of 
> synch" (some strings would show each other's translation in the IT version)
>
> - Several typos corrected
>
> _ph00

Commited in r16219
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071203/fbf52455/attachment.pgp 

From toad at amphibian.dyndns.org  Mon Dec  3 14:24:07 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 3 Dec 2007 14:24:07 +0000
Subject: [freenet-dev] [freenet-cvs] r16220 -
	trunk/freenet/src/freenet/support
In-Reply-To: <20071203141944.0373E479871@freenetproject.org>
References: <20071203141944.0373E479871@freenetproject.org>
Message-ID: <200712031424.07941.toad@amphibian.dyndns.org>

1L* ? That's also redundant, just put the L on the 1000, or the 7.

On Monday 03 December 2007 14:19, nextgens at freenetproject.org wrote:
> Author: nextgens
> Date: 2007-12-03 14:19:43 +0000 (Mon, 03 Dec 2007)
> New Revision: 16220
> 
> Modified:
>    trunk/freenet/src/freenet/support/TimeUtil.java
> Log:
> TimeUtil: remove all the casting altogether
> 
> Modified: trunk/freenet/src/freenet/support/TimeUtil.java
> ===================================================================
> --- trunk/freenet/src/freenet/support/TimeUtil.java	2007-12-03 13:08:51 UTC 
(rev 16219)
> +++ trunk/freenet/src/freenet/support/TimeUtil.java	2007-12-03 14:19:43 UTC 
(rev 16220)
> @@ -57,41 +57,41 @@
>              return sb.toString();
>          }
>          //
> -        long weeks = (l / ((long)7*24*60*60*1000));
> +        long weeks = (l / (1L*7*24*60*60*1000));
>          if (weeks > 0) {
>              sb.append(weeks).append('w');
>              termCount++;
> -            l = l - (weeks * ((long)7*24*60*60*1000));
> +            l = l - (weeks * (1L*7*24*60*60*1000));
>          }
>          if(termCount >= maxTerms) {
>              return sb.toString();
>          }
>          //
> -        long days = (l / ((long)24*60*60*1000));
> +        long days = (l / (1L*24*60*60*1000));
>          if (days > 0) {
>              sb.append(days).append('d');
>              termCount++;
> -            l = l - (days * ((long)24*60*60*1000));
> +            l = l - (days * (1L*24*60*60*1000));
>          }
>          if(termCount >= maxTerms) {
>            return sb.toString();
>          }
>          //
> -        long hours = (l / ((long)60*60*1000));
> +        long hours = (l / (1L*60*60*1000));
>          if (hours > 0) {
>              sb.append(hours).append('h');
>              termCount++;
> -            l = l - (hours * ((long)60*60*1000));
> +            l = l - (hours * (1L*60*60*1000));
>          }
>          if(termCount >= maxTerms) {
>              return sb.toString();
>          }
>          //
> -        long minutes = (l / ((long)60*1000));
> +        long minutes = (l / (1L*60*1000));
>          if (minutes > 0) {
>              sb.append(minutes).append('m');
>              termCount++;
> -            l = l - (minutes * ((long)60*1000));
> +            l = l - (minutes * (1L*60*1000));
>          }
>          if(termCount >= maxTerms) {
>              return sb.toString();
> @@ -105,7 +105,7 @@
>                  //l = l - ((long)fractionalSeconds * (long)1000);
>              }
>          } else {
> -            long seconds = (l / (long)1000);
> +            long seconds = (l / 1L*1000);
>              if (seconds > 0) {
>                  sb.append(seconds).append('s');
>                  termCount++;
> 
> _______________________________________________
> cvs mailing list
> cvs at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/cvs
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071203/e7b2df52/attachment.pgp 

From nextgens at freenetproject.org  Mon Dec  3 14:26:30 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=)
Date: Mon, 3 Dec 2007 15:26:30 +0100
Subject: [freenet-dev] [freenet-cvs] r16220
	-	trunk/freenet/src/freenet/support
In-Reply-To: <200712031424.07941.toad@amphibian.dyndns.org>
References: <20071203141944.0373E479871@freenetproject.org>
	<200712031424.07941.toad@amphibian.dyndns.org>
Message-ID: <20071203142630.GF4285@freenetproject.org>

* Matthew Toseland <toad at amphibian.dyndns.org> [2007-12-03 14:24:07]:

> 1L* ? That's also redundant, just put the L on the 1000, or the 7.
> 

Imho it's clearer code-wise... so that if someone changes the value it
*will* be clear for him that it's meant to be casted into a long.

Arguably noone will ever touch that method again.

> On Monday 03 December 2007 14:19, nextgens at freenetproject.org wrote:
> > Author: nextgens
> > Date: 2007-12-03 14:19:43 +0000 (Mon, 03 Dec 2007)
> > New Revision: 16220
> > 
> > Modified:
> >    trunk/freenet/src/freenet/support/TimeUtil.java
> > Log:
> > TimeUtil: remove all the casting altogether
> > 
> > Modified: trunk/freenet/src/freenet/support/TimeUtil.java
> > ===================================================================
> > --- trunk/freenet/src/freenet/support/TimeUtil.java	2007-12-03 13:08:51 UTC 
> (rev 16219)
> > +++ trunk/freenet/src/freenet/support/TimeUtil.java	2007-12-03 14:19:43 UTC 
> (rev 16220)
> > @@ -57,41 +57,41 @@
> >              return sb.toString();
> >          }
> >          //
> > -        long weeks = (l / ((long)7*24*60*60*1000));
> > +        long weeks = (l / (1L*7*24*60*60*1000));
> >          if (weeks > 0) {
> >              sb.append(weeks).append('w');
> >              termCount++;
> > -            l = l - (weeks * ((long)7*24*60*60*1000));
> > +            l = l - (weeks * (1L*7*24*60*60*1000));
> >          }
> >          if(termCount >= maxTerms) {
> >              return sb.toString();
> >          }
> >          //
> > -        long days = (l / ((long)24*60*60*1000));
> > +        long days = (l / (1L*24*60*60*1000));
> >          if (days > 0) {
> >              sb.append(days).append('d');
> >              termCount++;
> > -            l = l - (days * ((long)24*60*60*1000));
> > +            l = l - (days * (1L*24*60*60*1000));
> >          }
> >          if(termCount >= maxTerms) {
> >            return sb.toString();
> >          }
> >          //
> > -        long hours = (l / ((long)60*60*1000));
> > +        long hours = (l / (1L*60*60*1000));
> >          if (hours > 0) {
> >              sb.append(hours).append('h');
> >              termCount++;
> > -            l = l - (hours * ((long)60*60*1000));
> > +            l = l - (hours * (1L*60*60*1000));
> >          }
> >          if(termCount >= maxTerms) {
> >              return sb.toString();
> >          }
> >          //
> > -        long minutes = (l / ((long)60*1000));
> > +        long minutes = (l / (1L*60*1000));
> >          if (minutes > 0) {
> >              sb.append(minutes).append('m');
> >              termCount++;
> > -            l = l - (minutes * ((long)60*1000));
> > +            l = l - (minutes * (1L*60*1000));
> >          }
> >          if(termCount >= maxTerms) {
> >              return sb.toString();
> > @@ -105,7 +105,7 @@
> >                  //l = l - ((long)fractionalSeconds * (long)1000);
> >              }
> >          } else {
> > -            long seconds = (l / (long)1000);
> > +            long seconds = (l / 1L*1000);
> >              if (seconds > 0) {
> >                  sb.append(seconds).append('s');
> >                  termCount++;
> > 
> > _______________________________________________
> > cvs mailing list
> > cvs at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/cvs
> > 
> > 



> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071203/3be33f3c/attachment.pgp 

From nextgens at freenetproject.org  Mon Dec  3 16:56:46 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=)
Date: Mon, 3 Dec 2007 17:56:46 +0100
Subject: [freenet-dev] [freenet-cvs] r16205
	-	trunk/freenet/src/freenet/node
In-Reply-To: <200712031217.17983.toad@amphibian.dyndns.org>
References: <20071202141659.AB0494797DF@freenetproject.org>
	<200712031217.17983.toad@amphibian.dyndns.org>
Message-ID: <20071203165645.GH4285@freenetproject.org>

* Matthew Toseland <toad at amphibian.dyndns.org> [2007-12-03 12:17:11]:

> That's one of nextgens' netbeans auto-indents, are you sure it's real? If it 
> is, we may have a lot more similar problems...
> 

Yeah, I apologize about that, most of the new-build-spree going on
lately could have been avoided...


Yes, tt turns out that it's a real IDE problem (I've reported the issue; see
: http://www.netbeans.org/issues/show_bug.cgi?id=123258).

In the meantime, I've written a script to verify "indent-only" commits
so that we can ensure that no more mistakes due to IDE bugs are made.
Emu will run that script against any commit which message is "indent" in
the 0.7's trunk.

According to what I recall and svn log, here is the sum up of indenting
commits I've done since I've installed netbeans...
I ran the bytecode comparison script I wrote against them and here is
the result :

r16197 OK
r16178 OK
r16141 OK
r16140 OK
r16125 NOK
r16090 OK
r16077 OK
r16072 OK
r16069 OK
r16067 OK
r15884 OK

I've fixed the bugs I've introduced in r16224.

The morale of the story is "never trust your IDE, even to perform basic
tasks like indenting".

Shame on me!
	NextGen$
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071203/6cbbc745/attachment.pgp 

From m.rogers at cs.ucl.ac.uk  Mon Dec  3 17:25:52 2007
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Mon, 03 Dec 2007 17:25:52 +0000
Subject: [freenet-dev] Why Freenet is so SLOW! Proposal for new load
 management
In-Reply-To: <200712031223.14490.toad@amphibian.dyndns.org>
References: <200711300126.45810.toad@amphibian.dyndns.org>
	<200712011618.30874.toad@amphibian.dyndns.org>
	<4751FB95.2030802@cs.ucl.ac.uk>
	<200712031223.14490.toad@amphibian.dyndns.org>
Message-ID: <47543C20.2070303@cs.ucl.ac.uk>

Matthew Toseland wrote:
> Well... yes and no. It was a bug at the LINK layer iirc. Remember the 
> pathetically low payload percentages?

What you call the link layer is supposed to be congestion-controlled, right?

> I'm not talking about stop/start signals, nor am I talking about tokens in the 
> sense that you use the word.

Then whey did you say your proposal was basically the same as token passing?

So just to be clear, you're talking about tokens that expire, backed up
by pre-emptive rejection if too many tokens are spent at once?

Will the grounds for pre-emptive rejection be the same as they are now
(bandwidth liability etc)? If so, how will tokens solve the current
problem of too many requests being rejected?

Cheers,
Michael


From toad at amphibian.dyndns.org  Mon Dec  3 17:32:36 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 3 Dec 2007 17:32:36 +0000
Subject: [freenet-dev] Why Freenet is so SLOW! Proposal for new load
	management
In-Reply-To: <47543C20.2070303@cs.ucl.ac.uk>
References: <200711300126.45810.toad@amphibian.dyndns.org>
	<200712031223.14490.toad@amphibian.dyndns.org>
	<47543C20.2070303@cs.ucl.ac.uk>
Message-ID: <200712031732.41169.toad@amphibian.dyndns.org>

On Monday 03 December 2007 17:25, Michael Rogers wrote:
> Matthew Toseland wrote:
> > Well... yes and no. It was a bug at the LINK layer iirc. Remember the 
> > pathetically low payload percentages?
> 
> What you call the link layer is supposed to be congestion-controlled, right?

Yes, at the link layer. Congestion control presently only applies to data 
packets, and yes I accept that this sucks and we have plans for a new link 
layer/packet format, which is more like TCP, but they haven't yet been 
implemented.
> 
> > I'm not talking about stop/start signals, nor am I talking about tokens in 
the 
> > sense that you use the word.
> 
> Then whey did you say your proposal was basically the same as token passing?
> 
> So just to be clear, you're talking about tokens that expire, backed up
> by pre-emptive rejection if too many tokens are spent at once?

Yes.
> 
> Will the grounds for pre-emptive rejection be the same as they are now
> (bandwidth liability etc)? If so, how will tokens solve the current
> problem of too many requests being rejected?

I'm not sure that that is the main problem we have to deal with right now. My 
suspicion is that pre-emptive rejection is fine, but we're not being sent 
enough requests in the first place. Output liability limiting *may* cause too 
many requests to be rejected, but I don't see any evidence for that right 
now. And anything we could do about that would run the risk of creating 
timeouts if we have a burst of unusually successful requests.
> 
> Cheers,
> Michael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071203/3479b431/attachment.pgp 

From m.rogers at cs.ucl.ac.uk  Mon Dec  3 17:44:55 2007
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Mon, 03 Dec 2007 17:44:55 +0000
Subject: [freenet-dev] Why Freenet is so SLOW! Proposal for new load
 management
In-Reply-To: <200712031732.41169.toad@amphibian.dyndns.org>
References: <200711300126.45810.toad@amphibian.dyndns.org>
	<200712031223.14490.toad@amphibian.dyndns.org>
	<47543C20.2070303@cs.ucl.ac.uk>
	<200712031732.41169.toad@amphibian.dyndns.org>
Message-ID: <47544097.40104@cs.ucl.ac.uk>

Matthew Toseland wrote:
> I'm not sure that that is the main problem we have to deal with right now. My 
> suspicion is that pre-emptive rejection is fine, but we're not being sent 
> enough requests in the first place.

Cool, I see what you mean now - I'd been thinking in terms of replacing
both backoff and pre-emptive rejection but I guess it makes sense to
replace backoff first and see if that solves the problem.

Cheers,
Michael


From m.rogers at cs.ucl.ac.uk  Mon Dec  3 18:52:59 2007
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Mon, 03 Dec 2007 18:52:59 +0000
Subject: [freenet-dev] Short refs was Re: alternative to #freenet-refs
In-Reply-To: <474EBFC4.7030201@WhenGendarmeSleeps.org>
References: <20071107220045.GA4554@freenetproject.org>
	<1195995849.10868.24.camel@localhost>
	<474B09FD.7010906@cs.ucl.ac.uk>
	<200711271144.47979.toad@amphibian.dyndns.org>
	<474DD412.307@cs.ucl.ac.uk>
	<474EBFC4.7030201@WhenGendarmeSleeps.org>
Message-ID: <4754508B.7000606@cs.ucl.ac.uk>

Thanks! Here's the Java version of the poem generator (same dictionaries
as before).

Cheers,
Michael

Volodya wrote:
> Michael Rogers wrote:
>> Matthew Toseland wrote:
>>> Base32 this is 35 chars; the base32 may be easier, and is certainly shorter, 
>>> but this would be a nice option if you port the code to java. :)
>> Yeah, base32 would definitely be more practical but I just love the idea
>> of using random poetry to set up a darknet. :-) I'll have a crack at
>> porting it to Java when I get a minute.
>>
>> Cheers,
>> Michael
> 
> This is something to start with. It assumes to find the words in the "words-#.dic" files where # represents the
> order of the words... so its probably not as clean as you'd want it to be.
> 
> Also i think the problem would be with loading the file into the memory, when it will get large enough to be
> representing the true key i think you'll be having some serious out of memory issues.
> 
>                           - Volodya
> 
> 
> 
> ------------------------------------------------------------------------
> 
> import java.io.BufferedReader;
> import java.io.FileReader;
> 
> import java.io.FileNotFoundException;
> import java.io.IOException;
> 
> public class PoemKey {
> 	public static int WORDS_PER_KEY = 4;
> 	public static int BYTES_PER_WORD = 2;
> 	
> 	private static String[][] words;
> 	
> 	static
> 	{
> 		words = new String[WORDS_PER_KEY][];
> 		final int numOfWords = (int)Math.pow(2, BYTES_PER_WORD * 8);
> 		
> 		for(int i=0; i<WORDS_PER_KEY; i++)
> 		{
> 			words[i] = new String[numOfWords];
> 			try
> 			{
> 				BufferedReader fin = new BufferedReader(new FileReader("words-"+i+".dic"));
> 				for(int j=0; j < numOfWords; j++)
> 				{
> 					words[i][j] = fin.readLine();
> 				}
> 				fin.close();
> 			}
> 			catch(FileNotFoundException e)
> 			{
> 				//Well i don't really know what you'd want to do then.
> 			}
> 			catch(IOException e)
> 			{
> 				//Well i don't really know what you'd want to do then.
> 			}
> 		}
> 		
> 	}
> 	
> 	public static int[] poemToKey(String poem[])
> 	{
> 		int result[] = new int[WORDS_PER_KEY];
> 		
> 		return result;
> 	}
> 	
> 	public static String[] keyToPoem(int[] key)
> 	{
> 		String result[] = new String[WORDS_PER_KEY];
> 		
> 		for(int i=0; i<WORDS_PER_KEY; i++)
> 		{
> 			result[i] = words[i][key[i]];
> 		}
> 		
> 		return result;
> 	}
> 	
> 	public static void main(String args[])
> 	{
> 		int key[] = new int[WORDS_PER_KEY];
> 		
> 		System.out.print("The random key with "+WORDS_PER_KEY+" blocks "+BYTES_PER_WORD+" bytes each is: ");
> 		
> 		for(int i=0; i<WORDS_PER_KEY; i++)
> 		{
> 			key[i]=(int)(Math.random()*8*BYTES_PER_WORD);
> 			System.out.print(Integer.toHexString(key[i])+" ");
> 		}
> 		
> 		System.out.println("");
> 		
> 		System.out.print("The 'Poem' key is: ");
> 		String[] poem = keyToPoem(key);
> 		for(int i=0; i<WORDS_PER_KEY; i++)
> 		{
> 			System.out.print(poem[i]+" ");
> 		}
> 		System.out.println("");
> 	}
> }
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

-------------- next part --------------
A non-text attachment was scrubbed...
Name: BasicEnglish.java
Type: text/x-java
Size: 1845 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071203/6aedb03b/attachment.java 

From toad at amphibian.dyndns.org  Mon Dec  3 19:26:01 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 3 Dec 2007 19:26:01 +0000
Subject: [freenet-dev] Why Freenet is so SLOW! Proposal for new load
	management
In-Reply-To: <47544097.40104@cs.ucl.ac.uk>
References: <200711300126.45810.toad@amphibian.dyndns.org>
	<200712031732.41169.toad@amphibian.dyndns.org>
	<47544097.40104@cs.ucl.ac.uk>
Message-ID: <200712031926.05613.toad@amphibian.dyndns.org>

On Monday 03 December 2007 17:44, Michael Rogers wrote:
> Matthew Toseland wrote:
> > I'm not sure that that is the main problem we have to deal with right now. 
My 
> > suspicion is that pre-emptive rejection is fine, but we're not being sent 
> > enough requests in the first place.
> 
> Cool, I see what you mean now - I'd been thinking in terms of replacing
> both backoff and pre-emptive rejection but I guess it makes sense to
> replace backoff first and see if that solves the problem.

Well, the initial idea was for pre-emptive rejection to be *mostly* replaced 
by a simple limit on the number of running requests (including those queued 
awaiting a node to be routed to). I would probably add this at least to begin 
with. However, some other forms of pre-emptive rejection also lend themselves 
to easily determining that we can accept a few more requests now e.g. 
bandwidth liability can do this calculation trivially.

As you can see there are some details to work out.

Lets consider the current pre-emptive rejection code. We currently reject if:
- We are over the thread count.
We can guesstimate how many threads will be used by a request, and therefore 
determine how many requests we can accept.
- The ping time is too high.
Not obvious how we could adapt this, but we can keep it.
- The bandwidth-limited-packets average delay is too high.
Not obvious how we could adapt this, but we can keep it. This is largely 
obsoleted by bandwidth liability limiting on most nodes anyway. We could even 
get rid of it.
- Input or output bandwidth liability limits would be exceeded.
We only accept any request if we have enough spare bandwidth for one of each 
type. This is a measure against accidentally favouring specific request 
types. We can either multiply this, to get a low figure, add an average value 
based on the average request, to get an average figure, or assume past the 
first they are all SSK requests, to get a high figure. One of these options 
should be workable, if not, just go one at a time.
- The high level token buckets don't have enough space for the expected bytes 
transferred.
Trivial to adapt, subject to the same worries about request types as above. 
Again, there may be an argument for getting rid of this.
- There isn't enough memory left.
Difficult to estimate a number, but not necessary most of the time. Hard 
resource limits are one obvious reason why we might reject a request even in 
a "traditional" token passing scheme.

Note that while being able to estimate a number greater than one is useful, 
it's not essential in the new scheme: we expect to send time-limited request 
solicitations to many of our peers anyway.

What happens if we have plenty of capacity? Well, we send out tokens to all 
our peers with a largish number, and when they make a request, we send 
another RequestARequest in response. This costs nothing (after packet 
padding), as we will have to send an Accepted or Reject* in response anyway. 
But what if there isn't much demand? What if the time-limited tokens expire 
before the node wants to send a request? Well, we don't specify any specific 
timeout, we simply reject any requests it sends us if we don't have the 
capacity to take them. If a node gets a RequestARequest, it updates its token 
count. If we reject a request due to overload, the token count is reset to 
zero, and the node won't send us any more requests until we solicit some 
explicitly by another RequestARequest.

Another option would be to issue tokens in terms of bytes rather than 
requests, but this would likely cause a preference for SSKs, so is probably a 
bad idea.

Does this make sense?


However, there is no urgency as the network appears to be behaving at the 
moment. It would be worth looking into after opennet is fully working 
(opennet will be fully implemented soon).
> 
> Cheers,
> Michael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071203/06b5139b/attachment.pgp 

From toad at amphibian.dyndns.org  Mon Dec  3 19:31:50 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 3 Dec 2007 19:31:50 +0000
Subject: [freenet-dev] Disk space usage choice in the post-setup wizard
Message-ID: <200712031931.51440.toad@amphibian.dyndns.org>

Would this make the setup process less user-friendly? Is it worth it anyway?

----- Anonymous ----- 2007.10.28 - 23:43:22GMT -----

I suggest setting the default storage to 5GB and putting a real big warning in 
the installation process telling the user something like "The store is only 
5GB by default becuse Freenet needs to be able to run on really OLD machines: 
you should set this value to 25GB or more, trying to aim for several hundreds 
GB"
Insisting a bit more about this point on the website, wiki, faq, etc., would 
also be a good idea.
This would considerably extend the average life of not-hugely-popular files.

----- toad at zceUWxlSaHLmvEMnbr4RHnVfehA ----- 2007.12.03 - 15:26:13GMT -----

Would it make it more obvious if we changed it from a dropdown box to a series 
of radio buttons? So you don't need to click on it to see that you can make 
it much bigger?

----- cptn_insano at _g2YxqIynCrs2bcwLGQkr+0b544 ----- 2007.12.03 - 
17:23:29GMT -----

Just don't give it a default so the user has to pick something.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071203/225bc51a/attachment.pgp 

From freenet-devl at osndok.com  Mon Dec  3 20:09:40 2007
From: freenet-devl at osndok.com (Robert Hailey)
Date: Mon, 3 Dec 2007 14:09:40 -0600
Subject: [freenet-dev] Why Freenet is so SLOW!
In-Reply-To: <200712020002.10322.toad@amphibian.dyndns.org>
References: <200711300126.45810.toad@amphibian.dyndns.org>
	<EA219B65-8C33-49C7-A3F0-475E1B3A35AA@osndok.com>
	<8EFC03C4-519E-48A0-8463-EE77E717B9BB@osndok.com>
	<200712020002.10322.toad@amphibian.dyndns.org>
Message-ID: <F06AC1AA-2771-4475-A1A8-D7C63795B4B5@osndok.com>


On Dec 1, 2007, at 6:02 PM, Matthew Toseland wrote:

> On Friday 30 November 2007 23:03, Robert Hailey wrote:
>>
>> To illustrate my point the following patch shows an approximate 100%
>> speedup for freenet, by my crude profiling.
>
> Speedup measured how? More comments below.

Measured in average time to handle a request (i.e. the realRun()-time  
of RequestHandler), which I reason is proportional to the time it  
takes for a request at-large (propagating through the network).

> One reason to keep it as-is is to deal with nodes connecting,  
> sending a load

> of requests, and then disconnecting, as a DoS attack. This is  
> implausible as
> a sensible the load limiting system won't let them start enough  
> requests to
> be a problem given the relative difficulty of getting new  
> connections. Of
> course, getting new connections is likely to be rather easy on  
> opennet...

I agree. Although, whereas sendSync() waits up to one minute for the  
given request to clear the queue, this attack would be available  
presently... only offset by a constant amount of time. And the  
attacker would have to accept enough packets to make the node believe  
you are 'connected'.

> Another reason is if a node or a link is hopelessly overloaded, we  
> don't want
> to run a request which will not serve any purpose because the node  
> will
> timeout on us and route it to another node. This isn't really dealt  
> with by
> the current code either.

I'm not sure I understand your point, but (not being a point of  
contention) you don't have to explain it. It seems to me that this  
would HELP overloaded nodes, as we process their requests without  
waiting for the ack which is a mile down there send queue; they might  
actually get there data (once it arrives and makes it to the front of  
the send queue).

> So it looks like your patch is worthwhile...
>
> You can use the same logic on all the Handler's and Sender's. Either  
> send me a
> new patch, or send me your preferred username, password and email  
> address so
> I can give you an SVN account and you can commit it yourself.

Can do.

--
Robert Hailey



From gmaxwell at gmail.com  Mon Dec  3 20:17:36 2007
From: gmaxwell at gmail.com (Gregory Maxwell)
Date: Mon, 3 Dec 2007 15:17:36 -0500
Subject: [freenet-dev] Disk space usage choice in the post-setup wizard
In-Reply-To: <200712031931.51440.toad@amphibian.dyndns.org>
References: <200712031931.51440.toad@amphibian.dyndns.org>
Message-ID: <e692861c0712031217w3e4b1bd8j6d824aabbfa63829@mail.gmail.com>

Eh.. The default value is powerful. Why not suggest a default based on
free disk space?

On Dec 3, 2007 2:31 PM, Matthew Toseland <toad at amphibian.dyndns.org> wrote:
> Would this make the setup process less user-friendly? Is it worth it anyway?
>
> ----- Anonymous ----- 2007.10.28 - 23:43:22GMT -----
>
> I suggest setting the default storage to 5GB and putting a real big warning in
> the installation process telling the user something like "The store is only
> 5GB by default becuse Freenet needs to be able to run on really OLD machines:
> you should set this value to 25GB or more, trying to aim for several hundreds
> GB"
> Insisting a bit more about this point on the website, wiki, faq, etc., would
> also be a good idea.
> This would considerably extend the average life of not-hugely-popular files.
>
> ----- toad at zceUWxlSaHLmvEMnbr4RHnVfehA ----- 2007.12.03 - 15:26:13GMT -----
>
> Would it make it more obvious if we changed it from a dropdown box to a series
> of radio buttons? So you don't need to click on it to see that you can make
> it much bigger?
>
> ----- cptn_insano at _g2YxqIynCrs2bcwLGQkr+0b544 ----- 2007.12.03 -
> 17:23:29GMT -----
>
> Just don't give it a default so the user has to pick something.
>
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>


From toad at amphibian.dyndns.org  Mon Dec  3 21:42:30 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 3 Dec 2007 21:42:30 +0000
Subject: [freenet-dev] Why Freenet is so SLOW!
In-Reply-To: <F06AC1AA-2771-4475-A1A8-D7C63795B4B5@osndok.com>
References: <200711300126.45810.toad@amphibian.dyndns.org>
	<200712020002.10322.toad@amphibian.dyndns.org>
	<F06AC1AA-2771-4475-A1A8-D7C63795B4B5@osndok.com>
Message-ID: <200712032142.35501.toad@amphibian.dyndns.org>

On Monday 03 December 2007 20:09, Robert Hailey wrote:
> 
> On Dec 1, 2007, at 6:02 PM, Matthew Toseland wrote:
> 
> > On Friday 30 November 2007 23:03, Robert Hailey wrote:
> >>
> >> To illustrate my point the following patch shows an approximate 100%
> >> speedup for freenet, by my crude profiling.
> >
> > Speedup measured how? More comments below.
> 
> Measured in average time to handle a request (i.e. the realRun()-time  
> of RequestHandler), which I reason is proportional to the time it  
> takes for a request at-large (propagating through the network).

Not necessarily, but it looks like the Accepted round-trip is a big part of 
the total request time yeah. *Very* nice catch!
> 
> > One reason to keep it as-is is to deal with nodes connecting,  
> > sending a load
> > of requests, and then disconnecting, as a DoS attack. This is  
> > implausible as
> > a sensible the load limiting system won't let them start enough  
> > requests to
> > be a problem given the relative difficulty of getting new  
> > connections. Of
> > course, getting new connections is likely to be rather easy on  
> > opennet...
> 
> I agree. Although, whereas sendSync() waits up to one minute for the  
> given request to clear the queue, this attack would be available  
> presently... only offset by a constant amount of time. 

Well, without the sendSync(), the attacker doesn't have to ack the packet, 
which saves him some bandwidth.

> And the   
> attacker would have to accept enough packets to make the node believe  
> you are 'connected'.

Okay, on further research...

If an attacker connects to a node and starts sending requests, he can stay 
connected forever simply by keeping on sending packets. He can quickly gag 
the node simply by never acknowledging any packet, but the node will continue 
to accept and propagate his requests subject to global limits. Furthermore, 
the node will leak memory: all the packets that would have been sent to the 
attacker get queued in RAM. Eventually the node will OOM, and be restarted by 
the wrapper. Once it loads back up, it will start accepting requests all over 
again. Ouch. Fixed (or greatly ameliorated) in trunk by amongst other things 
rejecting requests when the queue to the node is over 4MB, or when it would 
take over 10 minutes to transfer at current throttle rates. With that fix, 
the attack we were talking about is very much reduced.
> 
> > Another reason is if a node or a link is hopelessly overloaded, we  
> > don't want
> > to run a request which will not serve any purpose because the node  
> > will
> > timeout on us and route it to another node. This isn't really dealt  
> > with by
> > the current code either.
> 
> I'm not sure I understand your point, but (not being a point of  
> contention) you don't have to explain it. It seems to me that this  
> would HELP overloaded nodes, as we process their requests without  
> waiting for the ack which is a mile down there send queue; they might  
> actually get there data (once it arrives and makes it to the front of  
> the send queue).

If a request comes in, and the node which sent the request is so hopelessly 
overloaded that it will miss our Accepted and timeout on us, and send it to 
another node, then it's not very useful to do the request. Actually that's 
not true: if it goes to another node, it will get coalesced, or it will get 
the data that we would have sent to it. So it's not a big deal.
> 
> > So it looks like your patch is worthwhile...
> >
> > You can use the same logic on all the Handler's and Sender's. Either  
> > send me a
> > new patch, or send me your preferred username, password and email  
> > address so
> > I can give you an SVN account and you can commit it yourself.
> 
> Can do.

I look forward to your commit!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071203/f3b567d4/attachment.pgp 

From m.rogers at cs.ucl.ac.uk  Mon Dec  3 23:43:09 2007
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Mon, 03 Dec 2007 23:43:09 +0000
Subject: [freenet-dev] Short refs was Re: alternative to #freenet-refs
In-Reply-To: <474EBFC4.7030201@WhenGendarmeSleeps.org>
References: <20071107220045.GA4554@freenetproject.org>
	<1195995849.10868.24.camel@localhost>
	<474B09FD.7010906@cs.ucl.ac.uk>
	<200711271144.47979.toad@amphibian.dyndns.org>
	<474DD412.307@cs.ucl.ac.uk>
	<474EBFC4.7030201@WhenGendarmeSleeps.org>
Message-ID: <4754948D.5060400@cs.ucl.ac.uk>

Argh, same mistake again, replace "rand (1)" with "rand (2)" in the Java
version.  =0)

Volodya wrote:
> Michael Rogers wrote:
>> Matthew Toseland wrote:
>>> Base32 this is 35 chars; the base32 may be easier, and is certainly shorter, 
>>> but this would be a nice option if you port the code to java. :)
>> Yeah, base32 would definitely be more practical but I just love the idea
>> of using random poetry to set up a darknet. :-) I'll have a crack at
>> porting it to Java when I get a minute.
>>
>> Cheers,
>> Michael
> 
> This is something to start with. It assumes to find the words in the "words-#.dic" files where # represents the
> order of the words... so its probably not as clean as you'd want it to be.
> 
> Also i think the problem would be with loading the file into the memory, when it will get large enough to be
> representing the true key i think you'll be having some serious out of memory issues.
> 
>                           - Volodya


From toad at amphibian.dyndns.org  Tue Dec  4 00:00:50 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Tue, 4 Dec 2007 00:00:50 +0000
Subject: [freenet-dev] [freenet-cvs] r16249 -
	trunk/freenet/src/freenet/node
In-Reply-To: <20071203234539.1F2F14798BD@freenetproject.org>
References: <20071203234539.1F2F14798BD@freenetproject.org>
Message-ID: <200712040000.50645.toad@amphibian.dyndns.org>

That should probably be two commits, since the two changes are unrelated.

On Monday 03 December 2007 23:45, you wrote:
> Author: robert
> Date: 2007-12-03 23:45:38 +0000 (Mon, 03 Dec 2007)
> New Revision: 16249
> 
> Modified:
>    trunk/freenet/src/freenet/node/PeerNode.java
>    trunk/freenet/src/freenet/node/RequestHandler.java
> Log:
> Fix synchronization target, remove redundant connection check.
> 
> 
> Modified: trunk/freenet/src/freenet/node/PeerNode.java
> ===================================================================
> --- trunk/freenet/src/freenet/node/PeerNode.java	2007-12-03 23:17:43 UTC 
(rev 16248)
> +++ trunk/freenet/src/freenet/node/PeerNode.java	2007-12-03 23:45:38 UTC 
(rev 16249)
> @@ -872,8 +872,11 @@
>  	public void sendAsync(Message msg, AsyncMessageCallback cb, int 
alreadyReportedBytes, ByteCounter ctr) throws NotConnectedException {
>  		if(logMINOR)
>  			Logger.minor(this, "Sending async: " + msg + " : " + cb + " on " + 
this);
> -		if(!isConnected())
> +		if(!isConnected()) {
> +            if (cb instanceof SyncMessageCallback)
> +                Logger.error(this, "Tried to send " + msg + " but not 
connected to " + this, new Exception("debug"));
>  			throw new NotConnectedException();
> +        }
>  		addToLocalNodeSentMessagesToStatistic(msg);
>  		MessageItem item = new MessageItem(msg, cb == null ? null : new 
AsyncMessageCallback[]{cb}, alreadyReportedBytes, ctr);
>  		item.getData(this);
> @@ -1255,15 +1258,9 @@
>  	}
>  
>  	/**
> -	* Send a message, right now, on this thread, to this node.
> +	* Enqueue a message to be sent to this node and wait up to a minute for it 
to be transmitted.
>  	*/
>  	public void sendSync(Message req, ByteCounter ctr) throws 
NotConnectedException {
> -		synchronized(this) {
> -			if(!isConnected()) {
> -				Logger.error(this, "Tried to send " + req + " but not connected to " + 
this, new Exception("debug"));
> -				throw new NotConnectedException();
> -			}
> -		}
>  		SyncMessageCallback cb = new SyncMessageCallback();
>  		sendAsync(req, cb, 0, ctr);
>  		cb.waitForSend(60 * 1000);
> 
> Modified: trunk/freenet/src/freenet/node/RequestHandler.java
> ===================================================================
> --- trunk/freenet/src/freenet/node/RequestHandler.java	2007-12-03 23:17:43 
UTC (rev 16248)
> +++ trunk/freenet/src/freenet/node/RequestHandler.java	2007-12-03 23:45:38 
UTC (rev 16249)
> @@ -85,7 +85,7 @@
>              if((!finalTransferFailed) && rs != null && status != 
RequestSender.TIMED_OUT && status != 
RequestSender.GENERATED_REJECTED_OVERLOAD 
>              		&& status != RequestSender.INTERNAL_ERROR && !thrown) {
>              	int sent, rcvd;
> -            	synchronized(this) {
> +            	synchronized(bytesSync) {
>              		sent = sentBytes;
>              		rcvd = receivedBytes;
>              	}
> 
> _______________________________________________
> cvs mailing list
> cvs at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/cvs
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071204/297bb8ed/attachment.pgp 

From toad at amphibian.dyndns.org  Tue Dec  4 00:05:47 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Tue, 4 Dec 2007 00:05:47 +0000
Subject: [freenet-dev] Disk space usage choice in the post-setup wizard
In-Reply-To: <e692861c0712031217w3e4b1bd8j6d824aabbfa63829@mail.gmail.com>
References: <200712031931.51440.toad@amphibian.dyndns.org>
	<e692861c0712031217w3e4b1bd8j6d824aabbfa63829@mail.gmail.com>
Message-ID: <200712040005.48180.toad@amphibian.dyndns.org>

On Monday 03 December 2007 20:17, Gregory Maxwell wrote:
> Eh.. The default value is powerful. Why not suggest a default based on
> free disk space?

It is possible to do this with java 1.6, but not before. Maybe we could use 
reflection to attempt it.
> 
> On Dec 3, 2007 2:31 PM, Matthew Toseland <toad at amphibian.dyndns.org> wrote:
> > Would this make the setup process less user-friendly? Is it worth it 
anyway?
> >
> > ----- Anonymous ----- 2007.10.28 - 23:43:22GMT -----
> >
> > I suggest setting the default storage to 5GB and putting a real big 
warning in
> > the installation process telling the user something like "The store is 
only
> > 5GB by default becuse Freenet needs to be able to run on really OLD 
machines:
> > you should set this value to 25GB or more, trying to aim for several 
hundreds
> > GB"
> > Insisting a bit more about this point on the website, wiki, faq, etc., 
would
> > also be a good idea.
> > This would considerably extend the average life of not-hugely-popular 
files.
> >
> > ----- toad at zceUWxlSaHLmvEMnbr4RHnVfehA ----- 2007.12.03 - 
15:26:13GMT -----
> >
> > Would it make it more obvious if we changed it from a dropdown box to a 
series
> > of radio buttons? So you don't need to click on it to see that you can 
make
> > it much bigger?
> >
> > ----- cptn_insano at _g2YxqIynCrs2bcwLGQkr+0b544 ----- 2007.12.03 -
> > 17:23:29GMT -----
> >
> > Just don't give it a default so the user has to pick something.
> >
> > _______________________________________________
> > Devl mailing list
> > Devl at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> >
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071204/431c26c4/attachment.pgp 

From toad at amphibian.dyndns.org  Tue Dec  4 13:27:59 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Tue, 4 Dec 2007 13:27:59 +0000
Subject: [freenet-dev] Freenet 0.7 build 1087
Message-ID: <200712041328.06601.toad@amphibian.dyndns.org>

Freenet 0.7 build 1087 is now available. This will be mandatory at 0:00 on 
Thursday, not because of any critical bugfix but because of a performance 
enhancement that we need network-wide. Please upgrade. The main changes in 
this build are:
- A significant reduction in per-hop latency found by our new developer, 
Robert Hailey. Should be around a 50% cut in per hop latency, but I don't 
expect any long term gain in throughput.
- Prevent a nasty memory/bandwidth denial of service attack: Don't let a node 
make requests faster than it accepts the data resulting from it.
- Slightly more accurate request bytes usage statistics, which should mean 
slightly more accurate load limiting.
- Don't keep old queued messages around (dump them after an explicit 
disconnect, an hour of being disconnected, etc).
- Clarifications in the first-time wizard.

Please let us know if you find any bugs, via the bug tracker, the mailing 
lists or Frost.

Thanks to:
juiceman
nextgens
robert
toad (me)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071204/a16b7923/attachment.pgp 

From batosai at batosai.net  Tue Dec  4 17:10:05 2007
From: batosai at batosai.net (Julien Cornuwel)
Date: Tue, 04 Dec 2007 18:10:05 +0100
Subject: [freenet-dev] French translation update for 1087
Message-ID: <475589ED.1060608@batosai.net>

Hi,

This translation file features several new strings and numerous fixes of
chains that had been modified in the english version.

Regards.
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: freenet.l10n.fr.override.properties
Url: http://emu.freenetproject.org/pipermail/devl/attachments/20071204/dd96a60b/attachment.txt 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 250 bytes
Desc: OpenPGP digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071204/dd96a60b/attachment.pgp 

From nextgens at freenetproject.org  Tue Dec  4 17:14:19 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=)
Date: Tue, 4 Dec 2007 18:14:19 +0100
Subject: [freenet-dev] French translation update for 1087
In-Reply-To: <475589ED.1060608@batosai.net>
References: <475589ED.1060608@batosai.net>
Message-ID: <20071204171419.GB4322@freenetproject.org>

* Julien Cornuwel <batosai at batosai.net> [2007-12-04 18:10:05]:

> Hi,
> 
> This translation file features several new strings and numerous fixes of
> chains that had been modified in the english version.
> 
> Regards.

Commited in r16265
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071204/f61d32bf/attachment.pgp 

From toad at amphibian.dyndns.org  Tue Dec  4 17:23:35 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Tue, 4 Dec 2007 17:23:35 +0000
Subject: [freenet-dev] Updating the People page
In-Reply-To: <4741D958.7090009@batosai.net>
References: <200711191729.43587.toad@amphibian.dyndns.org>
	<4741D958.7090009@batosai.net>
Message-ID: <200712041723.40669.toad@amphibian.dyndns.org>

Added. Anyone else who has contributed in the last few years who wants to be 
on the People page, please either add yourself (I will vet outrageous 
entries!) or ask.

On Monday 19 November 2007 18:43, Julien Cornuwel aka batosai wrote:
> Matthew Toseland a ?crit :
> > Would anyone like to be added to the People page? If you have
> contributed code
> > or translations either via SVN or via patches, and you want to be
> included,
> > please reply to this email. Or edit the people page directly in the
> website
> > SVN (we will edit any outrageous entries!).
> 
> Thanks ! I'd be glad to be added on this page for my work on French
> translations (fProxy, Frost and now jSite) and on the documentation.
> 
> Maybe you could add a link to the documentation site ? It's not
> finished and french only (ATM, internationalization is planned) but
> you can see it at http://www.freenet-doc.fr It's a common work with
> Utopiah and Dieppe. I don't know their real names or if they want to
> appear on the page.
> 
> My old French documentation site (freenet-fr.info) is no longer
> maintained and will disappear as soon as the new one will be validated.
> 
> Regards
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071204/c5f9a66e/attachment.pgp 

From batosai at batosai.net  Tue Dec  4 17:25:48 2007
From: batosai at batosai.net (Julien Cornuwel)
Date: Tue, 04 Dec 2007 18:25:48 +0100
Subject: [freenet-dev] Updating the People page
In-Reply-To: <200712041723.40669.toad@amphibian.dyndns.org>
References: <200711191729.43587.toad@amphibian.dyndns.org>	<4741D958.7090009@batosai.net>
	<200712041723.40669.toad@amphibian.dyndns.org>
Message-ID: <47558D9C.8050903@batosai.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matthew Toseland a ?crit :
> Added. Anyone else who has contributed in the last few years who wants to be 
> on the People page, please either add yourself (I will vet outrageous 
> entries!) or ask.

Thanks ;)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHVY2co6N05NzaOvURApdwAJ9K9/cGTwcW1v3x09ERNd/TZKjnpgCfQThs
svVwkxtI1OMrnyNY5uWJqt0=
=6clk
-----END PGP SIGNATURE-----


From freenet-devl at osndok.com  Tue Dec  4 19:58:27 2007
From: freenet-devl at osndok.com (Robert Hailey)
Date: Tue, 4 Dec 2007 13:58:27 -0600
Subject: [freenet-dev] Updating the People page
In-Reply-To: <200712041723.40669.toad@amphibian.dyndns.org>
References: <200711191729.43587.toad@amphibian.dyndns.org>
	<4741D958.7090009@batosai.net>
	<200712041723.40669.toad@amphibian.dyndns.org>
Message-ID: <BABF96EA-1CEF-49CB-AA6F-9D7E72E0F960@osndok.com>


On Dec 4, 2007, at 11:23 AM, Matthew Toseland wrote:

> Added. Anyone else who has contributed in the last few years who  
> wants to be
> on the People page, please either add yourself (I will vet outrageous
> entries!) or ask.

I was going to add:

<p><b>Robert Hailey</b><br>
Robert has helped the speed and security of Freenet. Credited with  
finding
the '1010' crypto bug.
</p>

But on trying to commit the page I get 403/Forbidden. I would also  
recommend either bolding the "And hundreds of others" (or a horizontal  
rule) to separate the postscript from the list.

--
Robert Hailey



From toad at amphibian.dyndns.org  Tue Dec  4 20:07:15 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Tue, 4 Dec 2007 20:07:15 +0000
Subject: [freenet-dev] Freenet 0.7 build 1088
Message-ID: <200712042007.21097.toad@amphibian.dyndns.org>

Freenet 0.7 build 1088 is now available. Please upgrade. The major changes in 
this build relate to opennet: We are trying to get full opennet working, so 
that Freenet will work out of the box with no need for reference exchange on 
#freenet-refs . Build 1088 includes a hopefully working announcement 
protocol, which is the second half of the bootstrapping mechanism: once a 
node has connected to a seednode (a node from a list we provide at install 
time), it does an announcement to get a bunch of nodes to connect to. After 
that, opennet path folding will keep the node busy and integrated.

Because we need to get full opennet working as soon as possible, this is 
another mandatory build (at 0:00 on Thursday). Sorry, but the end justifies 
the means in this case: Getting full opennet working (hopefully in time for 
Christmas) will greatly improve Freenet's ease of installation.

Thank you for using Freenet. If you find any bugs, please report them.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071204/5b890494/attachment.pgp 

From juiceman69 at gmail.com  Wed Dec  5 00:42:17 2007
From: juiceman69 at gmail.com (Juiceman)
Date: Tue, 4 Dec 2007 19:42:17 -0500
Subject: [freenet-dev] NPE in trunk (rev ~16295)
Message-ID: <8b525dee0712041642j4da0c27kebaafae5e54f4109@mail.gmail.com>

INFO   | jvm 1    | 2007/12/04 19:38:37 | Error in
WrapperListener.start callback.  java.lang.NullPointerException
INFO   | jvm 1    | 2007/12/04 19:38:37 | java.lang.NullPointerException
INFO   | jvm 1    | 2007/12/04 19:38:37 | 	at
java.lang.System.arraycopy(Native Method)
INFO   | jvm 1    | 2007/12/04 19:38:37 | 	at
freenet.crypt.ciphers.Rijndael.initialize(Rijndael.java:73)
INFO   | jvm 1    | 2007/12/04 19:38:37 | 	at
freenet.node.NodeCrypto.<init>(NodeCrypto.java:160)
INFO   | jvm 1    | 2007/12/04 19:38:37 | 	at
freenet.node.Node.<init>(Node.java:693)
INFO   | jvm 1    | 2007/12/04 19:38:37 | 	at
freenet.node.NodeStarter.start(NodeStarter.java:152)
INFO   | jvm 1    | 2007/12/04 19:38:37 | 	at
org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:2788)
STATUS | wrapper  | 2007/12/04 19:38:38 | <-- Wrapper Stopped

-- 
I may disagree with what you have to say, but I shall defend, to the
death, your right to say it. - Voltaire
Those who would give up Liberty, to purchase temporary Safety, deserve
neither Liberty nor Safety. - Ben Franklin


From toad at amphibian.dyndns.org  Thu Dec  6 02:01:50 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Thu, 6 Dec 2007 02:01:50 +0000
Subject: [freenet-dev] Opennet progress report
Message-ID: <200712060201.55033.toad@amphibian.dyndns.org>

Announcement is working, including connecting to initial seednodes. We don't 
have any automated way of gathering seednodes yet. There are a couple of 
largish bugs in the deployed code which cause announcements to bounce around 
the network indefinitely; fortunately, the deployed code doesn't actually 
make announcement requests. The trunk code will out of the box announce 
itself to randomly chosen nodes in its seednodes.fref (if there is one) until 
it has 10 opennet peers (if opennet is enabled).

So, we need:
- Yet another mandatory (1089). Sorry folk.
- A list of public seednodes. If a few people with static IPs/dyndns and 
opennet would send me their noderefs, we can build a temporary seednodes.fref 
from that. In the longer term, the node will detect if you are eligible to be 
a seednode, ask if you want to be one, and if so upload regularly to a server 
on emu.
- To bundle this list with the node.

In conclusion: the end of #freenet-refs is nigh, finally!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071206/04260f06/attachment.pgp 

From m.rogers at cs.ucl.ac.uk  Thu Dec  6 03:25:11 2007
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Thu, 06 Dec 2007 03:25:11 +0000
Subject: [freenet-dev] Opennet progress report
In-Reply-To: <200712060201.55033.toad@amphibian.dyndns.org>
References: <200712060201.55033.toad@amphibian.dyndns.org>
Message-ID: <47576B97.9070407@cs.ucl.ac.uk>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matthew Toseland wrote:
> - A list of public seednodes. If a few people with static IPs/dyndns and 
> opennet would send me their noderefs, we can build a temporary seednodes.fref 
> from that. In the longer term, the node will detect if you are eligible to be 
> a seednode, ask if you want to be one, and if so upload regularly to a server 
> on emu.

Attached - but I'll probably be taking my node offline between the 12th
and the 18th, so if that's likely to be a critical period then it might
be better not to distribute my ref.

> In conclusion: the end of #freenet-refs is nigh, finally!

Nice work! :-)

Cheers,
Michael

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHV2uXyua14OQlJ3sRAr68AJ9Ppw6yUHxs9m4WIXH5eZpPa5g51QCeKOcW
d/8GUkxxFr/91u6mp6eULbw=
=7gai
-----END PGP SIGNATURE-----
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: myref.txt
Url: http://emu.freenetproject.org/pipermail/devl/attachments/20071206/9c12489f/attachment.txt 

From m.rogers at cs.ucl.ac.uk  Thu Dec  6 03:37:38 2007
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Thu, 06 Dec 2007 03:37:38 +0000
Subject: [freenet-dev] Opennet progress report
In-Reply-To: <47576B97.9070407@cs.ucl.ac.uk>
References: <200712060201.55033.toad@amphibian.dyndns.org>
	<47576B97.9070407@cs.ucl.ac.uk>
Message-ID: <47576E82.6010409@cs.ucl.ac.uk>

Sorry, that was supposed to be off-list. Argh.


From Volodya at WhenGendarmeSleeps.org  Thu Dec  6 07:45:01 2007
From: Volodya at WhenGendarmeSleeps.org (Volodya)
Date: Thu, 06 Dec 2007 10:45:01 +0300
Subject: [freenet-dev] Short refs was Re: alternative to #freenet-refs
In-Reply-To: <4754508B.7000606@cs.ucl.ac.uk>
References: <20071107220045.GA4554@freenetproject.org>	<1195995849.10868.24.camel@localhost>	<474B09FD.7010906@cs.ucl.ac.uk>	<200711271144.47979.toad@amphibian.dyndns.org>	<474DD412.307@cs.ucl.ac.uk>	<474EBFC4.7030201@WhenGendarmeSleeps.org>
	<4754508B.7000606@cs.ucl.ac.uk>
Message-ID: <4757A87D.8040201@WhenGendarmeSleeps.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael Rogers wrote:
> Thanks! Here's the Java version of the poem generator (same dictionaries
> as before).
> 
> Cheers,
> Michael

Have you tried to give it enough words to represent a full key? I have a feeling that what might be needed to
not to load the whole file, but rather use it sax style. So you know that you need 1047th line, and that is the
only one you actually store in the memory. I'll see if i can improve on your code a bit. I really loved it. q;-)

                 - Volodya

- --
http://freedom.libsyn.com/       Voice of Freedom, Radical Podcast
http://eng.anarchopedia.org/     Anarchopedia, A Free Knowledge Portal

 "None of us are free until all of us are free."    ~ Mihail Bakunin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHV6h9uWy2EFICg+0RArltAKC2qOsYqxnfmwh1CQeqFlMMN0rLPQCfc2R2
8pvaCGIsHHU/plJbn5OmGgs=
=IRo/
-----END PGP SIGNATURE-----


From nextgens at freenetproject.org  Thu Dec  6 09:53:16 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=)
Date: Thu, 6 Dec 2007 10:53:16 +0100
Subject: [freenet-dev] Opennet progress report
In-Reply-To: <200712060201.55033.toad@amphibian.dyndns.org>
References: <200712060201.55033.toad@amphibian.dyndns.org>
Message-ID: <20071206095315.GA4201@freenetproject.org>

* Matthew Toseland <toad at amphibian.dyndns.org> [2007-12-06 02:01:50]:

> Announcement is working, including connecting to initial seednodes. We don't 
> have any automated way of gathering seednodes yet. There are a couple of 
> largish bugs in the deployed code which cause announcements to bounce around 
> the network indefinitely; fortunately, the deployed code doesn't actually 
> make announcement requests. The trunk code will out of the box announce 
> itself to randomly chosen nodes in its seednodes.fref (if there is one) until 
> it has 10 opennet peers (if opennet is enabled).
> 
> So, we need:
> - Yet another mandatory (1089). Sorry folk.
> - A list of public seednodes. If a few people with static IPs/dyndns and 
> opennet would send me their noderefs, we can build a temporary seednodes.fref 
> from that. In the longer term, the node will detect if you are eligible to be 
> a seednode, ask if you want to be one, and if so upload regularly to a server 
> on emu.

On .5 Emu was polling the data out of some nodes... I would prefer it to
remain that way in the longer-term ;)

NextGen$
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071206/36845e22/attachment.pgp 

From toad at amphibian.dyndns.org  Thu Dec  6 14:10:21 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Thu, 6 Dec 2007 14:10:21 +0000
Subject: [freenet-dev] Opennet progress report
In-Reply-To: <20071206095315.GA4201@freenetproject.org>
References: <200712060201.55033.toad@amphibian.dyndns.org>
	<20071206095315.GA4201@freenetproject.org>
Message-ID: <200712061410.27765.toad@amphibian.dyndns.org>

On Thursday 06 December 2007 09:53, Florent Daigni?re wrote:
> * Matthew Toseland <toad at amphibian.dyndns.org> [2007-12-06 02:01:50]:
> 
> On .5 Emu was polling the data out of some nodes... I would prefer it to
> remain that way in the longer-term ;)

IMHO it's not really feasible as most nodes won't be eligible to be seednodes 
(generally due to being NATed).
> 
> NextGen$
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071206/a5a9111c/attachment.pgp 

From toad at amphibian.dyndns.org  Thu Dec  6 16:40:52 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Thu, 6 Dec 2007 16:40:52 +0000
Subject: [freenet-dev] Freenet 0.7 build 1089
Message-ID: <200712061640.58234.toad@amphibian.dyndns.org>

Freenet 0.7 build 1089 is now available. Please upgrade. This will be 
mandatory 0:00 Saturday (GMT). The main changes in this build relate to 
opennet: opennet is now 95% working, if the node has no peers, has opennet 
enabled, and has a seednodes file, it will try to connect to randomly chosen 
seednodes and do an announcement to them. Sadly 1088 had some major bugs in 
the announcement code, so we need 1089 to be deployed as quickly as possible 
so we can test to see whether they are fixed before actually adding a 
seednodes file to the installer. But certainly within the coming week 
#freenet-refs will be no more. So please upgrade! Testing, of both this 
build, and trunk builds, would be appreciated, and please report any bugs you 
find to the bug tracker (https://bugs.freenetproject.org/ ), to the mailing 
lists or to Frost. Thanks.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071206/58a4c8c4/attachment.pgp 

From freenet-devl at david.sowder.com  Thu Dec  6 17:32:06 2007
From: freenet-devl at david.sowder.com (David Sowder (Zothar))
Date: Thu, 06 Dec 2007 11:32:06 -0600
Subject: [freenet-dev] Opennet progress report
In-Reply-To: <200712061410.27765.toad@amphibian.dyndns.org>
References: <200712060201.55033.toad@amphibian.dyndns.org>	<20071206095315.GA4201@freenetproject.org>
	<200712061410.27765.toad@amphibian.dyndns.org>
Message-ID: <47583216.8030102@david.sowder.com>

I believe eligible seednodes pushing voluntarily to emu is much more
maintainable than emu needing to be configured to pull from specific
volunteer nodes that may or may not still be around in six months. 
Nodes pushing is more automatic.

Matthew Toseland wrote:
> On Thursday 06 December 2007 09:53, Florent Daigni?re wrote:
>   
>> * Matthew Toseland <toad at amphibian.dyndns.org> [2007-12-06 02:01:50]:
>>
>> On .5 Emu was polling the data out of some nodes... I would prefer it to
>> remain that way in the longer-term ;)
>>     
>
> IMHO it's not really feasible as most nodes won't be eligible to be seednodes 
> (generally due to being NATed).
>   



From ian.clarke at gmail.com  Thu Dec  6 17:57:27 2007
From: ian.clarke at gmail.com (Ian Clarke)
Date: Thu, 6 Dec 2007 11:57:27 -0600
Subject: [freenet-dev] Opennet progress report
In-Reply-To: <47583216.8030102@david.sowder.com>
References: <200712060201.55033.toad@amphibian.dyndns.org>
	<20071206095315.GA4201@freenetproject.org>
	<200712061410.27765.toad@amphibian.dyndns.org>
	<47583216.8030102@david.sowder.com>
Message-ID: <823242bd0712060957i32c28998ycdc9c3229fd3c1b1@mail.gmail.com>

Agreed, I think a push approach will be better, and probably simpler
to implement.

Ian.

On 12/6/07, David Sowder (Zothar) <freenet-devl at david.sowder.com> wrote:
> I believe eligible seednodes pushing voluntarily to emu is much more
> maintainable than emu needing to be configured to pull from specific
> volunteer nodes that may or may not still be around in six months.
> Nodes pushing is more automatic.
>
> Matthew Toseland wrote:
> > On Thursday 06 December 2007 09:53, Florent Daigni?re wrote:
> >
> >> * Matthew Toseland <toad at amphibian.dyndns.org> [2007-12-06 02:01:50]:
> >>
> >> On .5 Emu was polling the data out of some nodes... I would prefer it to
> >> remain that way in the longer-term ;)
> >>
> >
> > IMHO it's not really feasible as most nodes won't be eligible to be seednodes
> > (generally due to being NATed).
> >
>
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>
>


-- 
Email: ian.clarke at gmail.com
Cell: +1 512 422 3588
AIM: ian.clarke at mac.com
Skype: sanity


From freenet-devl at osndok.com  Thu Dec  6 18:27:21 2007
From: freenet-devl at osndok.com (Robert Hailey)
Date: Thu, 6 Dec 2007 12:27:21 -0600
Subject: [freenet-dev] NPE in trunk
Message-ID: <18A11276-4F8F-4726-B178-8C845E8385E6@osndok.com>

Someone's sending packets w/o a source, or something.

java.lang.NullPointerException
	at freenet.io.comm.Message.<init>(Message.java:121)
	at freenet.io.comm.Message.<init>(Message.java:114)
	at freenet.io.comm.DMT.createFNPVoid(DMT.java:1198)
	at freenet.node.PacketSender.realRun(PacketSender.java:308)
	at freenet.node.PacketSender.run(PacketSender.java:160)
	at java.lang.Thread.run(Thread.java:613)

--
Robert Hailey



From freenet-devl at osndok.com  Thu Dec  6 20:16:34 2007
From: freenet-devl at osndok.com (Robert Hailey)
Date: Thu, 6 Dec 2007 14:16:34 -0600
Subject: [freenet-dev] Why Freenet is so SLOW! / r16372
In-Reply-To: <200712020002.10322.toad@amphibian.dyndns.org>
References: <200711300126.45810.toad@amphibian.dyndns.org>
	<EA219B65-8C33-49C7-A3F0-475E1B3A35AA@osndok.com>
	<8EFC03C4-519E-48A0-8463-EE77E717B9BB@osndok.com>
	<200712020002.10322.toad@amphibian.dyndns.org>
Message-ID: <15D4A571-25AD-48C4-A12D-8E79E66E2F12@osndok.com>


On Dec 1, 2007, at 6:02 PM, Matthew Toseland wrote:

> On Friday 30 November 2007 23:03, Robert Hailey wrote:
>>
>> A much-more aggressive patch (which makes all RequestHandler sends
>> Async except the 'last') shows could be 12x; but am concerned over  
>> the
>> byte-counter side effects with which I am not familiar.
>
> [...]
> - To get the correct byte counts.
> This is really important, because it underlies the major load limiting
> systems. However it is only important for terminal messages: we can
> reasonably assume the rest will be piggybacked if necessary.
>
> In RequestHandler, most of the sendSync()'s are terminal, and  
> therefore must
> remain. The one that is the most interesting is the one you  
> mentioned above:
> sending the Accepted message.


Well... I found the time to get the "more aggressive" patch working.

The general idea is that (after the final request) because the ~only~  
reason to keep the thread waiting on the final transmission is the  
byte count, to delegate that to the 'sent()' callback thread.

On my machine (which was showing pre-emptive reject cause as  
">threadLimit", because of the high bandwidth ceiling I have set), I  
saw a increase in node bandwidth (in & out), and saw a new pre-emptive  
rejection cause "insufficient output bandwidth". Realizing then that  
the bandwidth ceiling (being so high) was the cause for my node  
behaving differently than others, I changed my config to less-than  
that observed value, and "output bandwidth liability" became  
predominate (as Matthew said), and re-ran the tests with the more- 
expected results:

For my machine, it appears to save 10 to 49 threads (the JVM-supplied  
value), varying largely presumably due to variances in the length of  
send queues. Also, I've seen quite high numbers for 'pooled threads  
awaiting work'.

After cursory inspection, I suspect that the other related handlers  
(Insert/SSK) would not see the same benefit as RequestHandler; and are  
not modified.

It may not therefore show a significant change to the common user (as  
the ack appears to have), but (at least for me, i.e. ">threadLimit"  
reject) it appears to give the user the ability to use more bandwidth,  
or perhaps to have longer send-queues (service slower peers).

Question: Is it a bug that messages back to the source are not counted  
as the liability for handling a request, or is this intentional? (e.g.  
sendSync(dnf, null); )

Question: Should there be a note in the config page about bandwidth  
limits effect on threads? It seems the more bandwidth you allocate,  
the more requests you serve, the more threads freenet uses.

Commited in r16372

--
Robert Hailey



From m.rogers at cs.ucl.ac.uk  Thu Dec  6 22:03:32 2007
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Thu, 06 Dec 2007 22:03:32 +0000
Subject: [freenet-dev] Short refs was Re: alternative to #freenet-refs
In-Reply-To: <4757A87D.8040201@WhenGendarmeSleeps.org>
References: <20071107220045.GA4554@freenetproject.org>
	<1195995849.10868.24.camel@localhost>
	<474B09FD.7010906@cs.ucl.ac.uk>
	<200711271144.47979.toad@amphibian.dyndns.org>
	<474DD412.307@cs.ucl.ac.uk>
	<474EBFC4.7030201@WhenGendarmeSleeps.org>
	<4754508B.7000606@cs.ucl.ac.uk>
	<4757A87D.8040201@WhenGendarmeSleeps.org>
Message-ID: <475871B4.1070705@cs.ucl.ac.uk>

Volodya wrote:
> Have you tried to give it enough words to represent a full key?

Yup, it can represent any amount of data as poetry without using any
extra memory - try feeding it a very large command line argument.

Of course, at the moment it's just representing random data. Where it
currently calls rand() to generate array indices it should actually be
reading data from stdin, and there should be a companion program that
reads poetry from stdin, uses a hashtable to recover the array indices
of the words, and prints the original data on stdout. I've done
something similar in C before so I know it works in principle, I just
need to get round to finishing this version.

Cheers,
Michael


From toad at amphibian.dyndns.org  Fri Dec  7 00:00:28 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Fri, 7 Dec 2007 00:00:28 +0000
Subject: [freenet-dev] Why Freenet is so SLOW! / r16372
In-Reply-To: <15D4A571-25AD-48C4-A12D-8E79E66E2F12@osndok.com>
References: <200711300126.45810.toad@amphibian.dyndns.org>
	<200712020002.10322.toad@amphibian.dyndns.org>
	<15D4A571-25AD-48C4-A12D-8E79E66E2F12@osndok.com>
Message-ID: <200712070000.34903.toad@amphibian.dyndns.org>

On Thursday 06 December 2007 20:16, Robert Hailey wrote:
> 
> On Dec 1, 2007, at 6:02 PM, Matthew Toseland wrote:
> 
> > On Friday 30 November 2007 23:03, Robert Hailey wrote:
> >>
> >> A much-more aggressive patch (which makes all RequestHandler sends
> >> Async except the 'last') shows could be 12x; but am concerned over  
> >> the
> >> byte-counter side effects with which I am not familiar.
> >
> > [...]
> > - To get the correct byte counts.
> > This is really important, because it underlies the major load limiting
> > systems. However it is only important for terminal messages: we can
> > reasonably assume the rest will be piggybacked if necessary.
> >
> > In RequestHandler, most of the sendSync()'s are terminal, and  
> > therefore must
> > remain. The one that is the most interesting is the one you  
> > mentioned above:
> > sending the Accepted message.
> 
> 
> Well... I found the time to get the "more aggressive" patch working.
> 
> The general idea is that (after the final request) because the ~only~  
> reason to keep the thread waiting on the final transmission is the  
> byte count, to delegate that to the 'sent()' callback thread.

Makes sense.
> 
> On my machine (which was showing pre-emptive reject cause as  
> ">threadLimit", because of the high bandwidth ceiling I have set), I  
> saw a increase in node bandwidth (in & out), and saw a new pre-emptive  
> rejection cause "insufficient output bandwidth". Realizing then that  
> the bandwidth ceiling (being so high) was the cause for my node  
> behaving differently than others, I changed my config to less-than  
> that observed value, and "output bandwidth liability" became  
> predominate (as Matthew said), and re-ran the tests with the more- 
> expected results:
> 
> For my machine, it appears to save 10 to 49 threads (the JVM-supplied  
> value), varying largely presumably due to variances in the length of  
> send queues. Also, I've seen quite high numbers for 'pooled threads  
> awaiting work'.

Right, no real network-level effects, unless lots of nodes are bottlenecked on 
threads, but a good local optimisation for thread-scarce nodes.
> 
> After cursory inspection, I suspect that the other related handlers  
> (Insert/SSK) would not see the same benefit as RequestHandler; and are  
> not modified.

Well they generally take longer, and use more threads (especially CHK 
inserts).
> 
> It may not therefore show a significant change to the common user (as  
> the ack appears to have), but (at least for me, i.e. ">threadLimit"  
> reject) it appears to give the user the ability to use more bandwidth,  
> or perhaps to have longer send-queues (service slower peers).
> 
> Question: Is it a bug that messages back to the source are not counted  
> as the liability for handling a request, or is this intentional? (e.g.  
> sendSync(dnf, null); )

Probably. If it was intentional there should be a comment explaining it.
> 
> Question: Should there be a note in the config page about bandwidth  
> limits effect on threads? It seems the more bandwidth you allocate,  
> the more requests you serve, the more threads freenet uses.

Isn't that obvious? The more load the node is under, the more resources it's 
likely to use?
> 
> Commited in r16372
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071207/23261e60/attachment.pgp 

From jolantern60 at yahoo.com  Fri Dec  7 00:27:47 2007
From: jolantern60 at yahoo.com (Jack O'Lantern)
Date: Thu, 6 Dec 2007 16:27:47 -0800 (PST)
Subject: [freenet-dev] 0.5 unstable build.xml
Message-ID: <159763.52877.qm@web44812.mail.sp1.yahoo.com>

Hi,

I've applied the Diffie-Hellman patch to unstable now. Could Nextgens
or a different site admin apply the same patch to the unstable
build.xml that was applied to the stable one in revs 16002/16003/16113,
please? Thank you!

Jack


      ____________________________________________________________________________________
Be a better friend, newshound, and 
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ 




From toad at amphibian.dyndns.org  Fri Dec  7 13:09:22 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Fri, 7 Dec 2007 13:09:22 +0000
Subject: [freenet-dev] 0.5 unstable build.xml
In-Reply-To: <159763.52877.qm@web44812.mail.sp1.yahoo.com>
References: <159763.52877.qm@web44812.mail.sp1.yahoo.com>
Message-ID: <200712071309.29170.toad@amphibian.dyndns.org>

On Friday 07 December 2007 00:27, Jack O'Lantern wrote:
> Hi,
> 
> I've applied the Diffie-Hellman patch to unstable now. Could Nextgens
> or a different site admin apply the same patch to the unstable
> build.xml that was applied to the stable one in revs 16002/16003/16113,
> please? Thank you!

Are you planning on maintaining the unstable branch (0.6) ? If so, why? We 
deprecated it, is there any good reason to revive it?
> 
> Jack
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071207/3619feef/attachment.pgp 

From toad at amphibian.dyndns.org  Fri Dec  7 18:34:10 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Fri, 7 Dec 2007 18:34:10 +0000
Subject: [freenet-dev] [freenet-cvs] r16372 -
	trunk/freenet/src/freenet/node
In-Reply-To: <20071206202408.D8A18480F1F@freenetproject.org>
References: <20071206202408.D8A18480F1F@freenetproject.org>
Message-ID: <200712071834.15332.toad@amphibian.dyndns.org>

Some issues..

On Thursday 06 December 2007 20:24, you wrote:
> Author: robert
> Date: 2007-12-06 20:24:08 +0000 (Thu, 06 Dec 2007)
> New Revision: 16372
> 
> Modified:
>    trunk/freenet/src/freenet/node/MessageItem.java
>    trunk/freenet/src/freenet/node/RequestHandler.java
> Log:
> Don't hang onto a RequestHandler thread just to track the bytes.
> 
> @@ -144,11 +148,14 @@
>              		new BlockTransmitter(node.usm, source, uid, prb, 
node.outputThrottle, this);
>              	node.addTransferringRequestHandler(uid);
>              	if(bt.send(node.executor)) {
> -            		status = RequestSender.SUCCESS; // for byte logging
> +                    // for byte logging
> +            		status = RequestSender.SUCCESS;
>              		// We've fetched it from our datastore, so there won't be a 
downstream noderef.
>              		// But we want to send at least an FNPOpennetCompletedAck, 
otherwise the request source
>              		// may have to timeout waiting for one.
>             			finishOpennetNoRelay();
> +                    //also for byte logging, since the block is 
the 'terminal' message.
> +                    applyByteCounts();

Careful here! IMHO we need to include the opennet messages in the bytes total 
for the request. finishOpennetNoReply() may sendAsync a 
FNPOpennetCompletedAck or an actual noderef...

> @@ -172,13 +179,14 @@
>              if((waitStatus & RequestSender.WAIT_REJECTED_OVERLOAD) != 0) {
>              	// Forward RejectedOverload
>              	Message msg = DMT.createFNPRejectedOverload(uid, false);
> -            	source.sendAsync(msg, null, 0, null);
> +            	source.sendAsync(msg, null, 0, this);
>              }
>              
>              if((waitStatus & RequestSender.WAIT_TRANSFERRING_DATA) != 0) {
>              	// Is a CHK.
>                  Message df = DMT.createFNPCHKDataFound(uid, 
rs.getHeaders());
> -                source.sendSync(df, null);
> +                source.sendAsync(df, null, 0, this);
> +                
>                  PartiallyReceivedBlock prb = rs.getPRB();
>              	BlockTransmitter bt =
>              	    new BlockTransmitter(node.usm, source, uid, prb, 
node.outputThrottle, this);
> @@ -190,6 +198,8 @@
>             			finishOpennetChecked();
>              	}
>  				status = rs.getStatus();
> +                //for byte logging, since the block is the 'terminal' 
message.
> +                applyByteCounts();

This is fine btw, we wait for an explicit acknowledgement so we know the 
blocks have been sent.

> +        private boolean once=true;
> +		public void sent() {
> +            //For byte counting, this relies on the fact that the callback 
will only be excuted once. This check might be paranoid.
> +            if (once) {
> +                applyByteCounts();
> +            } else {
> +                Logger.error(this, "terminalMessage sent multiple times? 
for " + RequestHandler.this);
> +            }
> +        }

It's a sensible check but afaics you haven't implemented it - where is once 
set to false?!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20071207/5bd06be6/attachment.pgp 

From jolantern60 at yahoo.com  Fri Dec  7 18:51:46 2007
From: jolantern60 at yahoo.com (Jack O'Lantern)
Date: Fri, 7 Dec 2007 10:51:46 -0800 (PST)
Subject: [freenet-dev] 0.5 unstable build.xml
In-Reply-To: <200712071309.29170.toad@amphibian.dyndns.org>
Message-ID: <293123.20280.qm@web44816.mail.sp1.yahoo.com>

--- Matthew Toseland <toad at amphibian.dyndns.org> wrote:
> Are you planning on maintaining the unstable branch (0.6) ? If so,
> why? We deprecated it, is there any good reason to revive it?

The differences between stable and unstable are very small (try diff
-ruwB -x .svn stable/ unstable/, that returns only few dozen K of
differences, about half of which are unimportant). The thing is that
every commit to stable triggers the auto-builder and on the official
download page you get something which claims to be 5108 but is really
halfway through to 5109. So I thought I'd use unstable for regular
committing and stable only when a new build is due. I'm aware that this
is not the most efficient way to handle this problem and I'd be glad if
you would be willing to offer me a better solution.

Jack


      ____________________________________________________________________________________
Looking for last minute shopping deals?  
Find them fast with Yahoo! Search.  http://tools.search.yahoo.com/newsearch/category.php?category=shopping



From freenet-devl at osndok.com  Fri Dec  7 21:06:47 2007
From: freenet-devl at osndok.com (Robert Hailey)
Date: Fri, 7 Dec 2007 15:06:47 -0600
Subject: [freenet-dev] [freenet-cvs] r16372 -
	trunk/freenet/src/freenet/node
In-Reply-To: <200712071834.15332.toad@amphibian.dyndns.org>
References: <20071206202408.D8A18480F1F@freenetproject.org>
	<200712071834.15332.toad@amphibian.dyndns.org>
Message-ID: <BD2BB9C2-27A0-4BED-9B4F-FF02890A8D19@osndok.com>


On Dec 7, 2007, at 12:34 PM, Matthew Toseland wrote:

> Some issues..
>
> Careful here! IMHO we need to include the opennet messages in the  
> bytes total
> for the request. finishOpennetNoReply() may sendAsync a
> FNPOpennetCompletedAck or an actual noderef...

Ah, yes. I changed the ack in both open net finish..() to  
sendTerminal(), and moved applyByteCounts() to the error condition  
(when no ack is sent).

> It's a sensible check but afaics you haven't implemented it - where  
> is once
> set to false?!

Quite right. Both fixed in r16398.

--
Robert Hailey



From nextgens at freenetproject.org  Fri Dec  7 22:32:10 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=)
Date: Fri, 7 Dec 2007 23:32:10 +0100
Subject: [freenet-dev] 0.5 unstable build.xml
In-Reply-To: <293123.20280.qm@web44816.mail.sp1.yahoo.com>
References: <200712071309.29170.toad@amphibian.dyndns.org>
	<293123.20280.qm@web44816.mail.sp1.yahoo.com>
Message-ID: <20071207223210.GA3886@freenetproject.org>

* Jack O'Lantern <jolantern60 at yahoo.com> [2007-12-07 10:51:46]:

> --- Matthew Toseland <toad at amphibian.dyndns.org> wrote:
> > Are you planning on maintaining the unstable branch (0.6) ? If so,
> > why? We deprecated it, is there any good reason to revive it?
> 
> The differences between stable and unstable are very small (try diff
> -ruwB -x .svn stable/ unstable/, that returns only few dozen K of
> differences, about half of which are unimportant).

That's a good reason to focus on .5 ;)

> The thing is that
> every commit to stable triggers the auto-builder and on the official
> download page you get something which claims to be 5108 but is really
> halfway through to 5109. 

I'm not convinced that back-porting my scripts to the .5 tree (auto-release
gets triggered only when me or toad change the vers