From evanbd at gmail.com  Fri Feb  1 00:48:50 2008
From: evanbd at gmail.com (Evan Daniel)
Date: Thu, 31 Jan 2008 19:48:50 -0500
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <200801302249.42207.toad@amphibian.dyndns.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<Prayer.1.0.16.0801301734100.93197@bells3.cs.ucl.ac.uk>
	<0E7521BC-7329-458D-ADAC-229BE691A6A2@freenetproject.org>
	<200801302249.42207.toad@amphibian.dyndns.org>
Message-ID: <4f9383510801311648h354b8e41h98148553a26b6348@mail.gmail.com>

On Jan 30, 2008 5:49 PM, Matthew Toseland <toad at amphibian.dyndns.org> wrote:

> You also need an escape-route mechanism - a way to find an entrance into
> another network once regular routing has exhausted the local network.

Doesn't this allow an attacker to selectively DOS the bottleneck
points by sending out requests for non-existant data?

Evan Daniel


From robert at freenetproject.org  Fri Feb  1 17:00:04 2008
From: robert at freenetproject.org (Robert Hailey)
Date: Fri, 1 Feb 2008 11:00:04 -0600
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <8ECAA540-7437-473B-9573-19E3955B9980@freenetproject.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<Prayer.1.0.16.0801301524240.92105@bells3.cs.ucl.ac.uk>
	<91F6684A-8661-4E4D-8512-35F3D1DE0316@freenetproject.org>
	<Prayer.1.0.16.0801301734100.93197@bells3.cs.ucl.ac.uk>
	<0E7521BC-7329-458D-ADAC-229BE691A6A2@freenetproject.org>
	<Prayer.1.0.16.0801311133340.98065@bells3.cs.ucl.ac.uk>
	<8ECAA540-7437-473B-9573-19E3955B9980@freenetproject.org>
Message-ID: <9B1249F7-AED1-470C-8F55-6FAE33C45B8E@freenetproject.org>


On Jan 31, 2008, at 11:03 AM, Robert Hailey wrote:

>> How do you authenticate the routed pings, to prevent an attacker from
>> replying on behalf of another node?
>
> Excellent question. Surely the "true/false" response of present is
> woefully inadequate. Since we have a direct connection to the peer
> that we are pinging a challange-and-response mechanism is easy, no?
>
> Consider node "B" who is between "A" & "C" (A-B-C). He tells "C" a UID
> & Secret [a randomly generated long?], and "C" stores that secret/uid
> as part of our peernode record. Node "B" then sends node "A" a routed
> ping with the same UID, and if node "A" returns the pong with the
> correct secret it is a success.

I was supposing that these pings would be sent at less-than-max htl  
(since we are not searching the network but doing a connectivity  
test), but wouldn't that possibly allow an attacker to learn who your  
peers are?

That is, if an attacker has a node connected to your node and your  
peers node, he could put together the ping from yours, the reply from  
your peer, plus the fact that the reply comes from a node of the same  
location as the ping, and be reasonably sure he is your peer. Whereas  
with the probabilistic decrement at the real maxHTL, they could not be  
nearly so sure.

--
Robert Hailey

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://emu.freenetproject.org/pipermail/devl/attachments/20080201/88402788/attachment.htm 

From robert at freenetproject.org  Fri Feb  1 17:00:08 2008
From: robert at freenetproject.org (Robert Hailey)
Date: Fri, 1 Feb 2008 11:00:08 -0600
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <4f9383510801311648h354b8e41h98148553a26b6348@mail.gmail.com>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<Prayer.1.0.16.0801301734100.93197@bells3.cs.ucl.ac.uk>
	<0E7521BC-7329-458D-ADAC-229BE691A6A2@freenetproject.org>
	<200801302249.42207.toad@amphibian.dyndns.org>
	<4f9383510801311648h354b8e41h98148553a26b6348@mail.gmail.com>
Message-ID: <BE97F36A-93F6-41D3-9032-2E8C37BE16E3@freenetproject.org>


On Jan 31, 2008, at 6:48 PM, Evan Daniel wrote:

> On Jan 30, 2008 5:49 PM, Matthew Toseland  
> <toad at amphibian.dyndns.org> wrote:
>
>> You also need an escape-route mechanism - a way to find an entrance  
>> into
>> another network once regular routing has exhausted the local network.
>
> Doesn't this allow an attacker to selectively DOS the bottleneck
> points by sending out requests for non-existant data?
>
> Evan Daniel

If we allow the requestor to specify which network they are trying to  
get to, then maybe (but the node still can rejectoverload like any  
other). I think it would work better to the negative; specify which  
networks *not* to route to, this would not only help on a reject of a  
network-gateway node, but it also lets nodes w/o a good routing table  
to use the same mechanism.

--
Robert Hailey



From evanbd at gmail.com  Fri Feb  1 17:20:36 2008
From: evanbd at gmail.com (Evan Daniel)
Date: Fri, 1 Feb 2008 12:20:36 -0500
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <BE97F36A-93F6-41D3-9032-2E8C37BE16E3@freenetproject.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<Prayer.1.0.16.0801301734100.93197@bells3.cs.ucl.ac.uk>
	<0E7521BC-7329-458D-ADAC-229BE691A6A2@freenetproject.org>
	<200801302249.42207.toad@amphibian.dyndns.org>
	<4f9383510801311648h354b8e41h98148553a26b6348@mail.gmail.com>
	<BE97F36A-93F6-41D3-9032-2E8C37BE16E3@freenetproject.org>
Message-ID: <4f9383510802010920h42edfb1j57e3ed7b8ab961f8@mail.gmail.com>

On Feb 1, 2008 12:00 PM, Robert Hailey <robert at freenetproject.org> wrote:
>
>
> On Jan 31, 2008, at 6:48 PM, Evan Daniel wrote:
>
> > On Jan 30, 2008 5:49 PM, Matthew Toseland
> > <toad at amphibian.dyndns.org> wrote:
> >
> >> You also need an escape-route mechanism - a way to find an entrance
> >> into
> >> another network once regular routing has exhausted the local network.
> >
> > Doesn't this allow an attacker to selectively DOS the bottleneck
> > points by sending out requests for non-existant data?
> >
> > Evan Daniel
>
> If we allow the requestor to specify which network they are trying to
> get to, then maybe (but the node still can rejectoverload like any
> other). I think it would work better to the negative; specify which
> networks *not* to route to, this would not only help on a reject of a
> network-gateway node, but it also lets nodes w/o a good routing table
> to use the same mechanism.

Even if the requestor can't specify a target network, I think it
works.  If the model is that the request is first routed within the
network, and if that fails it tries to find an escape route -- then
that "escape route" is a bottleneck (by definition).

The nodes using rejectoverload is insufficient, I think -- they'll
reject the attacker's requests and real requests with similar
probability, and so performance for real requests will degrade
substantially.  Now the attacker only needs resources comparable to
the bottlenecks; they don't even have to know where those bottlenecks
are in order to seriously degrade the network topology.

I'm not familiar enough with the details of the proposed ULPRs and how
USKs and Frost and the like check for new updates / messages, but it
seems possible that simple legitimate checks for new content would
have a similar effect.  Of course, failure tables would help a lot
with that case, but they wouldn't help against a malicious attacker.

Evan Daniel


From toad at amphibian.dyndns.org  Fri Feb  1 17:55:24 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Fri, 1 Feb 2008 17:55:24 +0000
Subject: [freenet-dev] Still getting timeouts
In-Reply-To: <0E15C02C-C332-4925-BB44-91EFA62E6E02@freenetproject.org>
References: <200801311441.39340.toad@amphibian.dyndns.org>
	<200801311943.43702.toad@amphibian.dyndns.org>
	<0E15C02C-C332-4925-BB44-91EFA62E6E02@freenetproject.org>
Message-ID: <200802011755.30323.toad@amphibian.dyndns.org>

On Thursday 31 January 2008 21:21, Robert Hailey wrote:
> 
> >>> Oh?
> >>
> >> Every time I look at my opennet peers, I *always* have at least two
> >> with pings greater than 2 seconds. Right now, one with 4.5 secs, and
> >> one with 8.9 (the rest are sane).
> >
> > Hmmm. Doesn't happen for me, although I only have 4 or 5 opennet  
> > peers.
> >
> > It seems extraordinarily unlikely that this is real - either this is  
> > a stats
> > bug, or a message layer bug.
> 
> And if it is a message layer bug, that means it may be directly  
> related to the timeouts.

It's also possible it's just due to nodes being hideously overloaded. Which 
can be due to several causes:
- The startup spike. Which can last a long time because we have no request 
resuming.
- Out of memory causing continual garbage collection. Several users have 
reported that this happens after 12 hours or so of uptime.
- ....
> 
> >>>> In the past while examining the throttle controls, I have suspected
> >>>> that (with priority queues) the "90-seconds at full throttle"
> >>>> constant
> >>>> might actually reduce to taking on too many concurrent chk  
> >>>> transfers
> >>>> for them all to complete on time.
> >>>
> >>> Why? IIRC we include a fudge factor in that calculation, admittedly
> >>> it isn't
> >>> very accurate and should be made more so by using stats on bandwidth
> >>> usage...
> >>
> >> Just that the CHKs all use the same throttle, so they all throttle-
> >> down when we accept another CHK transfer.
> >
> > Well sure, but if the mechanism is working we won't accept enough to  
> > be a
> > problem.
> 
> I'm not saying this is an issue, but when a node is busy the 90-second- 
> standard might actually make the average chk transfer time (over long  
> distances) always exactly 90 seconds (through the busiest node). Since  
> the transfer timeout is 120 seconds, this actually leaves only 30  
> seconds to accumulate acceptable latency; by your previous value of 30  
> hops, this means one second per hop (1/2 ping time plus coalescing  
> delay?).

Hmmm. Perhaps. So we should reduce the 90 seconds to say 60 seconds? That 
might cut actual bandwidth usage...
> 
> Or else, how many transfers are aborted because nodes disconnect, and  
> if they would succeed if the target transfer time was shorter than 90  
> seconds? Particularly as the CHK is streaming, that the traffic up  
> unto the abort is wasted (50% payload?).

Hmmm. IIRC that is fatal?
> 
> >>>>> Do timeouts show up in simulation?
> >>>>
> >>>> I don't normally watch for them, I've started a new run with  
> >>>> Accepted
> >>>> & Fatal request timeouts being logged. So far nothing.
> >>>
> >>> Ok.
> >>
> >> After running the simulator for two hours w/ ten nodes, I spot  
> >> exactly
> >> one Accepted timeout (17 minutes into the simulation).
> >>
> >> So the answer is yes... timeouts still occur in the simulator.
> >
> > Suggests a messaging bug, although it's possible it's an artifact of  
> > java's
> > lack of thread priorities on *nix (i.e. cpu issues).
> 
> I would be more inclined to think a messaging bug, it is a beefy  
> machine and it occurred some time into the simulation.
> 
> >>>>> What can we do to debug this?
> >>>>
> >>>> Probably:
> >>>> (1) a simulated high-ping times seen in the public network at about
> >>>> the same rate,
> >>>
> >>> You mean bugs cause high ping times and high ping times cause
> >>> timeouts?
> >>>
> >>>> (2) a message/link layer stress test complete with rekeying/
> >>>> disconnects/and [busy/not-busy] spikes
> >>>
> >>> This would be a good idea, I dunno how much work would be involved?
> >>>
> >>> What can I usefully work on in this area? AFAICS:
> >>> - The window-grows-while-unused bug.
> >>> - More accurate bandwidth liability limiting.
> >>> - Debug the not-forwarded detection and make assumeNATed false by
> >>> default.
> >>> (Reduce baseload bandwidth usage).
> >>>
> >>> Anything else? You want to take any of these on?
> >>
> >> I don't think I can take on a big project right now.
> >
> > Is there anything I can do?
> 
> I am not familiar with the window-grows-while-unused bug, and am not  
> working on/debugging the message layer right now. It's up to you.

I will fix the window-grows-while-unused bug.

W.r.t. messaging layer bugs, please explain how to reproduce your simulation; 
commit whatever source is needed.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080201/6ae6c9d0/attachment.pgp 

From toad at amphibian.dyndns.org  Fri Feb  1 17:57:50 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Fri, 1 Feb 2008 17:57:50 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <4f9383510802010920h42edfb1j57e3ed7b8ab961f8@mail.gmail.com>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<BE97F36A-93F6-41D3-9032-2E8C37BE16E3@freenetproject.org>
	<4f9383510802010920h42edfb1j57e3ed7b8ab961f8@mail.gmail.com>
Message-ID: <200802011757.51113.toad@amphibian.dyndns.org>

On Friday 01 February 2008 17:20, Evan Daniel wrote:
> On Feb 1, 2008 12:00 PM, Robert Hailey <robert at freenetproject.org> wrote:
> >
> >
> > On Jan 31, 2008, at 6:48 PM, Evan Daniel wrote:
> >
> > > On Jan 30, 2008 5:49 PM, Matthew Toseland
> > > <toad at amphibian.dyndns.org> wrote:
> > >
> > >> You also need an escape-route mechanism - a way to find an entrance
> > >> into
> > >> another network once regular routing has exhausted the local network.
> > >
> > > Doesn't this allow an attacker to selectively DOS the bottleneck
> > > points by sending out requests for non-existant data?
> > >
> > > Evan Daniel
> >
> > If we allow the requestor to specify which network they are trying to
> > get to, then maybe (but the node still can rejectoverload like any
> > other). I think it would work better to the negative; specify which
> > networks *not* to route to, this would not only help on a reject of a
> > network-gateway node, but it also lets nodes w/o a good routing table
> > to use the same mechanism.
> 
> Even if the requestor can't specify a target network, I think it
> works.  If the model is that the request is first routed within the
> network, and if that fails it tries to find an escape route -- then
> that "escape route" is a bottleneck (by definition).
> 
> The nodes using rejectoverload is insufficient, I think -- they'll
> reject the attacker's requests and real requests with similar
> probability, and so performance for real requests will degrade
> substantially.  Now the attacker only needs resources comparable to
> the bottlenecks; they don't even have to know where those bottlenecks
> are in order to seriously degrade the network topology.
> 
> I'm not familiar enough with the details of the proposed ULPRs and how
> USKs and Frost and the like check for new updates / messages, but it
> seems possible that simple legitimate checks for new content would
> have a similar effect.  Of course, failure tables would help a lot
> with that case, but they wouldn't help against a malicious attacker.

Could ULPRs help to resolve it? Would it be possible to estimate the demand 
for a key (in a way which doesn't favour single nodes that constantly 
rerequest, and is biased by links so that an attacker could only attack 
proportionately to the number of connections he has), in order to decide 
which requests to let through?
> 
> Evan Daniel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080201/f39d7371/attachment.pgp 

From toad at amphibian.dyndns.org  Fri Feb  1 17:58:33 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Fri, 1 Feb 2008 17:58:33 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <9B1249F7-AED1-470C-8F55-6FAE33C45B8E@freenetproject.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<8ECAA540-7437-473B-9573-19E3955B9980@freenetproject.org>
	<9B1249F7-AED1-470C-8F55-6FAE33C45B8E@freenetproject.org>
Message-ID: <200802011758.33216.toad@amphibian.dyndns.org>

On Friday 01 February 2008 17:00, Robert Hailey wrote:
> 
> On Jan 31, 2008, at 11:03 AM, Robert Hailey wrote:
> 
> >> How do you authenticate the routed pings, to prevent an attacker from
> >> replying on behalf of another node?
> >
> > Excellent question. Surely the "true/false" response of present is
> > woefully inadequate. Since we have a direct connection to the peer
> > that we are pinging a challange-and-response mechanism is easy, no?
> >
> > Consider node "B" who is between "A" & "C" (A-B-C). He tells "C" a UID
> > & Secret [a randomly generated long?], and "C" stores that secret/uid
> > as part of our peernode record. Node "B" then sends node "A" a routed
> > ping with the same UID, and if node "A" returns the pong with the
> > correct secret it is a success.
> 
> I was supposing that these pings would be sent at less-than-max htl  
> (since we are not searching the network but doing a connectivity  
> test), but wouldn't that possibly allow an attacker to learn who your  
> peers are?
> 
> That is, if an attacker has a node connected to your node and your  
> peers node, he could put together the ping from yours, the reply from  
> your peer, plus the fact that the reply comes from a node of the same  
> location as the ping, and be reasonably sure he is your peer. Whereas  
> with the probabilistic decrement at the real maxHTL, they could not be  
> nearly so sure.

He can already know this, because of swapping.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080201/470d99a7/attachment.pgp 

From robert at freenetproject.org  Fri Feb  1 18:25:32 2008
From: robert at freenetproject.org (Robert Hailey)
Date: Fri, 1 Feb 2008 12:25:32 -0600
Subject: [freenet-dev] Still getting timeouts
In-Reply-To: <200802011755.30323.toad@amphibian.dyndns.org>
References: <200801311441.39340.toad@amphibian.dyndns.org>
	<200801311943.43702.toad@amphibian.dyndns.org>
	<0E15C02C-C332-4925-BB44-91EFA62E6E02@freenetproject.org>
	<200802011755.30323.toad@amphibian.dyndns.org>
Message-ID: <8D527369-6E27-4BB6-8DCD-91E06DE838CF@freenetproject.org>


On Feb 1, 2008, at 11:55 AM, Matthew Toseland wrote:

>>> Is there anything I can do?
>>
>> I am not familiar with the window-grows-while-unused bug, and am not
>> working on/debugging the message layer right now. It's up to you.
>
> I will fix the window-grows-while-unused bug.
>
> W.r.t. messaging layer bugs, please explain how to reproduce your  
> simulation;
> commit whatever source is needed.

The only modifications beyond previous commits are quite minor;  
committed in r17462.

To see timeouts in the default logging output I increased them to  
'error' as depicted below (not committed), and ran that code with this  
command:

java -Xmx512m -cp freenet-ext.jar:freenet-cvs-snapshot.jar  
freenet.node.simulator.RealNodeRequestInsertTest

--
Robert Hailey

Index: src/freenet/node/RequestSender.java
===================================================================
--- src/freenet/node/RequestSender.java	(revision 17413)
+++ src/freenet/node/RequestSender.java	(working copy)
@@ -268,7 +268,7 @@
                  }

              	if(msg == null) {
- if(logMINOR) Logger.minor(this, "Timeout waiting for Accepted");
+ Logger.error(this, "Timeout waiting for Accepted");
              		// Timeout waiting for Accepted
              		next.localRejectedOverload("AcceptedTimeout");
              		forwardRejectedOverload();
@@ -341,7 +341,7 @@
              	if(logMINOR) Logger.minor(this, "second part got "+msg);

              	if(msg == null) {
- Logger.normal(this, "request fatal-timeout (null) after accept  
("+gotMessages+" messages; last="+lastMessage+")");
+ Logger.error(this, "request fatal-timeout (null) after accept  
("+gotMessages+" messages; last="+lastMessage+")");
              		// Fatal timeout
              		next.localRejectedOverload("FatalTimeout");
              		forwardRejectedOverload();


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://emu.freenetproject.org/pipermail/devl/attachments/20080201/9241ae53/attachment.htm 

From evanbd at gmail.com  Fri Feb  1 18:25:41 2008
From: evanbd at gmail.com (Evan Daniel)
Date: Fri, 1 Feb 2008 13:25:41 -0500
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <200802011757.51113.toad@amphibian.dyndns.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<BE97F36A-93F6-41D3-9032-2E8C37BE16E3@freenetproject.org>
	<4f9383510802010920h42edfb1j57e3ed7b8ab961f8@mail.gmail.com>
	<200802011757.51113.toad@amphibian.dyndns.org>
Message-ID: <4f9383510802011025x63780eaco133de96ae2c577a6@mail.gmail.com>

On Feb 1, 2008 12:57 PM, Matthew Toseland <toad at amphibian.dyndns.org> wrote:
> > Even if the requestor can't specify a target network, I think it
> > works.  If the model is that the request is first routed within the
> > network, and if that fails it tries to find an escape route -- then
> > that "escape route" is a bottleneck (by definition).
> >
> > The nodes using rejectoverload is insufficient, I think -- they'll
> > reject the attacker's requests and real requests with similar
> > probability, and so performance for real requests will degrade
> > substantially.  Now the attacker only needs resources comparable to
> > the bottlenecks; they don't even have to know where those bottlenecks
> > are in order to seriously degrade the network topology.
> >
> > I'm not familiar enough with the details of the proposed ULPRs and how
> > USKs and Frost and the like check for new updates / messages, but it
> > seems possible that simple legitimate checks for new content would
> > have a similar effect.  Of course, failure tables would help a lot
> > with that case, but they wouldn't help against a malicious attacker.
>
> Could ULPRs help to resolve it? Would it be possible to estimate the demand
> for a key (in a way which doesn't favour single nodes that constantly
> rerequest, and is biased by links so that an attacker could only attack
> proportionately to the number of connections he has), in order to decide
> which requests to let through?

I think ULPRs will do a good job of preventing legitimate traffic from
creating such an effect.  A malicious attacker, however, would have no
reason to repeat keys, so any technique that simply tries to make
re-requests more efficient would have no effect.  Biasing on
popularity is probably a good thing, and if it can be done in a
relatively attack-proof manner, might be the solution.

Do we have any understanding of how well network clusters will
correlate with content clusters?  That is, if there are effectively
two networks, especially if they result from cultural and language
barriers, to what extent will the two sides be uninterested in
communicating with each other?  I think having a ballpark answer to
that question will go a long way in determining how big a problem this
really is, and also what sort of solutions might be appropriate.  Of
course, it sounds hard to answer :)

Evan Daniel


From robert at freenetproject.org  Fri Feb  1 18:33:33 2008
From: robert at freenetproject.org (Robert Hailey)
Date: Fri, 1 Feb 2008 12:33:33 -0600
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <200802011757.51113.toad@amphibian.dyndns.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<BE97F36A-93F6-41D3-9032-2E8C37BE16E3@freenetproject.org>
	<4f9383510802010920h42edfb1j57e3ed7b8ab961f8@mail.gmail.com>
	<200802011757.51113.toad@amphibian.dyndns.org>
Message-ID: <98822195-F3C0-4B9A-93E9-EAFF4A196AF6@freenetproject.org>


On Feb 1, 2008, at 11:57 AM, Matthew Toseland wrote:

>> I'm not familiar enough with the details of the proposed ULPRs and  
>> how
>> USKs and Frost and the like check for new updates / messages, but it
>> seems possible that simple legitimate checks for new content would
>> have a similar effect.  Of course, failure tables would help a lot
>> with that case, but they wouldn't help against a malicious attacker.
>
> Could ULPRs help to resolve it? Would it be possible to estimate the  
> demand
> for a key (in a way which doesn't favour single nodes that constantly
> rerequest, and is biased by links so that an attacker could only  
> attack
> proportionately to the number of connections he has), in order to  
> decide
> which requests to let through?

I guess you could add to the failure table which distinct links have  
requested a given key, and be more likely to let those through with  
more links (once the failure is timed out). It doesn't seem very  
granular, as I would suppose (in a small world network) that a re- 
request from a non-peer node would be very likely to show up on a  
different connection.

--
Robert Hailey

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://emu.freenetproject.org/pipermail/devl/attachments/20080201/d8b0ca4d/attachment.htm 

From sback at sback.it  Fri Feb  1 18:33:53 2008
From: sback at sback.it (Alberto Bacchelli)
Date: Fri, 01 Feb 2008 19:33:53 +0100
Subject: [freenet-dev] BitArray bug
Message-ID: <47A36611.2020904@sback.it>

Hi,

 trying some automatic test case generator on the code, I found that
there is a bug in freenet.support.BitArray.
It is not a problem in the Freenet code, but it seems a Java problem
(maybe only with the VM version I am using which is Java(TM) SE Runtime
Environment (build 1.6.0_03-b05) ).

if you run this test:

public void testConstructorThrowsEOFException1() throws Throwable {
        DataInputStream dis = new DataInputStream(new
ByteArrayInputStream("testString".getBytes()));
        try {
            new BitArray(dis);
            fail("Expected EOFException to be thrown");
        } catch (EOFException ex) {
            assertEquals("ex.getClass()", EOFException.class,
ex.getClass());
            assertThrownBy(DataInputStream.class, ex);
            assertEquals("dis.available()", 0, dis.available());
        }
}

you will probably get a java.lang.OutOfMemoryError.

This is caused by a bad value returned by readInt() in the following code:


    public BitArray(DataInputStream dis) throws IOException {
        _size = dis.readInt();
        _bits = new byte[(_size / 8) + (_size % 8 == 0 ? 0 : 1)];
        dis.readFully(_bits);
    }  

the simpliest way to solve this issue imho is to find a way to avoid
using readInt().

Sback


From robert at freenetproject.org  Fri Feb  1 18:51:59 2008
From: robert at freenetproject.org (Robert Hailey)
Date: Fri, 1 Feb 2008 12:51:59 -0600
Subject: [freenet-dev] BitArray bug
In-Reply-To: <47A36611.2020904@sback.it>
References: <47A36611.2020904@sback.it>
Message-ID: <4955D73F-5AD8-43E1-BD61-511528D79B4A@freenetproject.org>


On Feb 1, 2008, at 12:33 PM, Alberto Bacchelli wrote:

> Hi,
>
> trying some automatic test case generator on the code, I found that
> there is a bug in freenet.support.BitArray.
> It is not a problem in the Freenet code, but it seems a Java problem
> (maybe only with the VM version I am using which is Java(TM) SE  
> Runtime
> Environment (build 1.6.0_03-b05) ).
>
> if you run this test:
>
> public void testConstructorThrowsEOFException1() throws Throwable {
>        DataInputStream dis = new DataInputStream(new
> ByteArrayInputStream("testString".getBytes()));
>        try {
>            new BitArray(dis);
>            fail("Expected EOFException to be thrown");
>        } catch (EOFException ex) {
>            assertEquals("ex.getClass()", EOFException.class,
> ex.getClass());
>            assertThrownBy(DataInputStream.class, ex);
>            assertEquals("dis.available()", 0, dis.available());
>        }
> }
>
> you will probably get a java.lang.OutOfMemoryError.
>
> This is caused by a bad value returned by readInt() in the following  
> code:
>
>
>    public BitArray(DataInputStream dis) throws IOException {
>        _size = dis.readInt();
>        _bits = new byte[(_size / 8) + (_size % 8 == 0 ? 0 : 1)];
>        dis.readFully(_bits);
>    }
>
> the simpliest way to solve this issue imho is to find a way to avoid
> using readInt().
>
> Sback

I'm not really sure this is so much a bug as garbage-in-garbage-out.

By the code provided, the size for the bitarray is given first, then  
the raw data; it just so happens that when you translate the first  
several bytes of "testString" into an integer, it comes out to be very  
large number (more bytes than are available for allocation). Note that  
there *is* a legitimate bitarray data stream which will begin with  
"testString".asBytes(), it is probably just too large for practical use.

--
Robert Hailey



From toad at amphibian.dyndns.org  Fri Feb  1 18:57:34 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Fri, 1 Feb 2008 18:57:34 +0000
Subject: [freenet-dev] BitArray bug
In-Reply-To: <4955D73F-5AD8-43E1-BD61-511528D79B4A@freenetproject.org>
References: <47A36611.2020904@sback.it>
	<4955D73F-5AD8-43E1-BD61-511528D79B4A@freenetproject.org>
Message-ID: <200802011857.40006.toad@amphibian.dyndns.org>

On Friday 01 February 2008 18:51, Robert Hailey wrote:
> 
> On Feb 1, 2008, at 12:33 PM, Alberto Bacchelli wrote:
> 
> > Hi,
> >
> > trying some automatic test case generator on the code, I found that
> > there is a bug in freenet.support.BitArray.
> > It is not a problem in the Freenet code, but it seems a Java problem
> > (maybe only with the VM version I am using which is Java(TM) SE  
> > Runtime
> > Environment (build 1.6.0_03-b05) ).
> >
> > if you run this test:
> >
> > public void testConstructorThrowsEOFException1() throws Throwable {
> >        DataInputStream dis = new DataInputStream(new
> > ByteArrayInputStream("testString".getBytes()));
> >        try {
> >            new BitArray(dis);
> >            fail("Expected EOFException to be thrown");
> >        } catch (EOFException ex) {
> >            assertEquals("ex.getClass()", EOFException.class,
> > ex.getClass());
> >            assertThrownBy(DataInputStream.class, ex);
> >            assertEquals("dis.available()", 0, dis.available());
> >        }
> > }
> >
> > you will probably get a java.lang.OutOfMemoryError.
> >
> > This is caused by a bad value returned by readInt() in the following  
> > code:
> >
> >
> >    public BitArray(DataInputStream dis) throws IOException {
> >        _size = dis.readInt();
> >        _bits = new byte[(_size / 8) + (_size % 8 == 0 ? 0 : 1)];
> >        dis.readFully(_bits);
> >    }
> >
> > the simpliest way to solve this issue imho is to find a way to avoid
> > using readInt().
> >
> > Sback
> 
> I'm not really sure this is so much a bug as garbage-in-garbage-out.
> 
> By the code provided, the size for the bitarray is given first, then  
> the raw data; it just so happens that when you translate the first  
> several bytes of "testString" into an integer, it comes out to be very  
> large number (more bytes than are available for allocation). Note that  
> there *is* a legitimate bitarray data stream which will begin with  
> "testString".asBytes(), it is probably just too large for practical use.

Do we use this anywhere, that is the question?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080201/338424eb/attachment.pgp 

From robert at freenetproject.org  Fri Feb  1 19:28:09 2008
From: robert at freenetproject.org (Robert Hailey)
Date: Fri, 1 Feb 2008 13:28:09 -0600
Subject: [freenet-dev] BitArray bug
In-Reply-To: <200802011857.40006.toad@amphibian.dyndns.org>
References: <47A36611.2020904@sback.it>
	<4955D73F-5AD8-43E1-BD61-511528D79B4A@freenetproject.org>
	<200802011857.40006.toad@amphibian.dyndns.org>
Message-ID: <A07D7E43-AD4B-4986-B604-B8C0FB23374B@freenetproject.org>


On Feb 1, 2008, at 12:57 PM, Matthew Toseland wrote:

> On Friday 01 February 2008 18:51, Robert Hailey wrote:
>>
>> I'm not really sure this is so much a bug as garbage-in-garbage-out.
>>
>> By the code provided, the size for the bitarray is given first, then
>> the raw data; it just so happens that when you translate the first
>> several bytes of "testString" into an integer, it comes out to be  
>> very
>> large number (more bytes than are available for allocation). Note  
>> that
>> there *is* a legitimate bitarray data stream which will begin with
>> "testString".asBytes(), it is probably just too large for practical  
>> use.
>
> Do we use this anywhere, that is the question?

Yes, when reading a DMT.packetTransmit; It is  
"freenet.support.BitArray", but it has the same issue.

We could either put a hard-coded cap in BitArray (ugly), or make  
another constructor with a maximum allowable size (from Serializer).

Fixed in r17464.

--
Robert Hailey



From robert at freenetproject.org  Fri Feb  1 20:26:40 2008
From: robert at freenetproject.org (Robert Hailey)
Date: Fri, 1 Feb 2008 14:26:40 -0600
Subject: [freenet-dev] Still getting timeouts
In-Reply-To: <200802011755.30323.toad@amphibian.dyndns.org>
References: <200801311441.39340.toad@amphibian.dyndns.org>
	<200801311943.43702.toad@amphibian.dyndns.org>
	<0E15C02C-C332-4925-BB44-91EFA62E6E02@freenetproject.org>
	<200802011755.30323.toad@amphibian.dyndns.org>
Message-ID: <11E00EE2-48D2-4C7E-B776-B26D0F907FB6@freenetproject.org>


On Feb 1, 2008, at 11:55 AM, Matthew Toseland wrote:

>> I'm not saying this is an issue, but when a node is busy the 90- 
>> second-
>> standard might actually make the average chk transfer time (over long
>> distances) always exactly 90 seconds (through the busiest node).  
>> Since
>> the transfer timeout is 120 seconds, this actually leaves only 30
>> seconds to accumulate acceptable latency; by your previous value of  
>> 30
>> hops, this means one second per hop (1/2 ping time plus coalescing
>> delay?).
>
> Hmmm. Perhaps. So we should reduce the 90 seconds to say 60 seconds?  
> That
> might cut actual bandwidth usage...

If my understanding is right, that would only be the case during the  
transition period (other nodes feed us at the 90 rate, but we expect  
the 60). This would be a good reason to not drastically cut it in one  
build (like, to 30; as for the time nodes would be running at 1/3  
bandwidth).

>> Or else, how many transfers are aborted because nodes disconnect, and
>> if they would succeed if the target transfer time was shorter than 90
>> seconds? Particularly as the CHK is streaming, that the traffic up
>> unto the abort is wasted (50% payload?).
>
> Hmmm. IIRC that is fatal?

For the request, yes. But my point is that we have already transferred  
a lot of data which then does not count as payload, right? And it will  
probably be retried almost immediately; there is no stat AFAICS which  
represents request failure percentages (except Success, how many  
TRANSFER_FAILED, ACCEPT_TIMEOUT, FATAL, etc).

--
Robert Hailey

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://emu.freenetproject.org/pipermail/devl/attachments/20080201/e59b3adf/attachment.htm 

From toad at amphibian.dyndns.org  Fri Feb  1 22:42:40 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Fri, 1 Feb 2008 22:42:40 +0000
Subject: [freenet-dev] Still getting timeouts
In-Reply-To: <11E00EE2-48D2-4C7E-B776-B26D0F907FB6@freenetproject.org>
References: <200801311441.39340.toad@amphibian.dyndns.org>
	<200802011755.30323.toad@amphibian.dyndns.org>
	<11E00EE2-48D2-4C7E-B776-B26D0F907FB6@freenetproject.org>
Message-ID: <200802012242.41095.toad@amphibian.dyndns.org>

On Friday 01 February 2008 20:26, Robert Hailey wrote:
> 
> On Feb 1, 2008, at 11:55 AM, Matthew Toseland wrote:
> 
> >> I'm not saying this is an issue, but when a node is busy the 90- 
> >> second-
> >> standard might actually make the average chk transfer time (over long
> >> distances) always exactly 90 seconds (through the busiest node).  
> >> Since
> >> the transfer timeout is 120 seconds, this actually leaves only 30
> >> seconds to accumulate acceptable latency; by your previous value of  
> >> 30
> >> hops, this means one second per hop (1/2 ping time plus coalescing
> >> delay?).
> >
> > Hmmm. Perhaps. So we should reduce the 90 seconds to say 60 seconds?  
> > That
> > might cut actual bandwidth usage...
> 
> If my understanding is right, that would only be the case during the  
> transition period (other nodes feed us at the 90 rate, but we expect  
> the 60). This would be a good reason to not drastically cut it in one  
> build (like, to 30; as for the time nodes would be running at 1/3  
> bandwidth).

Well even if this is true, one second per hop is plenty.
> 
> >> Or else, how many transfers are aborted because nodes disconnect, and
> >> if they would succeed if the target transfer time was shorter than 90
> >> seconds? Particularly as the CHK is streaming, that the traffic up
> >> unto the abort is wasted (50% payload?).
> >
> > Hmmm. IIRC that is fatal?
> 
> For the request, yes. But my point is that we have already transferred  
> a lot of data which then does not count as payload, right? 

For stats purposes it counts as payload.

> And it will   
> probably be retried almost immediately; there is no stat AFAICS which  
> represents request failure percentages (except Success, how many  
> TRANSFER_FAILED, ACCEPT_TIMEOUT, FATAL, etc).

Do we need one?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080201/0b6219a2/attachment.pgp 

From robert at freenetproject.org  Fri Feb  1 23:09:26 2008
From: robert at freenetproject.org (Robert Hailey)
Date: Fri, 1 Feb 2008 17:09:26 -0600
Subject: [freenet-dev] r17466 - trunk/freenet/src/freenet/support
In-Reply-To: <20080201230625.7F2AC3921FF@freenetproject.org>
References: <20080201230625.7F2AC3921FF@freenetproject.org>
Message-ID: <DB006F93-C821-4234-AA41-91203A6A39F4@freenetproject.org>


To the best of my knowledge, the only BitArray transported is 31 bits.  
I thought 128 was generous! The only thing increasing it will do is  
let neighbor peers make us queue large packets, no?

--
Robert Hailey


On Feb 1, 2008, at 5:06 PM, toad at freenetproject.org wrote:

> Author: toad
> Date: 2008-02-01 23:06:25 +0000 (Fri, 01 Feb 2008)
> New Revision: 17466
>
> Modified:
>   trunk/freenet/src/freenet/support/Serializer.java
> Log:
> Less aggressive size limit
>
> Modified: trunk/freenet/src/freenet/support/Serializer.java
> ===================================================================
> --- trunk/freenet/src/freenet/support/Serializer.java	2008-02-01  
> 20:53:20 UTC (rev 17465)
> +++ trunk/freenet/src/freenet/support/Serializer.java	2008-02-01  
> 23:06:25 UTC (rev 17466)
> @@ -42,7 +42,7 @@
> public class Serializer {
>
>     public static final String VERSION = "$Id: Serializer.java,v 1.5  
> 2005/09/15 18:16:04 amphibian Exp $";
> -	public static final int MAX_BITARRAY_SIZE = 128;
> +	public static final int MAX_BITARRAY_SIZE = 2048*8;
>
> 	public static List readListFromDataInputStream(Class elementType,  
> DataInputStream dis) throws IOException {
> 		LinkedList ret = new LinkedList();
>



From toad at amphibian.dyndns.org  Sat Feb  2 13:13:15 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 2 Feb 2008 13:13:15 +0000
Subject: [freenet-dev] r17466 - trunk/freenet/src/freenet/support
In-Reply-To: <DB006F93-C821-4234-AA41-91203A6A39F4@freenetproject.org>
References: <20080201230625.7F2AC3921FF@freenetproject.org>
	<DB006F93-C821-4234-AA41-91203A6A39F4@freenetproject.org>
Message-ID: <200802021313.15555.toad@amphibian.dyndns.org>

On Friday 01 February 2008 23:09, Robert Hailey wrote:
> 
> To the best of my knowledge, the only BitArray transported is 31 bits.  
> I thought 128 was generous! The only thing increasing it will do is  
> let neighbor peers make us queue large packets, no?

It will prevent future problems if we do use larger bitarrays. No message 
should be over 2KB, and it's not a big deal for memory allocation, so 2KB is 
reasonable.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080202/2b1f79d2/attachment.pgp 

From toad at amphibian.dyndns.org  Sat Feb  2 15:42:24 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Sat, 2 Feb 2008 15:42:24 +0000
Subject: [freenet-dev] Freenet 0.7 build 1108
Message-ID: <200802021542.29554.toad@amphibian.dyndns.org>

Freenet 0.7 build 1108 is now available. Please upgrade, it is mandatory on 
Tuesday. This build includes lots of changes, bugfixes and improvements:
- Some memory optimisations in the client layer code (the client layer is 
responsible for most of Freenet's memory usage if you have requests/inserts 
queued).
- Improvements to the logic that decides when to run the IP detection plugins 
(JSTUN and UPnP).
- Minor messaging fixes.
- Make load limiting react faster to changing ping times.
- Minor changes to the web interface.
- Try to prevent CPU starvation on startup by yielding in heavy threads 
(unfortunately thread priorities don't work in java on non-windows).
- Minor CPU optimisation in routing code.
- Fix swapping limiting logic.
- Re-implement probe requests. These are a special kind of request that helps 
us to understand how well routing is working.
- Code cleanups
- XMLSpider: Bigger sub-indexes, but with a size limit of 256kB before 
compression.

Please report any bugs you find to the bug tracker at 
https://bugs.freenetproject.org , or Frost. Thanks.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080202/4da6c0a8/attachment.pgp 

From m.rogers at cs.ucl.ac.uk  Sat Feb  2 16:24:32 2008
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Sat, 02 Feb 2008 16:24:32 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<Prayer.1.0.16.0801301524240.92105@bells3.cs.ucl.ac.uk>
	<91F6684A-8661-4E4D-8512-35F3D1DE0316@freenetproject.org>
	<Prayer.1.0.16.0801301734100.93197@bells3.cs.ucl.ac.uk>
	<0E7521BC-7329-458D-ADAC-229BE691A6A2@freenetproject.org>
	<Prayer.1.0.16.0801311133340.98065@bells3.cs.ucl.ac.uk>
	<8ECAA540-7437-473B-9573-19E3955B9980@freenetproject.org>
	<Prayer.1.0.16.0801311757460.913@bells3.cs.ucl.ac.uk>
	<73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
Message-ID: <47A49940.9050802@cs.ucl.ac.uk>

Robert Hailey wrote:
> Unless [in a later implementation] DNFs (maybe weighed against HTL) 
> become an indication of a separate network. Then it would do precisely that.

DNFs would detect a "black hole" attacker - what about an attacker who 
just wants to monitor a region of the key space or censor a few selected 
keys?

(Again, this criticism isn't specific to your proposal and I'm not 
saying your proposal creates this problem.)

>> what am I missing?
> 
> That following the algorithm will eventually color the entire 
> network/subnetwork/dungeon with the same network id, which is 
> presumed necessary for any sort of distant routing.

OK, I think I see what you mean now - the network IDs aren't just 
private labels used by each node to colour its private map of the 
network - they're used for routing, so there needs to be a rough 
consensus about which label applies to which area of the network.

I'm still not quite sure about how that consensus is reached. Agreeing 
on an ID for the local subnet doesn't seem to be a problem, but what 
happens if (for example) an attacker creates a subnet with the same ID 
as an existing subnet? Any searches that have visited the attacker's 
subnet and failed will be marked "don't look in subnet X" and therefore 
won't visit the 'real' subnet X, right?

And I guess the list of failed subnets can only be updated by the 
requester - otherwise an attacker could return a DNF saying "already 
tried subnets X, Y, Z"?

> So now we'll tell our peer's 
> that we can reach network 'w' (through this peer w/ that success). Even 
> if it is a Sybil-simulated subnetwork, routing to it (and perhaps wether 
> or not we advertise it to our peers) will probably end up being related 
> to success-probability; they could only become relied upon by providing 
> good data (not censoring), why not use the Sybil net for storage, then? 

This sounds promising. I'm not sure all attackers will be unreliable, 
though. An attacker might diligently store 99% of keys but censor the 
other 1% - but maybe FEC makes that a non-issue. Or the attacker might 
provide a fast, reliable storage service in order to monitor as much 
traffic as possible - but maybe that's unavoidable in opennet.

How do you measure success for inserts? Or maybe you can just measure 
success for requests and assume that inserts must be succeeding if 
requests are succeeding?

Cheeers,
Michael


From m.rogers at cs.ucl.ac.uk  Sat Feb  2 16:31:53 2008
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Sat, 02 Feb 2008 16:31:53 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <4f9383510802010920h42edfb1j57e3ed7b8ab961f8@mail.gmail.com>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<Prayer.1.0.16.0801301734100.93197@bells3.cs.ucl.ac.uk>
	<0E7521BC-7329-458D-ADAC-229BE691A6A2@freenetproject.org>
	<200801302249.42207.toad@amphibian.dyndns.org>
	<4f9383510801311648h354b8e41h98148553a26b6348@mail.gmail.com>
	<BE97F36A-93F6-41D3-9032-2E8C37BE16E3@freenetproject.org>
	<4f9383510802010920h42edfb1j57e3ed7b8ab961f8@mail.gmail.com>
Message-ID: <47A49AF9.3080907@cs.ucl.ac.uk>

Evan Daniel wrote:
> The nodes using rejectoverload is insufficient, I think -- they'll
> reject the attacker's requests and real requests with similar
> probability, and so performance for real requests will degrade
> substantially.  Now the attacker only needs resources comparable to
> the bottlenecks; they don't even have to know where those bottlenecks
> are in order to seriously degrade the network topology.

Are you sure RejectedOverload isn't adequate? If a gateway node becomes 
overloaded, the other nodes in both subnets will route around it, so 
traffic will stop crossing between the subnets but routing within each 
subnet should continue to work. AFAICS it would only be a problem if the 
gateway node was unavoidable in one or both of the subnets (eg ring 
topology with no shortcuts).

Cheers,
Michael


From m.rogers at cs.ucl.ac.uk  Mon Feb  4 01:45:47 2008
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Mon, 04 Feb 2008 01:45:47 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <4f9383510802011025x63780eaco133de96ae2c577a6@mail.gmail.com>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<BE97F36A-93F6-41D3-9032-2E8C37BE16E3@freenetproject.org>
	<4f9383510802010920h42edfb1j57e3ed7b8ab961f8@mail.gmail.com>
	<200802011757.51113.toad@amphibian.dyndns.org>
	<4f9383510802011025x63780eaco133de96ae2c577a6@mail.gmail.com>
Message-ID: <47A66E4B.6030305@cs.ucl.ac.uk>

Evan Daniel wrote:
> Do we have any understanding of how well network clusters will
> correlate with content clusters?  That is, if there are effectively
> two networks, especially if they result from cultural and language
> barriers, to what extent will the two sides be uninterested in
> communicating with each other?  I think having a ballpark answer to
> that question will go a long way in determining how big a problem this
> really is, and also what sort of solutions might be appropriate.  Of
> course, it sounds hard to answer :)

But all the interesting questions are hard to answer. :-) I asked a 
friend who works on computational trust and he pointed me to some papers 
by Jennifer Golbeck:

http://www.cs.umd.edu/~golbeck/publications.shtml

The short answer appears to be "somewhat".

Cheers,
Michael


From toad at amphibian.dyndns.org  Mon Feb  4 12:42:16 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 4 Feb 2008 12:42:16 +0000
Subject: [freenet-dev] What about Bazaar?
Message-ID: <200802041242.21043.toad@amphibian.dyndns.org>

Something I noticed this month's Linux Magazine. We've been looking for a good 
DVCS for a while: What about Bazaar?
- Publishes to static content, so should be easy to adapt for Freenet. 
(Hopefully - USK versioning wouldn't be a problem for a DVCS assuming there 
are repository version numbers of some kind to use as a basis for diffs???).
- GPL2+ (dual license with a closed source friendly license).
- Focus on user friendliness (sponsored by Canonical).

http://bazaar-vcs.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080204/a5f59a65/attachment.pgp 

From nextgens at freenetproject.org  Mon Feb  4 14:24:33 2008
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=)
Date: Mon, 4 Feb 2008 15:24:33 +0100
Subject: [freenet-dev] What about Bazaar?
In-Reply-To: <200802041242.21043.toad@amphibian.dyndns.org>
References: <200802041242.21043.toad@amphibian.dyndns.org>
Message-ID: <20080204142432.GE3394@freenetproject.org>

* Matthew Toseland <toad at amphibian.dyndns.org> [2008-02-04 12:42:16]:

> Something I noticed this month's Linux Magazine. We've been looking for a good 
> DVCS for a while: What about Bazaar?
> - Publishes to static content, so should be easy to adapt for Freenet. 
> (Hopefully - USK versioning wouldn't be a problem for a DVCS assuming there 
> are repository version numbers of some kind to use as a basis for diffs???).
> - GPL2+ (dual license with a closed source friendly license).
> - Focus on user friendliness (sponsored by Canonical).
> 
> http://bazaar-vcs.org/

Its main asset is its advanced merging algorithm... The main problem is
that it doesn't scale well... and the history of our tree is huge.

NextGen$
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080204/f5386d34/attachment.pgp 

From robert at freenetproject.org  Mon Feb  4 16:47:27 2008
From: robert at freenetproject.org (Robert Hailey)
Date: Mon, 4 Feb 2008 10:47:27 -0600
Subject: [freenet-dev] [freenet-cvs] r17473 -
	trunk/freenet/src/freenet/node
In-Reply-To: <20080202162541.99E5B391D51@freenetproject.org>
References: <20080202162541.99E5B391D51@freenetproject.org>
Message-ID: <22A1489C-A847-4342-A092-3810D2344E5E@freenetproject.org>


I think this is a more complicated case than this solution provides.  
With reasonable frequency we will incur a success (just too late). In  
which case, will adding a failure table entry notice that?

Maybe it should be closer too:

if (success)
	send offer?;
else
	addToFailureTable;

--
Robert Hailey


On Feb 2, 2008, at 10:25 AM, toad at freenetproject.org wrote:

> Author: toad
> Date: 2008-02-02 16:25:41 +0000 (Sat, 02 Feb 2008)
> New Revision: 17473
>
> Modified:
>   trunk/freenet/src/freenet/node/RequestHandler.java
> Log:
> If a request takes so long that the predecessor times out, but we do  
> eventually get the data, we need to offer it to the predecessor  
> through ULPRs.
>
> Modified: trunk/freenet/src/freenet/node/RequestHandler.java
> ===================================================================
> --- trunk/freenet/src/freenet/node/RequestHandler.java	2008-02-02  
> 16:06:24 UTC (rev 17472)
> +++ trunk/freenet/src/freenet/node/RequestHandler.java	2008-02-02  
> 16:25:41 UTC (rev 17473)
> @@ -228,6 +228,8 @@
> 		this.status=status;
> 		
> 		if (now > responseDeadline) {
> +			// Offer the data if there is any.
> +    		node.failureTable.onFailure(key, htl, new PeerNode[]  
> { source }, null, -1, System.currentTimeMillis());
> 			Logger.error(this, "requestsender took too long to respond to  
> requestor ("+TimeUtil.formatTime((now - searchStartTime), 2, true) 
> +"/"+rs.getStatusString()+")");
> 			applyByteCounts();
> 			unregisterRequestHandlerWithNode();
>


From toad at amphibian.dyndns.org  Mon Feb  4 17:55:23 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 4 Feb 2008 17:55:23 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <47A49940.9050802@cs.ucl.ac.uk>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
Message-ID: <200802041755.23675.toad@amphibian.dyndns.org>

On Saturday 02 February 2008 16:24, Michael Rogers wrote:
> Robert Hailey wrote:
> > Unless [in a later implementation] DNFs (maybe weighed against HTL) 
> > become an indication of a separate network. Then it would do precisely 
that.
> 
> DNFs would detect a "black hole" attacker - what about an attacker who 
> just wants to monitor a region of the key space or censor a few selected 
> keys?
> 
> (Again, this criticism isn't specific to your proposal and I'm not 
> saying your proposal creates this problem.)

Swapping creates this problem. Or does it? Could you perhaps do some 
simulations of two networks of different sizes weakly linked and show whether 
they get independant location spaces, or whether swapping tries to put one of 
them within the global keyspace for the other?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080204/a8cddec4/attachment.pgp 

From robert at freenetproject.org  Mon Feb  4 20:31:06 2008
From: robert at freenetproject.org (Robert Hailey)
Date: Mon, 4 Feb 2008 14:31:06 -0600
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <47A49940.9050802@cs.ucl.ac.uk>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<Prayer.1.0.16.0801301524240.92105@bells3.cs.ucl.ac.uk>
	<91F6684A-8661-4E4D-8512-35F3D1DE0316@freenetproject.org>
	<Prayer.1.0.16.0801301734100.93197@bells3.cs.ucl.ac.uk>
	<0E7521BC-7329-458D-ADAC-229BE691A6A2@freenetproject.org>
	<Prayer.1.0.16.0801311133340.98065@bells3.cs.ucl.ac.uk>
	<8ECAA540-7437-473B-9573-19E3955B9980@freenetproject.org>
	<Prayer.1.0.16.0801311757460.913@bells3.cs.ucl.ac.uk>
	<73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
Message-ID: <04775CD3-B6BB-490E-A816-6729867C3B83@freenetproject.org>


On Feb 2, 2008, at 10:24 AM, Michael Rogers wrote:
>>> what am I missing?
>>
>> That following the algorithm will eventually color the entire
>> network/subnetwork/dungeon with the same network id, which is
>> presumed necessary for any sort of distant routing.
>
> OK, I think I see what you mean now - the network IDs aren't just
> private labels used by each node to colour its private map of the
> network - they're used for routing, so there needs to be a rough
> consensus about which label applies to which area of the network.
>
> I'm still not quite sure about how that consensus is reached. Agreeing
> on an ID for the local subnet doesn't seem to be a problem, but what
> happens if (for example) an attacker creates a subnet with the same ID
> as an existing subnet? Any searches that have visited the attacker's
> subnet and failed will be marked "don't look in subnet X" and  
> therefore
> won't visit the 'real' subnet X, right?

I think that this is the beauty of this approach, that it handles that  
quite well.

The short answer is that (for any routing purposes) we *only* take  
into account the network ids which we assign to a node, so if they  
advertise a network id which our peers have (and cannot back that up  
with connectivity probes) they lose. If there are two networks (with  
the same id) that we are equidistant from, the more-successful ends up  
winning via the routing tables.

This situation can occur naturally though, not only via an attack.  
Consider a partial subnet -- like an arch -- that connects on it's two  
endpoints to the (otherwise-well-connected) network at large. Now, the  
case presented thus far is if the two end points are not well  
connected (and the network gets it's own network id / fully colored),  
but it is possible that on one end it is connected well enough to not  
be detected and the large network id (say... 1) is pushed into this  
arch from the left, whereas the other end (not being well connected)  
the edge nodes present a different network id to the subnet so a  
different network id (say... 2) is pushed into the arch from the  
right. If the subnet is colored "1" through-and-through, the edge  
nodes on the right may still choose not to route into it (as it  
appears to be a dungeon/subnet, which it is, as they assign "2", even  
if "1" is presented). More likely, though, is that half the arch will  
be colored with each network id; the side closer to the well-connected  
network will route through it using the dungeon-side as an escape  
route, and the dungeon side will route through it's opening using the  
network-side as an escape route.

> And I guess the list of failed subnets can only be updated by the
> requester - otherwise an attacker could return a DNF saying "already
> tried subnets X, Y, Z"?

Yes and no...

What I had in mind, was stripping away any network id's (from success  
or failures) which a node does not "recognize" (via successes). We  
could add to that stripping network id's for which we have assigned to  
other peers. Perhaps the proper solution would be to ignore/strip  
extra-failed network id's which come from a route we would not take to  
that network. Sadly, this makes the distant routing information much  
slower in coming; similar in nature to the assimilation problem with  
0.5. So in the natural case where a node (just put into a network/no  
distant routing table) gets a failure, that is the end.

If that were the extent of the routing, though, only nodes near these  
gateways would learn about the other network(s). Therefore, in order  
for the distant network information to propagate, a node which process  
a request which naturally DNFs, and has HTL left will continue the  
query (at the same HTL) with the DNF'd network id as the failed  
network list (which means skipping peers assigned to that network).

It makes perfect sense, however, for a node originating a request to  
try all of the network id's in it's "routable" list (or that is... to  
repeat the request with the given failures as the dont-try list), and  
would likely be the common case for success (as all the htl is spent  
trying to escape from the networks in that list).

Also, there is a relationship between the number of network ids which  
we:
(1) will route to,
(2) transmit with a packet (e.g. on request/failure/success)
(3) recognize,
(4) count successes.

I think that in order for it to work 1<2<3<4, but I'm not sure. Also,  
a timeout is most likely required for network info at level-1 (maybe  
+24hrs from last success); as a big-subnet (which becomes connected or  
is in network-id-limbo), would have a large number of successes (and  
therefore be very-high-and-secure in the routing table, but becomes  
totally obsolete routing wise once it is connected.

>> So now we'll tell our peer's
>> that we can reach network 'w' (through this peer w/ that success).  
>> Even
>> if it is a Sybil-simulated subnetwork, routing to it (and perhaps  
>> wether
>> or not we advertise it to our peers) will probably end up being  
>> related
>> to success-probability; they could only become relied upon by  
>> providing
>> good data (not censoring), why not use the Sybil net for storage,  
>> then?
>
> This sounds promising. I'm not sure all attackers will be unreliable,
> though. An attacker might diligently store 99% of keys but censor the
> other 1% - but maybe FEC makes that a non-issue. Or the attacker might
> provide a fast, reliable storage service in order to monitor as much
> traffic as possible - but maybe that's unavoidable in opennet.

Well... it would be exceptionally difficult to detect a node that  
plays nice 99% of the time in any event (opennet or darknet). :)

> How do you measure success for inserts?

I think that inserts naturally escape dungeons already. I do not  
suggest modifying their behavior. If it is inserted into a large- 
enough subnet where it is generally lost, I would hope that subnet is  
popular enough to be in the routing tables for a distant search.

--
Robert Hailey

> Or maybe you can just measure
> success for requests and assume that inserts must be succeeding if
> requests are succeeding?
>
> Cheeers,
> Michael




From m.rogers at cs.ucl.ac.uk  Tue Feb  5 01:24:31 2008
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Tue, 05 Feb 2008 01:24:31 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <200802041755.23675.toad@amphibian.dyndns.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
	<200802041755.23675.toad@amphibian.dyndns.org>
Message-ID: <47A7BACF.9000400@cs.ucl.ac.uk>

Matthew Toseland wrote:
> Swapping creates this problem. Or does it? Could you perhaps do some 
> simulations of two networks of different sizes weakly linked and show whether 
> they get independant location spaces, or whether swapping tries to put one of 
> them within the global keyspace for the other?

Here's a quick simulation that shows that two weakly-connected subnets 
move into separate regions of the key space. Each subnet has an ideal 
Kleinberg topology and starts out uniformly distributed across the whole 
key space, and there are also a few random links between the subnets - 
this is meant to represent what would happen if you created a few links 
between two mature networks, or between a real network and a Sybil network.

I couldn't be bothered to do a nice GUI so the output is just a series 
of histograms: on each line the key space is divided into 20 regions, 
and each column shows the number of nodes from the first subnet in that 
region. Initially there are roughly 50 nodes in each region, but 
swapping causes the subnets to segregate so that eventually most regions 
are almost exclusively occupied by one subnet or the other.

It's kind of interesting to compare this with "white flight" in sociology...

Cheers,
Michael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Simulation.java
Type: text/x-java
Size: 3194 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080205/fc0a238a/attachment.java 

From ian.clarke at gmail.com  Tue Feb  5 01:53:19 2008
From: ian.clarke at gmail.com (Ian Clarke)
Date: Mon, 4 Feb 2008 19:53:19 -0600
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <47A7BACF.9000400@cs.ucl.ac.uk>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
	<200802041755.23675.toad@amphibian.dyndns.org>
	<47A7BACF.9000400@cs.ucl.ac.uk>
Message-ID: <823242bd0802041753h59a2150bs5e2d220526176fee@mail.gmail.com>

On Feb 4, 2008 7:24 PM, Michael Rogers <m.rogers at cs.ucl.ac.uk> wrote:
> on each line the key space is divided into 20 regions,
> and each column shows the number of nodes from the first subnet in that
> region. Initially there are roughly 50 nodes in each region, but
> swapping causes the subnets to segregate so that eventually most regions
> are almost exclusively occupied by one subnet or the other.

I would have expected regions close to each other to tend to have the
same subnet, with one subnet occupying one side of the location space,
and the other region occupying the other, yet I'm seeing something
like this:

86 95 86 60 2 1 72 89 87 91 8 5 3 39 108 91 64 6 0 7

There seems to be three "clusters" here, which is rather strange - any theories?

Ian.


-- 
Email: ian at uprizer.com
Cell: +1 512 422 3588
Skype: sanity


From m.rogers at cs.ucl.ac.uk  Tue Feb  5 02:10:05 2008
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Tue, 05 Feb 2008 02:10:05 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <823242bd0802041753h59a2150bs5e2d220526176fee@mail.gmail.com>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
	<200802041755.23675.toad@amphibian.dyndns.org>
	<47A7BACF.9000400@cs.ucl.ac.uk>
	<823242bd0802041753h59a2150bs5e2d220526176fee@mail.gmail.com>
Message-ID: <47A7C57D.1010107@cs.ucl.ac.uk>

Ian Clarke wrote:
> I would have expected regions close to each other to tend to have the
> same subnet, with one subnet occupying one side of the location space,
> and the other region occupying the other, yet I'm seeing something
> like this:
> 
> 86 95 86 60 2 1 72 89 87 91 8 5 3 39 108 91 64 6 0 7
> 
> There seems to be three "clusters" here, which is rather strange - any theories?

I think it might have something to do with the number of links between 
the subnets (try playing with the JOINS constant), but eventually the 
clusters tend to coalesce if you leave the simulation running for long 
enough.

Cheers,
Michael


From toad at amphibian.dyndns.org  Tue Feb  5 11:03:29 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Tue, 5 Feb 2008 11:03:29 +0000
Subject: [freenet-dev] Generating more keys from JFK
Message-ID: <200802051103.36847.toad@amphibian.dyndns.org>

NewPacketFormat assumes that we can generate as many keys as we want from JFK 
securely. Is this true? JFK uses an HMAC with 0, 1, or 2, to generate the 
session key or the 2 internal keys it uses, but does not explicitly document 
the option to generate more keys by incrementing that number - and it refers 
to IKE key extension if you need more bits (it does *not* say increment the 
number and stick them together, as you might expect). Is it safe to do what 
we have planned, to get separate keys for each direction and in 
NewPacketFormat for the IV key and HMAC key?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080205/55852608/attachment.pgp 

From m.rogers at cs.ucl.ac.uk  Tue Feb  5 12:35:30 2008
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Tue, 05 Feb 2008 12:35:30 +0000
Subject: [freenet-dev] Generating more keys from JFK
In-Reply-To: <200802051103.36847.toad@amphibian.dyndns.org>
References: <200802051103.36847.toad@amphibian.dyndns.org>
Message-ID: <47A85812.2010106@cs.ucl.ac.uk>

Matthew Toseland wrote:
> NewPacketFormat assumes that we can generate as many keys as we want from JFK 
> securely. Is this true? JFK uses an HMAC with 0, 1, or 2, to generate the 
> session key or the 2 internal keys it uses, but does not explicitly document 
> the option to generate more keys by incrementing that number - and it refers 
> to IKE key extension if you need more bits (it does *not* say increment the 
> number and stick them together, as you might expect). Is it safe to do what 
> we have planned, to get separate keys for each direction and in 
> NewPacketFormat for the IV key and HMAC key?

Here's how Ferguson and Schneier do it in Practical Cryptography:

K is the master key for the channel

KeySendEnc = HASH (K || "Enc Alice to Bob")
KeyRecEnd = HASH (K || "Enc Bob to Alice")
KeySendAuth = HASH (K || "Auth Alice to Bob")
KeyRecAuth = HASH (K || "Auth Bob to Alice")

if (I am Bob) {
	swap (KeySendEnc, KeyRecEnc)
	swap (KeySendAuth, KeyRecAuth)
}

Chapter 8 of that book is really worth reading if you're starting work 
on NewPacketFormat.

Cheers,
Michael


From toad at amphibian.dyndns.org  Tue Feb  5 19:00:48 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Tue, 5 Feb 2008 19:00:48 +0000
Subject: [freenet-dev] Generating more keys from JFK
In-Reply-To: <47A85812.2010106@cs.ucl.ac.uk>
References: <200802051103.36847.toad@amphibian.dyndns.org>
	<47A85812.2010106@cs.ucl.ac.uk>
Message-ID: <200802051900.53295.toad@amphibian.dyndns.org>

On Tuesday 05 February 2008 12:35, Michael Rogers wrote:
> Matthew Toseland wrote:
> > NewPacketFormat assumes that we can generate as many keys as we want from 
JFK 
> > securely. Is this true? JFK uses an HMAC with 0, 1, or 2, to generate the 
> > session key or the 2 internal keys it uses, but does not explicitly 
document 
> > the option to generate more keys by incrementing that number - and it 
refers 
> > to IKE key extension if you need more bits (it does *not* say increment 
the 
> > number and stick them together, as you might expect). Is it safe to do 
what 
> > we have planned, to get separate keys for each direction and in 
> > NewPacketFormat for the IV key and HMAC key?
> 
> Here's how Ferguson and Schneier do it in Practical Cryptography:
> 
> K is the master key for the channel
> 
> KeySendEnc = HASH (K || "Enc Alice to Bob")
> KeyRecEnd = HASH (K || "Enc Bob to Alice")
> KeySendAuth = HASH (K || "Auth Alice to Bob")
> KeyRecAuth = HASH (K || "Auth Bob to Alice")
> 
> if (I am Bob) {
> 	swap (KeySendEnc, KeyRecEnc)
> 	swap (KeySendAuth, KeyRecAuth)
> }
> 
> Chapter 8 of that book is really worth reading if you're starting work 
> on NewPacketFormat.

JFK is:

Session key = H_<secret> { alice exponential, bob exponential, "2" } 
K_a = H_<secret> { alice exponential, bob exponential, "1" }
K_e = H_<secret> { alice exponential, bob exponential, "2" }

Where K_a and K_e are temporary keys used in phase 3 and 4.

It's the same principle, it's just a question of whether it's safer to derive 
further keys from the session key or from the secret key (= g^xy).

And no sadly there is no time to do that now, but it came up in discussions 
with kryptos.
> 
> Cheers,
> Michael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080205/0d4684da/attachment.pgp 

From robert at freenetproject.org  Tue Feb  5 19:05:30 2008
From: robert at freenetproject.org (Robert Hailey)
Date: Tue, 5 Feb 2008 13:05:30 -0600
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <47A7BACF.9000400@cs.ucl.ac.uk>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
	<200802041755.23675.toad@amphibian.dyndns.org>
	<47A7BACF.9000400@cs.ucl.ac.uk>
Message-ID: <49071261-11C5-4FA5-B951-9A764A135092@freenetproject.org>


On Feb 4, 2008, at 7:24 PM, Michael Rogers wrote:

> Matthew Toseland wrote:
>> Swapping creates this problem. Or does it? Could you perhaps do  
>> some simulations of two networks of different sizes weakly linked  
>> and show whether they get independant location spaces, or whether  
>> swapping tries to put one of them within the global keyspace for  
>> the other?
>
> Here's a quick simulation that shows that two weakly-connected  
> subnets move into separate regions of the key space. Each subnet has  
> an ideal Kleinberg topology and starts out uniformly distributed  
> across the whole key space, and there are also a few random links  
> between the subnets - this is meant to represent what would happen  
> if you created a few links between two mature networks, or between a  
> real network and a Sybil network.
>
> I couldn't be bothered to do a nice GUI so the output is just a  
> series of histograms: on each line the key space is divided into 20  
> regions, and each column shows the number of nodes from the first  
> subnet in that region. Initially there are roughly 50 nodes in each  
> region, but swapping causes the subnets to segregate so that  
> eventually most regions are almost exclusively occupied by one  
> subnet or the other.
>
> It's kind of interesting to compare this with "white flight" in  
> sociology...
>
> Cheers,
> Michael

For a "quick simulation", I think this is an excellent demonstration!  
However, I think that there are two major errors.

First, there is an equal chance of nodes from either network swapping.  
In the freenet network (since swapping is done through the network) it  
is not equal (if for no other cause than the scarce links between the  
two networks). More on this below.

Second, it looks like you forgot to actually make any connections  
between the two networks; without the stress points (plus the fact of  
equal weighting on swaps), it is no wonder the nodes flee so quickly!

Once I added the joins to the two networks on your sim, I noticed that  
the networks would "bunchup", but not split the keyspace. However, I'm  
not sure the lack-of-keyspace-split helps either side of the argument  
as these swaps are are applied to stable/mature networks to begin  
with. What this demonstrates is more the merging of two large networks  
(e.g. freenet-china and freenet-usa finally get linked up), and that  
they maintain there keyspace.

Speculating about the swap probability...

ASSUMING that the keyspace would be split (i.e. as the networks grow  
up together) [which I still don't think is the case], then as the  
probability of swapping becomes equal the two networks occupy the same  
key space (as Michael's supplied simulator). However, surely as the  
probability of swapping (between the two networks) approaches zero  
they overlap the keyspace. Actually, it is well before zero...  
logically it is that small probability of randomizing the location  
after the swap (new locations entering the network at the same rate  
they leave).

I've quite enjoyed playing around with this, particularly tracking  
particular nodes in the network to see if I could catch the two  
networks rotating to match each other, but also varying probabilities  
of swaps across the networks.

--
Robert Hailey

-------------- next part --------------
A non-text attachment was scrubbed...
Name: Simulation.java
Type: application/octet-stream
Size: 4827 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080205/1be11b26/attachment.obj 
-------------- next part --------------


From toad at amphibian.dyndns.org  Tue Feb  5 19:05:47 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Tue, 5 Feb 2008 19:05:47 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <04775CD3-B6BB-490E-A816-6729867C3B83@freenetproject.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
	<04775CD3-B6BB-490E-A816-6729867C3B83@freenetproject.org>
Message-ID: <200802051905.47618.toad@amphibian.dyndns.org>

Is this sufficiently well-defined for simulations yet? It seems to me there 
will be threshold parameters and so on?

On Monday 04 February 2008 20:31, Robert Hailey wrote:
> 
> On Feb 2, 2008, at 10:24 AM, Michael Rogers wrote:
> >>> what am I missing?
> >>
> >> That following the algorithm will eventually color the entire
> >> network/subnetwork/dungeon with the same network id, which is
> >> presumed necessary for any sort of distant routing.
> >
> > OK, I think I see what you mean now - the network IDs aren't just
> > private labels used by each node to colour its private map of the
> > network - they're used for routing, so there needs to be a rough
> > consensus about which label applies to which area of the network.
> >
> > I'm still not quite sure about how that consensus is reached. Agreeing
> > on an ID for the local subnet doesn't seem to be a problem, but what
> > happens if (for example) an attacker creates a subnet with the same ID
> > as an existing subnet? Any searches that have visited the attacker's
> > subnet and failed will be marked "don't look in subnet X" and  
> > therefore
> > won't visit the 'real' subnet X, right?
> 
> I think that this is the beauty of this approach, that it handles that  
> quite well.
> 
> The short answer is that (for any routing purposes) we *only* take  
> into account the network ids which we assign to a node, so if they  
> advertise a network id which our peers have (and cannot back that up  
> with connectivity probes) they lose. If there are two networks (with  
> the same id) that we are equidistant from, the more-successful ends up  
> winning via the routing tables.
> 
> This situation can occur naturally though, not only via an attack.  
> Consider a partial subnet -- like an arch -- that connects on it's two  
> endpoints to the (otherwise-well-connected) network at large. Now, the  
> case presented thus far is if the two end points are not well  
> connected (and the network gets it's own network id / fully colored),  
> but it is possible that on one end it is connected well enough to not  
> be detected and the large network id (say... 1) is pushed into this  
> arch from the left, whereas the other end (not being well connected)  
> the edge nodes present a different network id to the subnet so a  
> different network id (say... 2) is pushed into the arch from the  
> right. If the subnet is colored "1" through-and-through, the edge  
> nodes on the right may still choose not to route into it (as it  
> appears to be a dungeon/subnet, which it is, as they assign "2", even  
> if "1" is presented). More likely, though, is that half the arch will  
> be colored with each network id; the side closer to the well-connected  
> network will route through it using the dungeon-side as an escape  
> route, and the dungeon side will route through it's opening using the  
> network-side as an escape route.
> 
> > And I guess the list of failed subnets can only be updated by the
> > requester - otherwise an attacker could return a DNF saying "already
> > tried subnets X, Y, Z"?
> 
> Yes and no...
> 
> What I had in mind, was stripping away any network id's (from success  
> or failures) which a node does not "recognize" (via successes). We  
> could add to that stripping network id's for which we have assigned to  
> other peers. Perhaps the proper solution would be to ignore/strip  
> extra-failed network id's which come from a route we would not take to  
> that network. Sadly, this makes the distant routing information much  
> slower in coming; similar in nature to the assimilation problem with  
> 0.5. So in the natural case where a node (just put into a network/no  
> distant routing table) gets a failure, that is the end.
> 
> If that were the extent of the routing, though, only nodes near these  
> gateways would learn about the other network(s). Therefore, in order  
> for the distant network information to propagate, a node which process  
> a request which naturally DNFs, and has HTL left will continue the  
> query (at the same HTL) with the DNF'd network id as the failed  
> network list (which means skipping peers assigned to that network).
> 
> It makes perfect sense, however, for a node originating a request to  
> try all of the network id's in it's "routable" list (or that is... to  
> repeat the request with the given failures as the dont-try list), and  
> would likely be the common case for success (as all the htl is spent  
> trying to escape from the networks in that list).
> 
> Also, there is a relationship between the number of network ids which  
> we:
> (1) will route to,
> (2) transmit with a packet (e.g. on request/failure/success)
> (3) recognize,
> (4) count successes.
> 
> I think that in order for it to work 1<2<3<4, but I'm not sure. Also,  
> a timeout is most likely required for network info at level-1 (maybe  
> +24hrs from last success); as a big-subnet (which becomes connected or  
> is in network-id-limbo), would have a large number of successes (and  
> therefore be very-high-and-secure in the routing table, but becomes  
> totally obsolete routing wise once it is connected.
> 
> >> So now we'll tell our peer's
> >> that we can reach network 'w' (through this peer w/ that success).  
> >> Even
> >> if it is a Sybil-simulated subnetwork, routing to it (and perhaps  
> >> wether
> >> or not we advertise it to our peers) will probably end up being  
> >> related
> >> to success-probability; they could only become relied upon by  
> >> providing
> >> good data (not censoring), why not use the Sybil net for storage,  
> >> then?
> >
> > This sounds promising. I'm not sure all attackers will be unreliable,
> > though. An attacker might diligently store 99% of keys but censor the
> > other 1% - but maybe FEC makes that a non-issue. Or the attacker might
> > provide a fast, reliable storage service in order to monitor as much
> > traffic as possible - but maybe that's unavoidable in opennet.
> 
> Well... it would be exceptionally difficult to detect a node that  
> plays nice 99% of the time in any event (opennet or darknet). :)
> 
> > How do you measure success for inserts?
> 
> I think that inserts naturally escape dungeons already. I do not  
> suggest modifying their behavior. If it is inserted into a large- 
> enough subnet where it is generally lost, I would hope that subnet is  
> popular enough to be in the routing tables for a distant search.
> 
> --
> Robert Hailey
> 
> > Or maybe you can just measure
> > success for requests and assume that inserts must be succeeding if
> > requests are succeeding?
> >
> > Cheeers,
> > Michael
> 
> 
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080205/6162db78/attachment.pgp 

From toad at amphibian.dyndns.org  Tue Feb  5 19:25:56 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Tue, 5 Feb 2008 19:25:56 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <49071261-11C5-4FA5-B951-9A764A135092@freenetproject.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<47A7BACF.9000400@cs.ucl.ac.uk>
	<49071261-11C5-4FA5-B951-9A764A135092@freenetproject.org>
Message-ID: <200802051926.00733.toad@amphibian.dyndns.org>

On Tuesday 05 February 2008 19:05, Robert Hailey wrote:
> 
> On Feb 4, 2008, at 7:24 PM, Michael Rogers wrote:
> 
> > Matthew Toseland wrote:
> >> Swapping creates this problem. Or does it? Could you perhaps do  
> >> some simulations of two networks of different sizes weakly linked  
> >> and show whether they get independant location spaces, or whether  
> >> swapping tries to put one of them within the global keyspace for  
> >> the other?
> >
> > Here's a quick simulation that shows that two weakly-connected  
> > subnets move into separate regions of the key space. Each subnet has  
> > an ideal Kleinberg topology and starts out uniformly distributed  
> > across the whole key space, and there are also a few random links  
> > between the subnets - this is meant to represent what would happen  
> > if you created a few links between two mature networks, or between a  
> > real network and a Sybil network.
> >
> > I couldn't be bothered to do a nice GUI so the output is just a  
> > series of histograms: on each line the key space is divided into 20  
> > regions, and each column shows the number of nodes from the first  
> > subnet in that region. Initially there are roughly 50 nodes in each  
> > region, but swapping causes the subnets to segregate so that  
> > eventually most regions are almost exclusively occupied by one  
> > subnet or the other.
> >
> > It's kind of interesting to compare this with "white flight" in  
> > sociology...
> >
> > Cheers,
> > Michael
> 
> For a "quick simulation", I think this is an excellent demonstration!  
> However, I think that there are two major errors.
> 
> First, there is an equal chance of nodes from either network swapping.  
> In the freenet network (since swapping is done through the network) it  
> is not equal (if for no other cause than the scarce links between the  
> two networks). More on this below.

His simulation doesn't deal with this? There are two mechanisms that would 
limit it:
- The limit on swaps per second per link. We won't accept a swap request if 
we've had one within the last 900ms. However, this may be too lenient a 
limit, we could increase it.
- Swaps are routed as random possibly looping walks.
> 
> Second, it looks like you forgot to actually make any connections  
> between the two networks; without the stress points (plus the fact of  
> equal weighting on swaps), it is no wonder the nodes flee so quickly!
> 
> Once I added the joins to the two networks on your sim, I noticed that  
> the networks would "bunchup", but not split the keyspace. However, I'm  
> not sure the lack-of-keyspace-split helps either side of the argument  
> as these swaps are are applied to stable/mature networks to begin  
> with. What this demonstrates is more the merging of two large networks  
> (e.g. freenet-china and freenet-usa finally get linked up), and that  
> they maintain there keyspace.

Define "bunchup" ?

This is IMHO a very significant result:
- Link-based swap pressure causes the two networks to keep their own separate 
keyspaces.
- Local requests will still function 99% of the time.
- Exceptionally popular keys will eventually be routed to the other network, 
and with ULPRs will probably be propagated widely across the original network 
once found.
- It is not a viable attack to dangle a large virtual network off a few 
connections. Unfortunately there are other viable attacks on swapping (read 
the Pitch Black paper, last year somebody appears to have deployed it with 
considerable success although some of our problems were certainly due to 
churn).
- An escape-route mechanism may be interesting, however given the limited 
capacity it may be simplest to just rely on chance and ULPRs propagating the 
most sought after content? The problem with this is those few requests which 
cross the border will probably have low HTL when they get there, so if the 
content isn't popular on the other side it might not be found. So maybe we do 
need some sort of mechanism - but we will need a way to ration the scarce 
resources according to popularity.
> 
> Speculating about the swap probability...
> 
> ASSUMING that the keyspace would be split (i.e. as the networks grow  
> up together) [which I still don't think is the case], 

What happens as we get more and more links between the networks? Catastrophic 
merging? What about a few high capacity links? Etc.. more work can be done 
here.

> then as the   
> probability of swapping becomes equal the two networks occupy the same  
> key space (as Michael's supplied simulator). However, surely as the  
> probability of swapping (between the two networks) approaches zero  
> they overlap the keyspace. Actually, it is well before zero...  
> logically it is that small probability of randomizing the location  
> after the swap (new locations entering the network at the same rate  
> they leave).
> 
> I've quite enjoyed playing around with this, particularly tracking  
> particular nodes in the network to see if I could catch the two  
> networks rotating to match each other, but also varying probabilities  
> of swaps across the networks.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080205/d32c3666/attachment.pgp 

From m.rogers at cs.ucl.ac.uk  Tue Feb  5 20:02:22 2008
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Tue, 05 Feb 2008 20:02:22 +0000
Subject: [freenet-dev] Generating more keys from JFK
In-Reply-To: <200802051900.53295.toad@amphibian.dyndns.org>
References: <200802051103.36847.toad@amphibian.dyndns.org>
	<47A85812.2010106@cs.ucl.ac.uk>
	<200802051900.53295.toad@amphibian.dyndns.org>
Message-ID: <47A8C0CE.2030804@cs.ucl.ac.uk>

Matthew Toseland wrote:
> JFK is:
> 
> Session key = H_<secret> { alice exponential, bob exponential, "2" } 
> K_a = H_<secret> { alice exponential, bob exponential, "1" }
> K_e = H_<secret> { alice exponential, bob exponential, "2" }
> 
> Where K_a and K_e are temporary keys used in phase 3 and 4.
> 
> It's the same principle, it's just a question of whether it's safer to derive 
> further keys from the session key or from the secret key (= g^xy).

 From this and the Ferguson/Schneier snippet it looks like you can 
generally derive any number of keys from a master key by hashing it with 
different values, wouldn't you say?

(Actually I'm kind of surprised by this... ideally we'd like there to be 
no relationship between the various keys but obviously there has to be a 
relationship if the other keys are derived from the master, so the 
problem is how to make sure the relationship isn't exploitable... I 
didn't realise hash functions guaranteed to conceal the relationship 
between the input and output, that sounds more like the guarantee 
provided by a block cipher, so I would have though it would make more 
sense to use a block cipher to derive the other keys... something like 
k1=enc_k(1), k2=enc_k(2), etc... in other words CTR mode. But what do I 
know?)

Cheers,
Michael


From m.rogers at cs.ucl.ac.uk  Tue Feb  5 20:21:14 2008
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Tue, 05 Feb 2008 20:21:14 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <49071261-11C5-4FA5-B951-9A764A135092@freenetproject.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
	<200802041755.23675.toad@amphibian.dyndns.org>
	<47A7BACF.9000400@cs.ucl.ac.uk>
	<49071261-11C5-4FA5-B951-9A764A135092@freenetproject.org>
Message-ID: <47A8C53A.9090505@cs.ucl.ac.uk>

Robert Hailey wrote:
> First, there is an equal chance of nodes from either network swapping. 
> In the freenet network (since swapping is done through the network) it 
> is not equal (if for no other cause than the scarce links between the 
> two networks).

I was under the impression that the random walks for swapping were long 
enough to reach any node with roughly equal probability - I believe that 
was Oskar's intention. If the random walks aren't escaping from local 
clusters then we'll never be able to smooth out the clusters...

> Second, it looks like you forgot to actually make any connections 
> between the two networks; without the stress points (plus the fact of 
> equal weighting on swaps), it is no wonder the nodes flee so quickly!

I'm probably overlooking some really obvious error, but doesn't this 
part of the code create connections between the two networks?

// Join the networks at a few randomly chosen points
for (int i = 0; i < JOINS; i++) {
	Node r = reds.get ((int) (Math.random() * REDS));
	Node b = blacks.get ((int) (Math.random() * BLACKS));
	r.neighbours.add (b);
	b.neighbours.add (r);
}

> Once I added the joins to the two networks on your sim, I noticed that 
> the networks would "bunchup", but not split the keyspace.

Yes, it seems like the more connections there are between the networks, 
the longer it takes them to segregate and the more pieces each subnet 
breaks into. However, in the very long term the pieces of each subnet 
still tend to coalesce.

> ASSUMING that the keyspace would be split (i.e. as the networks grow up 
> together) [which I still don't think is the case],

I'm not sure what you mean by grow up together but I'll happily modify 
the simulation if you don't mind explaining.

> However, surely as the 
> probability of swapping (between the two networks) approaches zero they 
> overlap the keyspace. Actually, it is well before zero... logically it 
> is that small probability of randomizing the location after the swap 
> (new locations entering the network at the same rate they leave).

True, but I'm not sure that's desirable either - if we only allow nodes 
to swap within their own subnets then we'll never smooth out the subnets.

> I've quite enjoyed playing around with this, particularly tracking 
> particular nodes in the network to see if I could catch the two networks 
> rotating to match each other, but also varying probabilities of swaps 
> across the networks.

Glad you liked it. :-) Speaking of rotation, that's something else I 
meant to simulate: can a few attackers "spin" the network by gradually 
moving clockwise?

Cheers,
Michael


From robert at freenetproject.org  Tue Feb  5 21:10:43 2008
From: robert at freenetproject.org (Robert Hailey)
Date: Tue, 5 Feb 2008 15:10:43 -0600
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <47A7BACF.9000400@cs.ucl.ac.uk>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
	<200802041755.23675.toad@amphibian.dyndns.org>
	<47A7BACF.9000400@cs.ucl.ac.uk>
Message-ID: <0F272690-1DF1-4A9B-A215-22F9FC06B17D@freenetproject.org>


On Feb 4, 2008, at 7:24 PM, Michael Rogers wrote:

> Matthew Toseland wrote:
>> Swapping creates this problem. Or does it? Could you perhaps do  
>> some simulations of two networks of different sizes weakly linked  
>> and show whether they get independant location spaces, or whether  
>> swapping tries to put one of them within the global keyspace for  
>> the other?
>
> Here's a quick simulation that shows that two weakly-connected  
> subnets move into separate regions of the key space. Each subnet has  
> an ideal Kleinberg topology and starts out uniformly distributed  
> across the whole key space, and there are also a few random links  
> between the subnets - this is meant to represent what would happen  
> if you created a few links between two mature networks, or between a  
> real network and a Sybil network.
>
> I couldn't be bothered to do a nice GUI so the output is just a  
> series of histograms: on each line the key space is divided into 20  
> regions, and each column shows the number of nodes from the first  
> subnet in that region. Initially there are roughly 50 nodes in each  
> region, but swapping causes the subnets to segregate so that  
> eventually most regions are almost exclusively occupied by one  
> subnet or the other.
>
> It's kind of interesting to compare this with "white flight" in  
> sociology...
>
> Cheers,
> Michael

Ok. I see it Michael's way now.

I'm not sure to what degree this effects our present network, but when  
graphed out, the individual location movements from this swapping  
simulator often mirror zothar's previous graph (of location jumping);  
and while one network is making a hole in the other, the sliding/ 
compression looks just like what I saw previously as network rotation.

This is quite a problem. In fact, this may be the fundamental problem  
with swapping. Because of this, a sybil network could presently even  
choose which segment of the keyspace to occupy with very few links.

The same network coloring/routing logic *might* be applicable to  
swapping. That is, to simply confine swaps to the network they came  
from. I'm not aware of another way to both secure sybil nets from  
invasion and keep major keyspaces separate. Unfortunately it has the  
obvious problem of a dependency feedback loop:

(1) swaps are fundamental to routing,
(2) routing is required for my auth-ping idea,
(3) these auth pings are required for secured network-id coloring,
(4) then... could we use the network id's to modify swapping???

It seems like the network-id idea would have to be changed to be a  
little more relaxed; either by not effecting swapping until we have  
computed the assigned network-ids (fall-open while in transition), or  
by simply accepting (for the moment) the most common network id from  
our peer set (rather than strictly random id on startup).

My question is, *if* such an idea is considered valid and in such a  
case how could we be assured that us labeling and isolating a subnet  
is not what *keeps* it labeled as a subnet because it's routing is  
messed up for lack of swapping? I guess that would require all the  
bordering nodes to consider that simultaneously, which would be a rare  
and unstable condition in a well connected network.

--
Robert Hailey



From toad at amphibian.dyndns.org  Tue Feb  5 22:23:53 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Tue, 5 Feb 2008 22:23:53 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <47A8C53A.9090505@cs.ucl.ac.uk>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<49071261-11C5-4FA5-B951-9A764A135092@freenetproject.org>
	<47A8C53A.9090505@cs.ucl.ac.uk>
Message-ID: <200802052223.57761.toad@amphibian.dyndns.org>

On Tuesday 05 February 2008 20:21, Michael Rogers wrote:
> Robert Hailey wrote:
> > First, there is an equal chance of nodes from either network swapping. 
> > In the freenet network (since swapping is done through the network) it 
> > is not equal (if for no other cause than the scarce links between the 
> > two networks).
> 
> I was under the impression that the random walks for swapping were long 
> enough to reach any node with roughly equal probability - I believe that 
> was Oskar's intention. If the random walks aren't escaping from local 
> clusters then we'll never be able to smooth out the clusters...

Yes and no - in the case of poorly connected networks, the sheer capacity of 
the limited connections between the two networks produces a bias, doesn't it?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080205/814a94da/attachment.pgp 

From robert at freenetproject.org  Tue Feb  5 22:46:25 2008
From: robert at freenetproject.org (Robert Hailey)
Date: Tue, 5 Feb 2008 16:46:25 -0600
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <200802051926.00733.toad@amphibian.dyndns.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<47A7BACF.9000400@cs.ucl.ac.uk>
	<49071261-11C5-4FA5-B951-9A764A135092@freenetproject.org>
	<200802051926.00733.toad@amphibian.dyndns.org>
Message-ID: <B8822D8B-27AF-412E-B74A-D02B2CF7161E@freenetproject.org>


On Feb 5, 2008, at 1:25 PM, Matthew Toseland wrote:

> On Tuesday 05 February 2008 19:05, Robert Hailey wrote:
>> For a "quick simulation", I think this is an excellent demonstration!
>> However, I think that there are two major errors.
>>
>> First, there is an equal chance of nodes from either network  
>> swapping.
>> In the freenet network (since swapping is done through the network)  
>> it
>> is not equal (if for no other cause than the scarce links between the
>> two networks). More on this below.
>
> His simulation doesn't deal with this? There are two mechanisms that  
> would
> limit it:
> - The limit on swaps per second per link. We won't accept a swap  
> request if
> we've had one within the last 900ms. However, this may be too  
> lenient a
> limit, we could increase it.
> - Swaps are routed as random possibly looping walks.

What I mean is that a node is more likely to swap with another node on  
it's same well-connected network. I don't know how much so. In the sim  
it the chance of any node swapping with any other node is the same.

>> Once I added the joins to the two networks on your sim, I noticed  
>> that
>> the networks would "bunchup", but not split the keyspace. However,  
>> I'm
>> not sure the lack-of-keyspace-split helps either side of the argument
>> as these swaps are are applied to stable/mature networks to begin
>> with. What this demonstrates is more the merging of two large  
>> networks
>> (e.g. freenet-china and freenet-usa finally get linked up), and that
>> they maintain there keyspace.
>
> Define "bunchup" ?

It seems to be luck of the draw, where the connections are between the  
two networks. Many times one network will catastrophically be absorbed  
into the other (keyspace-wise), other times it is split into 3 or 4  
segments.

> This is IMHO a very significant result:
> - Link-based swap pressure causes the two networks to keep their own  
> separate
> keyspaces.

I don't know... the pressures you mentioned (900ms and walking loops)  
are not in this sim, it is swaptheory-only. Link pressure *could* help  
keep a steady state, but I think it would only make the attack slower  
to both deploy and recover from; unless of course it becomes so slow  
that it is overcome by the location-randomizing of nodes in the network.

> [...]
> - It is not a viable attack to dangle a large virtual network off a  
> few
> connections.

I think it is, and particularly so (1) the larger the virtual network,  
and (2) the fewer the links into the real net.

> Unfortunately there are other viable attacks on swapping (read
> the Pitch Black paper, last year somebody appears to have deployed  
> it with
> considerable success although some of our problems were certainly  
> due to
> churn).

On my todo list.

> - An escape-route mechanism may be interesting, however given the  
> limited
> capacity it may be simplest to just rely on chance and ULPRs  
> propagating the
> most sought after content? The problem with this is those few  
> requests which
> cross the border will probably have low HTL when they get there, so  
> if the
> content isn't popular on the other side it might not be found. So  
> maybe we do
> need some sort of mechanism - but we will need a way to ration the  
> scarce
> resources according to popularity.
>>
>> Speculating about the swap probability...
>>
>> ASSUMING that the keyspace would be split (i.e. as the networks grow
>> up together) [which I still don't think is the case],
>
> What happens as we get more and more links between the networks?  
> Catastrophic
> merging? What about a few high capacity links? Etc.. more work can  
> be done
> here.

 From what I have run so far, the more links the slower the merging  
(and/or the more likely it is to arrive at an ackward-but-stable state).

--
Robert Hailey



From toad at amphibian.dyndns.org  Tue Feb  5 23:01:25 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Tue, 5 Feb 2008 23:01:25 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <0F272690-1DF1-4A9B-A215-22F9FC06B17D@freenetproject.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<47A7BACF.9000400@cs.ucl.ac.uk>
	<0F272690-1DF1-4A9B-A215-22F9FC06B17D@freenetproject.org>
Message-ID: <200802052301.30684.toad@amphibian.dyndns.org>

On Tuesday 05 February 2008 21:10, Robert Hailey wrote:
> 
> On Feb 4, 2008, at 7:24 PM, Michael Rogers wrote:
> 
> > Matthew Toseland wrote:
> >> Swapping creates this problem. Or does it? Could you perhaps do  
> >> some simulations of two networks of different sizes weakly linked  
> >> and show whether they get independant location spaces, or whether  
> >> swapping tries to put one of them within the global keyspace for  
> >> the other?
> >
> > Here's a quick simulation that shows that two weakly-connected  
> > subnets move into separate regions of the key space. Each subnet has  
> > an ideal Kleinberg topology and starts out uniformly distributed  
> > across the whole key space, and there are also a few random links  
> > between the subnets - this is meant to represent what would happen  
> > if you created a few links between two mature networks, or between a  
> > real network and a Sybil network.
> >
> > I couldn't be bothered to do a nice GUI so the output is just a  
> > series of histograms: on each line the key space is divided into 20  
> > regions, and each column shows the number of nodes from the first  
> > subnet in that region. Initially there are roughly 50 nodes in each  
> > region, but swapping causes the subnets to segregate so that  
> > eventually most regions are almost exclusively occupied by one  
> > subnet or the other.
> >
> > It's kind of interesting to compare this with "white flight" in  
> > sociology...
> >
> > Cheers,
> > Michael
> 
> Ok. I see it Michael's way now.
> 
> I'm not sure to what degree this effects our present network, but when  
> graphed out, the individual location movements from this swapping  
> simulator often mirror zothar's previous graph (of location jumping);  
> and while one network is making a hole in the other, the sliding/ 
> compression looks just like what I saw previously as network rotation.
> 
> This is quite a problem. In fact, this may be the fundamental problem  
> with swapping. Because of this, a sybil network could presently even  
> choose which segment of the keyspace to occupy with very few links.
> 
> The same network coloring/routing logic *might* be applicable to  
> swapping. That is, to simply confine swaps to the network they came  
> from. I'm not aware of another way to both secure sybil nets from  
> invasion and keep major keyspaces separate. Unfortunately it has the  
> obvious problem of a dependency feedback loop:
> 
> (1) swaps are fundamental to routing,
> (2) routing is required for my auth-ping idea,
> (3) these auth pings are required for secured network-id coloring,
> (4) then... could we use the network id's to modify swapping???
> 
> It seems like the network-id idea would have to be changed to be a  
> little more relaxed; either by not effecting swapping until we have  
> computed the assigned network-ids (fall-open while in transition), or  
> by simply accepting (for the moment) the most common network id from  
> our peer set (rather than strictly random id on startup).
> 
> My question is, *if* such an idea is considered valid and in such a  
> case how could we be assured that us labeling and isolating a subnet  
> is not what *keeps* it labeled as a subnet because it's routing is  
> messed up for lack of swapping? I guess that would require all the  
> bordering nodes to consider that simultaneously, which would be a rare  
> and unstable condition in a well connected network.

Over short distances we can expose the topology, does that help?

The other problem with swapping - which may also be a fatal flaw, and may be 
another variant of the same bug - is that an attacker can send bogus swap 
requests, which can be catastrophic.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080205/ad1f5925/attachment.pgp 

From robert at freenetproject.org  Tue Feb  5 23:39:53 2008
From: robert at freenetproject.org (Robert Hailey)
Date: Tue, 5 Feb 2008 17:39:53 -0600
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <200802051905.47618.toad@amphibian.dyndns.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
	<04775CD3-B6BB-490E-A816-6729867C3B83@freenetproject.org>
	<200802051905.47618.toad@amphibian.dyndns.org>
Message-ID: <ED15DC10-5C5B-420E-96DB-AFE8A86FDFCA@freenetproject.org>


On Feb 5, 2008, at 1:05 PM, Matthew Toseland wrote:
> Is this sufficiently well-defined for simulations yet? It seems to  
> me there
> will be threshold parameters and so on?

Is it well-defined enough to be implemented for simulations? The  
biggest threshold business will be in attempting to break up peers  
into network groups. Since this coloring mechanism would be oblivious  
to present routing, it could be deployed on the live network up unto  
any of the routing changes, plus the coloring my take so long from a  
zero-state that it would be necessary.

(1) Challange/Response Pings

Requires at least one new message type (PongWithSecret?CRAMPong?), but  
it may be better to separate it into it's own request/response pair  
(PingSecret/PongSecret). So... two message types, and one handler (for  
PingSecret) which is just a minor extension of the routableRequest.

(2) SecretNodePinger (much different task than present 'NodePinger')

Alerted to connect/disconnect/add/remove peernodes, this class would  
keep a running list of pingable nodes for each peer node; it does so  
by methodically setting 'secrets' in a node and sending requests for  
that secret from other nodes. Requires a maxHtl, and some time  
threshold (or percentage) to keep a peernode in the list despite  
missing a ping or two. Decision asto reusing the same secret for all  
connectivity to a given node or reseting it each time. Presumably not  
getting an ack from a set-secret/uid request is cause for backoff.  
Decision asto trying to ascertain connectivity from backed off nodes.  
I would suggest a separate message for alerting peers to the uid/ 
secret (rather than dealing with separate node ref/n2n queues etc).  
So... one message type, and a large class.

This should be easily testable in a simulation, but finding a good HTL  
may best be done on the live network (would we want to use 10?)

(3) NetworkIDManager

Take the lists from #2 and somehow try to intelligently break them up  
into separate network groups. It is expected that most of the nodes  
will be reachable from most of the other nodes; or that one or two  
nodes will be totally isolated... in the middle there is a definite  
threshold sensitivity here with respect to algorithms and constants.  
Also supply an assigned network-id on a per-peer baises, and one for  
the node at large. Requires adding a noderef-ish network id to get it  
to our peers, we might want to use a separate info message though, as  
we would be sending a potentially different id to each peer. Other  
contants include how long to wait between reanalyzing the pings from #2.

At least on a large scale, this would be more difficult to simulate,  
but doable.

-

At this point the coloring would have to be applied to the live  
network, as there is little point speculating about the wonders and  
thresholds of routing/swapping benefits if the coloring does not work,  
much less augmenting requests to carry network id's, etc. I suggest  
probe requests carrying back network id's for exploring the network  
here.

--
Robert Hailey



From ian.clarke at gmail.com  Tue Feb  5 23:41:00 2008
From: ian.clarke at gmail.com (Ian Clarke)
Date: Tue, 5 Feb 2008 17:41:00 -0600
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <47A8C53A.9090505@cs.ucl.ac.uk>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
	<200802041755.23675.toad@amphibian.dyndns.org>
	<47A7BACF.9000400@cs.ucl.ac.uk>
	<49071261-11C5-4FA5-B951-9A764A135092@freenetproject.org>
	<47A8C53A.9090505@cs.ucl.ac.uk>
Message-ID: <823242bd0802051541v50463eaalef2d9a3fd0858d51@mail.gmail.com>

On Feb 5, 2008 2:21 PM, Michael Rogers <m.rogers at cs.ucl.ac.uk> wrote:
> Robert Hailey wrote:
> > First, there is an equal chance of nodes from either network swapping.
> > In the freenet network (since swapping is done through the network) it
> > is not equal (if for no other cause than the scarce links between the
> > two networks).
>
> I was under the impression that the random walks for swapping were long
> enough to reach any node with roughly equal probability - I believe that
> was Oskar's intention. If the random walks aren't escaping from local
> clusters then we'll never be able to smooth out the clusters...

I'm not sure - lets say we have two clusters, call them A and B, each
with 1000 nodes, with 10 connections between the clusters, each
connection having 5 connections.

For a random walk of 10 hops to get from cluster A to cluster B, it
needs to find one of the 10 connections between them - yet there are
about 1,250 connections - a 1/125 probability per hop, or a 1/12.5
probability that a random walk will find one of them.

This means (if my math and assumptions are reasonable - and they might
not be) that there is only an (approximately) 1/12 probability that a
swap will occur between two nodes in different clusters.

Ian.

-- 
Email: ian at uprizer.com
Cell: +1 512 422 3588
Skype: sanity


From m.rogers at cs.ucl.ac.uk  Wed Feb  6 09:32:03 2008
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: 06 Feb 2008 09:32:03 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <200802052223.57761.toad@amphibian.dyndns.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<49071261-11C5-4FA5-B951-9A764A135092@freenetproject.org>
	<47A8C53A.9090505@cs.ucl.ac.uk>
	<200802052223.57761.toad@amphibian.dyndns.org>
Message-ID: <Prayer.1.0.16.0802060932030.39041@bells3.cs.ucl.ac.uk>

On Feb 5 2008, Matthew Toseland wrote:
>> I was under the impression that the random walks for swapping were long 
>> enough to reach any node with roughly equal probability - I believe that 
>> was Oskar's intention. If the random walks aren't escaping from local 
>> clusters then we'll never be able to smooth out the clusters...
>
> Yes and no - in the case of poorly connected networks, the sheer capacity 
> of the limited connections between the two networks produces a bias, 
> doesn't it?

Good point - so even if there's no explicit swap limit it might be hard for 
locations to move between clusters...

Here's an updated simulation with looping random walks and swap limits. The 
last column of the output shows the fraction of swaps accepted - if you 
multiply the swap limit by 4, all swaps are accepted so there's effectively 
no limit.

The swap limit does prevent the subnets from completely segragating (even 
without randomly resetting the locations), but there still are many columns 
with 10-20% or 80-90% red nodes.

Is this good or bad? It limits the impact of Sybil attacks, but on the 
other hand it suggests that some "natural" topologies might never become 
properly routable...

Cheers,
Michael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Simulation.java
Type: application/octet-stream
Size: 3862 bytes
Desc: Simulation.java
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080206/a9a0c951/attachment.obj 

From m.rogers at cs.ucl.ac.uk  Wed Feb  6 09:44:30 2008
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: 06 Feb 2008 09:44:30 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <823242bd0802051541v50463eaalef2d9a3fd0858d51@mail.gmail.com>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
	<200802041755.23675.toad@amphibian.dyndns.org>
	<47A7BACF.9000400@cs.ucl.ac.uk>
	<49071261-11C5-4FA5-B951-9A764A135092@freenetproject.org>
	<47A8C53A.9090505@cs.ucl.ac.uk>
	<823242bd0802051541v50463eaalef2d9a3fd0858d51@mail.gmail.com>
Message-ID: <Prayer.1.0.16.0802060944300.39041@bells3.cs.ucl.ac.uk>

On Feb 5 2008, Ian Clarke wrote:
>For a random walk of 10 hops to get from cluster A to cluster B, it
>needs to find one of the 10 connections between them - yet there are
>about 1,250 connections - a 1/125 probability per hop, or a 1/12.5
>probability that a random walk will find one of them.
>
>This means (if my math and assumptions are reasonable - and they might
>not be) that there is only an (approximately) 1/12 probability that a
>swap will occur between two nodes in different clusters.

I think your argument applies to short walks, but for longer walks the 
argument starts to cut both ways (excuse the pun): it's hard to get into 
clusters, but once you're in it's also hard to get out, so the probability 
of reaching any given node is less and less affected by the topology.

So I guess the question is how many hops we need to reach that point - 10? 
100? I was under the impression that short walks were adequate in 
small-world graphs, but maybe "short" means asymptotically small compared 
to the size of the graph, rather than usefully implementably short...

Cheers,
Michael


From m.rogers at cs.ucl.ac.uk  Wed Feb  6 09:57:05 2008
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: 06 Feb 2008 09:57:05 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <0F272690-1DF1-4A9B-A215-22F9FC06B17D@freenetproject.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
	<200802041755.23675.toad@amphibian.dyndns.org>
	<47A7BACF.9000400@cs.ucl.ac.uk>
	<0F272690-1DF1-4A9B-A215-22F9FC06B17D@freenetproject.org>
Message-ID: <Prayer.1.0.16.0802060957050.39041@bells3.cs.ucl.ac.uk>

On Feb 5 2008, Robert Hailey wrote:
>My question is, *if* such an idea is considered valid and in such a  
>case how could we be assured that us labeling and isolating a subnet  
>is not what *keeps* it labeled as a subnet because it's routing is  
>messed up for lack of swapping?

We seem to be caught between a rock and a hard place - either we try to 
merge clusters into a single key space, in which case we're vulnerable to 
Sybil attacks, or we give every cluster its own key space, in which case we 
lose the benefits of greedy routing and have to route between clusters 
explicitly. But perhaps either alternative is preferable to the current 
situation: we don't successfully merge clusters but we don't label them 
either...

Cheers,
Michael


From toad at amphibian.dyndns.org  Wed Feb  6 14:26:16 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Wed, 6 Feb 2008 14:26:16 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <Prayer.1.0.16.0802060932030.39041@bells3.cs.ucl.ac.uk>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<200802052223.57761.toad@amphibian.dyndns.org>
	<Prayer.1.0.16.0802060932030.39041@bells3.cs.ucl.ac.uk>
Message-ID: <200802061426.17854.toad@amphibian.dyndns.org>

On Wednesday 06 February 2008 09:32, Michael Rogers wrote:
> On Feb 5 2008, Matthew Toseland wrote:
> >> I was under the impression that the random walks for swapping were long 
> >> enough to reach any node with roughly equal probability - I believe that 
> >> was Oskar's intention. If the random walks aren't escaping from local 
> >> clusters then we'll never be able to smooth out the clusters...
> >
> > Yes and no - in the case of poorly connected networks, the sheer capacity 
> > of the limited connections between the two networks produces a bias, 
> > doesn't it?
> 
> Good point - so even if there's no explicit swap limit it might be hard for 
> locations to move between clusters...
> 
> Here's an updated simulation with looping random walks and swap limits. The 
> last column of the output shows the fraction of swaps accepted - if you 
> multiply the swap limit by 4, all swaps are accepted so there's effectively 
> no limit.
> 
> The swap limit does prevent the subnets from completely segragating (even 
> without randomly resetting the locations), but there still are many columns 
> with 10-20% or 80-90% red nodes.

"Red" nodes?
> 
> Is this good or bad? It limits the impact of Sybil attacks, but on the 
> other hand it suggests that some "natural" topologies might never become 
> properly routable...
> 
> Cheers,
> Michael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20080206/1986e3ed/attachment.pgp 

From m.rogers at cs.ucl.ac.uk  Wed Feb  6 14:37:45 2008
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: 06 Feb 2008 14:37:45 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <200802061426.17854.toad@amphibian.dyndns.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<200802052223.57761.toad@amphibian.dyndns.org>
	<Prayer.1.0.16.0802060932030.39041@bells3.cs.ucl.ac.uk>
	<200802061426.17854.toad@amphibian.dyndns.org>
Message-ID: <Prayer.1.0.16.0802061437450.41263@bells3.cs.ucl.ac.uk>

On Feb 6 2008, Matthew Toseland wrote:
>> The swap limit does prevent the subnets from completely segragating 
>> (even without randomly resetting the locations), but there still are 
>> many columns with 10-20% or 80-90% red nodes.
>
>"Red" nodes?

Oh sorry, I labelled the two subnets red and black. :-)

Cheers,
Michael


From ian.clarke at gmail.com  Wed Feb  6 16:25:59 2008
From: ian.clarke at gmail.com (Ian Clarke)
Date: Wed, 6 Feb 2008 10:25:59 -0600
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <Prayer.1.0.16.0802060944300.39041@bells3.cs.ucl.ac.uk>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
	<200802041755.23675.toad@amphibian.dyndns.org>
	<47A7BACF.9000400@cs.ucl.ac.uk>
	<49071261-11C5-4FA5-B951-9A764A135092@freenetproject.org>
	<47A8C53A.9090505@cs.ucl.ac.uk>
	<823242bd0802051541v50463eaalef2d9a3fd0858d51@mail.gmail.com>
	<Prayer.1.0.16.0802060944300.39041@bells3.cs.ucl.ac.uk>
Message-ID: <823242bd0802060825s7fd21d35sc71b356e8ae304ea@mail.gmail.com>

On 06 Feb 2008 09:44:30 +0000, Michael Rogers <m.rogers at cs.ucl.ac.uk> wrote:
> On Feb 5 2008, Ian Clarke wrote:
> >For a random walk of 10 hops to get from cluster A to cluster B, it
> >needs to find one of the 10 connections between them - yet there are
> >about 1,250 connections - a 1/125 probability per hop, or a 1/12.5
> >probability that a random walk will find one of them.
> >
> >This means (if my math and assumptions are reasonable - and they might
> >not be) that there is only an (approximately) 1/12 probability that a
> >swap will occur between two nodes in different clusters.
>
> I think your argument applies to short walks, but for longer walks the
> argument starts to cut both ways (excuse the pun): it's hard to get into
> clusters, but once you're in it's also hard to get out, so the probability
> of reaching any given node is less and less affected by the topology.

Yes, but only if there enough hops to virtually assure that the random
walk will stumble from one cluster to the other.

> So I guess the question is how many hops we need to reach that point - 10?
> 100? I was under the impression that short walks were adequate in
> small-world graphs, but maybe "short" means asymptotically small compared
> to the size of the graph, rather than usefully implementably short...

The scenario I outlined above may be too contrived, but I do think we
should simulate short walks to find nodes for random swapping, rather
than just assuming that a short walk will select a random node with
even probability.

Ian.

-- 
Email: ian at uprizer.com
Cell: +1 512 422 3588
Skype: sanity


From robert at freenetproject.org  Wed Feb  6 16:38:55 2008
From: robert at freenetproject.org (Robert Hailey)
Date: Wed, 6 Feb 2008 10:38:55 -0600
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <Prayer.1.0.16.0802060957050.39041@bells3.cs.ucl.ac.uk>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
	<200802041755.23675.toad@amphibian.dyndns.org>
	<47A7BACF.9000400@cs.ucl.ac.uk>
	<0F272690-1DF1-4A9B-A215-22F9FC06B17D@freenetproject.org>
	<Prayer.1.0.16.0802060957050.39041@bells3.cs.ucl.ac.uk>
Message-ID: <A4A878E6-7ACA-4756-8045-D245E2B3FBC9@freenetproject.org>


On Feb 6, 2008, at 3:57 AM, Michael Rogers wrote:

> On Feb 5 2008, Robert Hailey wrote:
>> My question is, *if* such an idea is considered valid and in such a
>> case how could we be assured that us labeling and isolating a subnet
>> is not what *keeps* it labeled as a subnet because it's routing is
>> messed up for lack of swapping?
>
> We seem to be caught between a rock and a hard place - either we try  
> to
> merge clusters into a single key space, in which case we're  
> vulnerable to
> Sybil attacks, or we give every cluster its own key space, in which  
> case we
> lose the benefits of greedy routing and have to route between clusters
> explicitly. But perhaps either alternative is preferable to the  
> current
> situation: we don't successfully merge clusters but we don't label  
> them
> either...
>
> Cheers,
> Michael

I think you're absolutely right.

What's more, I bet there should be a particular relationship between  
the number of hops used for swapping and the number of hops for a  
connectivity test; that they would both be raised or lowered to make  
"Sybil" detection more/less sensitive.

--
Robert Hailey



From m.rogers at cs.ucl.ac.uk  Wed Feb  6 21:16:46 2008
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Wed, 06 Feb 2008 21:16:46 +0000
Subject: [freenet-dev] Securing swaps? Was: Alpha, Darknet routing, et al.
In-Reply-To: <200802052301.30684.toad@amphibian.dyndns.org>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<47A7BACF.9000400@cs.ucl.ac.uk>
	<0F272690-1DF1-4A9B-A215-22F9FC06B17D@freenetproject.org>
	<200802052301.30684.toad@amphibian.dyndns.org>
Message-ID: <47AA23BE.2040402@cs.ucl.ac.uk>

Matthew Toseland wrote:
> The other problem with swapping - which may also be a fatal flaw, and may be 
> another variant of the same bug - is that an attacker can send bogus swap 
> requests, which can be catastrophic.

Currently an attacker can wait until it sees the other node's location 
and peer-locations, then reply with a location and peer-locations that 
will persuade the other node to swap, right?

I wonder if we can work out a way for the two swapping nodes to commit 
to their locations and peer-locations without revealing them until the 
swap has been agreed? (For example by sending the hash of the list 
instead of the list?)

An attacker could still abort the swap after agreeing, but at least it 
would have to pick locations by trial and error instead of choosing them 
after seeing those of the other node. And the limit on the number of 
swap requests per link would limit the amount of trial and error...

Cheers,
Michael


From m.rogers at cs.ucl.ac.uk  Wed Feb  6 21:23:07 2008
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Wed, 06 Feb 2008 21:23:07 +0000
Subject: [freenet-dev] Alpha, Darknet routing, et al.
In-Reply-To: <823242bd0802060825s7fd21d35sc71b356e8ae304ea@mail.gmail.com>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<73EE72A1-FADA-4471-AA0F-AD7E0E620572@freenetproject.org>
	<47A49940.9050802@cs.ucl.ac.uk>
	<200802041755.23675.toad@amphibian.dyndns.org>
	<47A7BACF.9000400@cs.ucl.ac.uk>
	<49071261-11C5-4FA5-B951-9A764A135092@freenetproject.org>
	<47A8C53A.9090505@cs.ucl.ac.uk>
	<823242bd0802051541v50463eaalef2d9a3fd0858d51@mail.gmail.com>
	<Prayer.1.0.16.0802060944300.39041@bells3.cs.ucl.ac.uk>
	<823242bd0802060825s7fd21d35sc71b356e8ae304ea@mail.gmail.com>
Message-ID: <47AA253B.4010507@cs.ucl.ac.uk>

Ian Clarke wrote:
> The scenario I outlined above may be too contrived, but I do think we
> should simulate short walks to find nodes for random swapping, rather
> than just assuming that a short walk will select a random node with
> even probability.

The version of the simulator I sent to the list this afternoon uses 
random walks for swapping - you can adjust WALK_STEPS and SWAP_LIMIT to 
see whether short random walks have a similar effect to limiting the 
amount of swap traffic through each node.

Cheers,
Michael


From toad at amphibian.dyndns.org  Wed Feb  6 22:09:57 2008
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Wed, 6 Feb 2008 22:09:57 +0000
Subject: [freenet-dev] Securing swaps? Was: Alpha, Darknet routing,
	et al.
In-Reply-To: <47AA23BE.2040402@cs.ucl.ac.uk>
References: <81C0E4CC-C600-47D3-A9D1-C9A8DAC30407@freenetproject.org>
	<200802052301.30684.toad@amphibian.dyndns.org>
	<47AA23BE.2040402@cs.ucl.ac.uk>
Message-ID: <200802062209.58489.toad@amphibian.dyndns.org>

On Wednesday 06 February 2008 21:16, Michael Rogers wrote:
> Matthew Toseland wrote:
> > The other problem with swapping - which may also be a fatal flaw, and may 
be 
> > another variant of the same bug - is that an attacker can send bogus swap 
> > requests, which can be catastrophic.
> 
> Currently an attacker can wait until it sees the other node's location 
> and peer-locations, then reply with a location and peer-locations that 
> will persuade the other node to swap, right?
> 
> I wonder if we can work out a way for the two swapping nodes to commit 
> to their locations and peer-locations without revealing them until the 
> swap has been agreed? (For example by sending the hash of the list 
> instead of the list?)

We already do this.
> 
> An attacker could still abort the swap after agreeing, but at least it 
> would have to pick locations by trial and error instead of choosing them 
> after seeing those of the other node. And the limit on the number of 
> swap requests per link would limit the amount of trial and error...

Well... you want to guarantee a swap by making A (the current product of the 
differences) greater than B (the product of the differences if we do the 
swap). You can do this by making A bigger or making B smaller.

Here's what the Routing in the Dark paper says:

"Suppose an attacker node A intends to force a swap with
a victim N so that L(N) = m afterwards. Let N have
k neighbors. Then A will initiate a swap request with N
claiming to have at least k + 1 neighbors with locations favoring
a swap according to Equation (1). Specifically, the
locations of the neighbors should be either close to L(N)
or close to the maximum distance from L(A) = m. The
a