From alejandro at mosteo.com Fri Dec 1 09:08:12 2006 From: alejandro at mosteo.com (Jano) Date: Fri, 01 Dec 2006 10:08:12 +0100 Subject: [Tech] Re: Re: The effect of slow nodes References: <456EB2E0.5000806@cs.ucl.ac.uk> <20061130141009.GA7128@amphibian.dyndns.org> <20061130195625.GB8269@amphibian.dyndns.org> Message-ID: toad wrote: > On Thu, Nov 30, 2006 at 05:27:51PM +0100, Jano wrote: >> Jano wrote: >> Note that this is with the version without slow nodes. > > Ahh, that explains it. :) > > Can you do the same simulation with slow nodes included? Is this in svn? *cue Rogers* From m.rogers at cs.ucl.ac.uk Fri Dec 1 09:37:53 2006 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Fri, 01 Dec 2006 09:37:53 +0000 Subject: [Tech] Re: Re: The effect of slow nodes In-Reply-To: References: <456EB2E0.5000806@cs.ucl.ac.uk> <20061130141009.GA7128@amphibian.dyndns.org> <20061130195625.GB8269@amphibian.dyndns.org> Message-ID: <456FF7F1.2080805@cs.ucl.ac.uk> Jano wrote: > Is this in svn? *cue Rogers* Revision 11135. *Exits stage left* From alejandro at mosteo.com Fri Dec 1 10:12:48 2006 From: alejandro at mosteo.com (Jano) Date: Fri, 01 Dec 2006 11:12:48 +0100 Subject: [Tech] Re: Re: Re: The effect of slow nodes References: <456EB2E0.5000806@cs.ucl.ac.uk> <20061130141009.GA7128@amphibian.dyndns.org> <20061130195625.GB8269@amphibian.dyndns.org> <456FF7F1.2080805@cs.ucl.ac.uk> Message-ID: Michael Rogers wrote: > Jano wrote: >> Is this in svn? *cue Rogers* > > Revision 11135. *Exits stage left* Uck, I thought the changes would be outside of phase7. Now I don't know what revision used for my tests. Re-running them... From alejandro at mosteo.com Fri Dec 1 16:41:52 2006 From: alejandro at mosteo.com (Jano) Date: Fri, 01 Dec 2006 17:41:52 +0100 Subject: [Tech] Re: Re: Re: The effect of slow nodes References: <456EB2E0.5000806@cs.ucl.ac.uk> <20061130141009.GA7128@amphibian.dyndns.org> <20061130195625.GB8269@amphibian.dyndns.org> <456FF7F1.2080805@cs.ucl.ac.uk> Message-ID: Jano wrote: > Michael Rogers wrote: > >> Jano wrote: >>> Is this in svn? *cue Rogers* >> >> Revision 11135. *Exits stage left* > > Uck, I thought the changes would be outside of phase7. Now I don't know > what revision used for my tests. Re-running them... Now with slow nodes, 2 .. 28 step 2. -------------- next part -------------- A non-text attachment was scrubbed... Name: ratio.png Type: image/png Size: 3956 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061201/51a6f430/attachment.png -------------- next part -------------- A non-text attachment was scrubbed... Name: throughput.png Type: image/png Size: 4302 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061201/51a6f430/attachment-0001.png From toad at amphibian.dyndns.org Fri Dec 1 17:24:37 2006 From: toad at amphibian.dyndns.org (toad) Date: Fri, 1 Dec 2006 17:24:37 +0000 Subject: [Tech] Re: Re: Re: The effect of slow nodes In-Reply-To: References: <456EB2E0.5000806@cs.ucl.ac.uk> <20061130141009.GA7128@amphibian.dyndns.org> <20061130195625.GB8269@amphibian.dyndns.org> <456FF7F1.2080805@cs.ucl.ac.uk> Message-ID: <20061201172437.GA18289@amphibian.dyndns.org> Interesting. I'm not entirely sure what these graphs tell us though, as explained earlier ... It is very surprising that slow nodes don't drag down the whole network; throttling without backoff seems competitive with, if not better than, throttling with backoff, even on a heterogenous network. Ideas? Explanations? On Fri, Dec 01, 2006 at 05:41:52PM +0100, Jano wrote: > Jano wrote: > > > Michael Rogers wrote: > > > >> Jano wrote: > >>> Is this in svn? *cue Rogers* > >> > >> Revision 11135. *Exits stage left* > > > > Uck, I thought the changes would be outside of phase7. Now I don't know > > what revision used for my tests. Re-running them... > > Now with slow nodes, 2 .. 28 step 2. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061201/16cdb8db/attachment.pgp From alejandro at mosteo.com Fri Dec 1 18:32:52 2006 From: alejandro at mosteo.com (Jano) Date: Fri, 01 Dec 2006 19:32:52 +0100 Subject: [Tech] Re: The effect of slow nodes References: <456EB2E0.5000806@cs.ucl.ac.uk> <20061130141009.GA7128@amphibian.dyndns.org> <20061130195625.GB8269@amphibian.dyndns.org> <456FF7F1.2080805@cs.ucl.ac.uk> <20061201172437.GA18289@amphibian.dyndns.org> Message-ID: toad wrote: > Interesting. I'm not entirely sure what these graphs tell us though, as > explained earlier ... > > It is very surprising that slow nodes don't drag down the whole network; Actually throughput seems lower and wilder; around 30k instead of 40k. More runs could clarify this. > throttling without backoff seems competitive with, if not better than, > throttling with backoff, even on a heterogenous network. > > Ideas? Explanations? I don't know what throttling and backoff are doing, so I'm unable to help here. From toad at amphibian.dyndns.org Fri Dec 1 19:00:08 2006 From: toad at amphibian.dyndns.org (toad) Date: Fri, 1 Dec 2006 19:00:08 +0000 Subject: [Tech] Re: The effect of slow nodes In-Reply-To: References: <20061130141009.GA7128@amphibian.dyndns.org> <20061130195625.GB8269@amphibian.dyndns.org> <456FF7F1.2080805@cs.ucl.ac.uk> <20061201172437.GA18289@amphibian.dyndns.org> Message-ID: <20061201190008.GA27488@amphibian.dyndns.org> On Fri, Dec 01, 2006 at 07:32:52PM +0100, Jano wrote: > toad wrote: > > > Interesting. I'm not entirely sure what these graphs tell us though, as > > explained earlier ... > > > > It is very surprising that slow nodes don't drag down the whole network; > > Actually throughput seems lower and wilder; around 30k instead of 40k. More > runs could clarify this. MRogers said there was a lot of variation between runs... more runs are therefore a good thing in any case. > > > throttling without backoff seems competitive with, if not better than, > > throttling with backoff, even on a heterogenous network. > > > > Ideas? Explanations? > > I don't know what throttling and backoff are doing, so I'm unable to help > here. Throttling = request senders slow down when they see overload messages or timeouts. Derived from TCP. Backoff = node doesn't route to another node for a while after a timeout or overload message. This is doubled the following time, until we reach a limit; it is reset if a request completes without overload or timeout. Derived from ethernet. The theory goes that if there are many slow nodes on the network, throttling would pull the whole network down to match their speed. This is why we have backoff; to prevent this from happening. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061201/f6ed9cdd/attachment.pgp From alejandro at mosteo.com Fri Dec 1 19:20:39 2006 From: alejandro at mosteo.com (Jano) Date: Fri, 01 Dec 2006 20:20:39 +0100 Subject: [Tech] Re: Re: The effect of slow nodes References: <20061130141009.GA7128@amphibian.dyndns.org> <20061130195625.GB8269@amphibian.dyndns.org> <456FF7F1.2080805@cs.ucl.ac.uk> <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> Message-ID: toad wrote: > On Fri, Dec 01, 2006 at 07:32:52PM +0100, Jano wrote: >> toad wrote: >> >> > Interesting. I'm not entirely sure what these graphs tell us though, as >> > explained earlier ... >> > >> > It is very surprising that slow nodes don't drag down the whole >> > network; >> >> Actually throughput seems lower and wilder; around 30k instead of 40k. >> More runs could clarify this. > > MRogers said there was a lot of variation between runs... more runs are > therefore a good thing in any case. I'll leave some running overnight. >> >> > throttling without backoff seems competitive with, if not better than, >> > throttling with backoff, even on a heterogenous network. >> > >> > Ideas? Explanations? >> >> I don't know what throttling and backoff are doing, so I'm unable to help >> here. > > Throttling = request senders slow down when they see overload messages > or timeouts. Derived from TCP. Slow down for all peers or only overloaded ones? > Backoff = node doesn't route to another node for a while after a timeout > or overload message. This is doubled the following time, until we reach > a limit; it is reset if a request completes without overload or timeout. > Derived from ethernet. Thanks. > The theory goes that if there are many slow nodes on the network, > throttling would pull the whole network down to match their speed. This > is why we have backoff; to prevent this from happening. Some more data apart from success/failure must be extracted from these sims to see what's really happening, I guess... From toad at amphibian.dyndns.org Fri Dec 1 19:39:34 2006 From: toad at amphibian.dyndns.org (toad) Date: Fri, 1 Dec 2006 19:39:34 +0000 Subject: [Tech] Re: Re: The effect of slow nodes In-Reply-To: References: <20061130195625.GB8269@amphibian.dyndns.org> <456FF7F1.2080805@cs.ucl.ac.uk> <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> Message-ID: <20061201193934.GA10118@amphibian.dyndns.org> On Fri, Dec 01, 2006 at 08:20:39PM +0100, Jano wrote: > toad wrote: > > > Throttling = request senders slow down when they see overload messages > > or timeouts. Derived from TCP. > > Slow down for all peers or only overloaded ones? Nodes sending requests slow down when they see overload messages. > > > Backoff = node doesn't route to another node for a while after a timeout > > or overload message. This is doubled the following time, until we reach > > a limit; it is reset if a request completes without overload or timeout. > > Derived from ethernet. > > Thanks. Thanks. > > > The theory goes that if there are many slow nodes on the network, > > throttling would pull the whole network down to match their speed. This > > is why we have backoff; to prevent this from happening. > > Some more data apart from success/failure must be extracted from these sims > to see what's really happening, I guess... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061201/f282894c/attachment.pgp From m.rogers at cs.ucl.ac.uk Fri Dec 1 19:47:13 2006 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Fri, 01 Dec 2006 19:47:13 +0000 Subject: [Tech] Re: Re: The effect of slow nodes In-Reply-To: <20061201193934.GA10118@amphibian.dyndns.org> References: <20061130195625.GB8269@amphibian.dyndns.org> <456FF7F1.2080805@cs.ucl.ac.uk> <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> <20061201193934.GA10118@amphibian.dyndns.org> Message-ID: <457086C1.2080502@cs.ucl.ac.uk> toad wrote: > Nodes sending requests slow down when they see overload messages. Only when the overload messages are triggered by their own searches though, right? Cheers, Michael From m.rogers at cs.ucl.ac.uk Fri Dec 1 19:47:36 2006 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Fri, 01 Dec 2006 19:47:36 +0000 Subject: [Tech] Re: Re: The effect of slow nodes In-Reply-To: References: <20061130141009.GA7128@amphibian.dyndns.org> <20061130195625.GB8269@amphibian.dyndns.org> <456FF7F1.2080805@cs.ucl.ac.uk> <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> Message-ID: <457086D8.4010707@cs.ucl.ac.uk> Jano wrote: > Some more data apart from success/failure must be extracted from these sims > to see what's really happening, I guess... There's a lot of logging code, but most of it's disabled to speed things up when we're only looking for one piece of information. You can either reenable logging for an entire class by setting LOG = true at the top of the file, or selectively remove 'if (LOG)' from the items you want to examine. I wouldn't recommend setting LOG = true in the lower layers (Peer and NetworkInterface) except in a small-scale sim, because they produce a lot of output. Cheers, Michael From toad at amphibian.dyndns.org Sat Dec 2 16:21:53 2006 From: toad at amphibian.dyndns.org (toad) Date: Sat, 2 Dec 2006 16:21:53 +0000 Subject: [Tech] Re: Re: The effect of slow nodes In-Reply-To: <457086C1.2080502@cs.ucl.ac.uk> References: <456FF7F1.2080805@cs.ucl.ac.uk> <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> <20061201193934.GA10118@amphibian.dyndns.org> <457086C1.2080502@cs.ucl.ac.uk> Message-ID: <20061202162153.GA12510@amphibian.dyndns.org> On Fri, Dec 01, 2006 at 07:47:13PM +0000, Michael Rogers wrote: > toad wrote: > >Nodes sending requests slow down when they see overload messages. > > Only when the overload messages are triggered by their own searches > though, right? Yes I think so. > > Cheers, > Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061202/b5f94536/attachment.pgp From alejandro at mosteo.com Sun Dec 3 14:43:22 2006 From: alejandro at mosteo.com (Jano) Date: Sun, 03 Dec 2006 15:43:22 +0100 Subject: [Tech] Re: The effect of slow nodes References: <20061130141009.GA7128@amphibian.dyndns.org> <20061130195625.GB8269@amphibian.dyndns.org> <456FF7F1.2080805@cs.ucl.ac.uk> <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> Message-ID: Here's an additional run, going from 1 to 30 as load in steps of 1. -------------- next part -------------- A non-text attachment was scrubbed... Name: ratio.png Type: image/png Size: 3628 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061203/e5216828/attachment.png -------------- next part -------------- A non-text attachment was scrubbed... Name: throughput.png Type: image/png Size: 3387 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061203/e5216828/attachment-0001.png From alejandro at mosteo.com Sun Dec 3 14:50:30 2006 From: alejandro at mosteo.com (Jano) Date: Sun, 03 Dec 2006 15:50:30 +0100 Subject: [Tech] Re: The effect of slow nodes References: <20061130141009.GA7128@amphibian.dyndns.org> <20061130195625.GB8269@amphibian.dyndns.org> <456FF7F1.2080805@cs.ucl.ac.uk> <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> Message-ID: Jano wrote: > Here's an additional run, going from 1 to 30 as load in steps of 1. My wrong, these were old runs. The correct ones are this: -------------- next part -------------- A non-text attachment was scrubbed... Name: ratio.png Type: image/png Size: 4488 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061203/23c228ab/attachment.png -------------- next part -------------- A non-text attachment was scrubbed... Name: throughput.png Type: image/png Size: 4555 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061203/23c228ab/attachment-0001.png From toad at amphibian.dyndns.org Tue Dec 5 16:57:21 2006 From: toad at amphibian.dyndns.org (toad) Date: Tue, 5 Dec 2006 16:57:21 +0000 Subject: [Tech] Re: The effect of slow nodes In-Reply-To: References: <456FF7F1.2080805@cs.ucl.ac.uk> <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> Message-ID: <20061205165721.GA17212@amphibian.dyndns.org> On Sun, Dec 03, 2006 at 03:50:30PM +0100, Jano wrote: > Jano wrote: > > > Here's an additional run, going from 1 to 30 as load in steps of 1. > > My wrong, these were old runs. The correct ones are this: Which old runs? They look completely different to the new ones - at least the purple line does. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061205/a5d44165/attachment.pgp From alejandro at mosteo.com Tue Dec 5 20:31:48 2006 From: alejandro at mosteo.com (Jano) Date: Tue, 05 Dec 2006 21:31:48 +0100 Subject: [Tech] Re: The effect of slow nodes References: <456FF7F1.2080805@cs.ucl.ac.uk> <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> <20061205165721.GA17212@amphibian.dyndns.org> Message-ID: toad wrote: > On Sun, Dec 03, 2006 at 03:50:30PM +0100, Jano wrote: >> Jano wrote: >> >> > Here's an additional run, going from 1 to 30 as load in steps of 1. >> >> My wrong, these were old runs. The correct ones are this: > > Which old runs? They look completely different to the new ones - at > least the purple line does. I have posted three runs. The first one is here: http://article.gmane.org/gmane.network.freenet.technical/3912 which I wrongly posted two messages above again. I used a source which I believed not to have slow nodes. Then it was pointed to me that that was the branch with slow nodes, but only after a particular revision. This was after I had updated my local copy, so I hadn't a clue about which particular version I ran. Summary: a run where may be there are slow nodes, or maybe not (This seems more likely since the throughput matches the first MRogers runs). Second run, with slow nodes and using the 2..28/+2 range: http://article.gmane.org/gmane.network.freenet.technical/3921 And final (third) run, range 1..30/+1: http://article.gmane.org/gmane.network.freenet.technical/3931 About the purple line: these simulations die by OOM errors in the simulator. In particular, I guess you're concerned by the drop. I'd not give it any meaning, it's almost surely a partial execution. I'll take care to remove these wrong results next time. Nonetheless, I'm going to check it and come back to confirm. From alejandro at mosteo.com Wed Dec 6 01:55:53 2006 From: alejandro at mosteo.com (Jano) Date: Wed, 06 Dec 2006 02:55:53 +0100 Subject: [Tech] Re: The effect of slow nodes References: <456FF7F1.2080805@cs.ucl.ac.uk> <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> <20061205165721.GA17212@amphibian.dyndns.org> Message-ID: Jano wrote: > About the purple line: these simulations die by OOM errors in the > simulator. In particular, I guess you're concerned by the drop. I'd not > give it any meaning, it's almost surely a partial execution. I'll take > care to remove these wrong results next time. > > Nonetheless, I'm going to check it and come back to confirm. Mmmmm I was wrong. Some of these simulations are ending properly and most request are failing with RNF. Even the ones dying because of OOM show prevalence of RNF. I'm going to run a more detailed series in the critical range where absence of balancing collapses. From toad at amphibian.dyndns.org Wed Dec 6 02:00:23 2006 From: toad at amphibian.dyndns.org (toad) Date: Wed, 6 Dec 2006 02:00:23 +0000 Subject: [Tech] What I don't understand... (simulations) Message-ID: <20061206020023.GA6987@amphibian.dyndns.org> I don't understand why slow nodes don't pull down the throttle. Is the throttle using the same parameters as on the network proper? The thing is, it seems to me that this is exactly what is happening on the main network - average bandwidth usage is very low, precisely because the throttles aren't starting many requests. Do the simulations approximate the current shouldRejectRequest() behaviour reasonably well? In particular do they emulate the predictive bandwidth allocation code? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061206/8052536e/attachment.pgp From m.rogers at cs.ucl.ac.uk Wed Dec 6 09:44:25 2006 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Wed, 06 Dec 2006 09:44:25 +0000 Subject: [Tech] What I don't understand... (simulations) In-Reply-To: <20061206020023.GA6987@amphibian.dyndns.org> References: <20061206020023.GA6987@amphibian.dyndns.org> Message-ID: <457690F9.1050008@cs.ucl.ac.uk> toad wrote: > I don't understand why slow nodes don't pull down the throttle. Is the > throttle using the same parameters as on the network proper? No, the simulated throttle is much simpler - no RTT measurement, one throttle for all inserts and requests rather than 4 separate throttles. To be honest I'm not sure RTT measurement makes much sense when each search follows a different path, but I'm willing to emulate the current behaviour more closely if it would be useful. > The thing is, it seems to me that this is exactly what is happening on > the main network - average bandwidth usage is very low, precisely > because the throttles aren't starting many requests. I don't think the simulations show low bandwidth usage, but I'll check. > Do the simulations approximate the current shouldRejectRequest() > behaviour reasonably well? In particular do they emulate the predictive > bandwidth allocation code? Bandwidth prediction and ping time aren't simulated, I'm afraid - shouldRejectRequest() is just based on a running average of the bandwidth/congestion delay (ie how long messages are delayed beyond their coalescing deadlines). I'm happy to make the simulations more realistic, but there's a tradeoff between testing new features (eg token passing) and simulating existing features accurately - I'm happy to do either, so let me know which would be more useful. Cheers, Michael From m.rogers at cs.ucl.ac.uk Wed Dec 6 09:51:48 2006 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Wed, 06 Dec 2006 09:51:48 +0000 Subject: [Tech] Re: The effect of slow nodes In-Reply-To: References: <456FF7F1.2080805@cs.ucl.ac.uk> <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> <20061205165721.GA17212@amphibian.dyndns.org> Message-ID: <457692B4.40601@cs.ucl.ac.uk> Jano wrote: > Mmmmm I was wrong. Some of these simulations are ending properly and most > request are failing with RNF. Even the ones dying because of OOM show > prevalence of RNF. I'm going to run a more detailed series in the critical > range where absence of balancing collapses. I would guess the RNFs are probably happening because of timeouts - when a search is not accepted by a peer within a certain amount of time the handler moves on to the next peer, and returns RNF when it runs out of peers. Perhaps you could enable logging of timeouts before you run your series? Cheers, Michael From alejandro at mosteo.com Wed Dec 6 20:26:01 2006 From: alejandro at mosteo.com (Jano) Date: Wed, 06 Dec 2006 21:26:01 +0100 Subject: [Tech] The effect of slow nodes References: <456FF7F1.2080805@cs.ucl.ac.uk> <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> <20061205165721.GA17212@amphibian.dyndns.org> Message-ID: > Jano wrote: > >> About the purple line: these simulations die by OOM errors in the >> simulator. In particular, I guess you're concerned by the drop. I'd not >> give it any meaning, it's almost surely a partial execution. I'll take >> care to remove these wrong results next time. >> >> Nonetheless, I'm going to check it and come back to confirm. > > Mmmmm I was wrong. Some of these simulations are ending properly and most > request are failing with RNF. Even the ones dying because of OOM show > prevalence of RNF. I'm going to run a more detailed series in the critical > range where absence of balancing collapses. This is yet another run in the range 6..8/0.2, now without OOMs... -------------- next part -------------- A non-text attachment was scrubbed... Name: detail.fff.6-8.png Type: image/png Size: 9501 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061206/39000292/attachment.png -------------- next part -------------- A non-text attachment was scrubbed... Name: detailtotal.fff.6-8.png Type: image/png Size: 9965 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061206/39000292/attachment-0001.png From toad at amphibian.dyndns.org Thu Dec 7 15:19:21 2006 From: toad at amphibian.dyndns.org (toad) Date: Thu, 7 Dec 2006 15:19:21 +0000 Subject: [Tech] Proposed Freenet MIME types Message-ID: <20061207151921.GA7635@amphibian.dyndns.org> I propose the following MIME types for Freenet: application/x-freenet-reference = .fref A Freenet darknet reference. application/x-freenet-index = .fridx A Freenet index file. There are two formats, we can detect which is in use reasonably easily. The old format will be deprecated when Librarian supports the new (XML-based) format (which is used by Thaw for indexes). application/x-freenet-blob = .fblob A Freenet "binary blob" file. This is a collection of node-level keys, and possibly some user-level keys. It allows us to export a freesite to a file, then reinsert it on another node, without having to know the private SSK key. (The x- indicates that these are unofficial MIME types; if they were ever registered they would lose the x-). -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/f40f20ee/attachment.pgp From alejandro at mosteo.com Thu Dec 7 15:36:35 2006 From: alejandro at mosteo.com (Jano) Date: Thu, 07 Dec 2006 16:36:35 +0100 Subject: [Tech] Proposed Freenet MIME types References: <20061207151921.GA7635@amphibian.dyndns.org> Message-ID: toad wrote: > I propose the following MIME types for Freenet: > > application/x-freenet-reference = .fref > > A Freenet darknet reference. > > application/x-freenet-index = .fridx > > A Freenet index file. There are two formats, we can detect which is in > use reasonably easily. The old format will be deprecated when Librarian > supports the new (XML-based) format (which is used by Thaw for indexes). Correct me if I'm mistaken; Thaw indexes are a list of files. Are there provisions for hierarchies of files within a single index? (Folders, Categories...). I think it would be useful. > application/x-freenet-blob = .fblob > > A Freenet "binary blob" file. This is a collection of node-level keys, > and possibly some user-level keys. It allows us to export a freesite to > a file, then reinsert it on another node, without having to know the > private SSK key. If the purpose of the blob is that specific, perhaps it would be better to reflect it in the MIME type. x-freenet-fsite i.e. > (The x- indicates that these are unofficial MIME types; if they were > ever registered they would lose the x-). From toad at amphibian.dyndns.org Thu Dec 7 15:51:52 2006 From: toad at amphibian.dyndns.org (toad) Date: Thu, 7 Dec 2006 15:51:52 +0000 Subject: [Tech] The effect of slow nodes In-Reply-To: References: <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> <20061205165721.GA17212@amphibian.dyndns.org> Message-ID: <20061207155152.GA29152@amphibian.dyndns.org> What's the difference between search failure, RNF and DNF? Search failure is a timeout? If so, this is with throttling? There shouldn't be _that_ many timeouts... On Wed, Dec 06, 2006 at 09:26:01PM +0100, Jano wrote: > > Jano wrote: > > > >> About the purple line: these simulations die by OOM errors in the > >> simulator. In particular, I guess you're concerned by the drop. I'd not > >> give it any meaning, it's almost surely a partial execution. I'll take > >> care to remove these wrong results next time. > >> > >> Nonetheless, I'm going to check it and come back to confirm. > > > > Mmmmm I was wrong. Some of these simulations are ending properly and most > > request are failing with RNF. Even the ones dying because of OOM show > > prevalence of RNF. I'm going to run a more detailed series in the critical > > range where absence of balancing collapses. > > This is yet another run in the range 6..8/0.2, now without OOMs... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/862cd698/attachment.pgp From toad at amphibian.dyndns.org Thu Dec 7 15:53:24 2006 From: toad at amphibian.dyndns.org (toad) Date: Thu, 7 Dec 2006 15:53:24 +0000 Subject: [Tech] Proposed Freenet MIME types In-Reply-To: References: <20061207151921.GA7635@amphibian.dyndns.org> Message-ID: <20061207155324.GB29152@amphibian.dyndns.org> On Thu, Dec 07, 2006 at 04:36:35PM +0100, Jano wrote: > toad wrote: > > > I propose the following MIME types for Freenet: > > > > application/x-freenet-reference = .fref > > > > A Freenet darknet reference. > > > > application/x-freenet-index = .fridx > > > > A Freenet index file. There are two formats, we can detect which is in > > use reasonably easily. The old format will be deprecated when Librarian > > supports the new (XML-based) format (which is used by Thaw for indexes). > > Correct me if I'm mistaken; Thaw indexes are a list of files. Are there > provisions for hierarchies of files within a single index? (Folders, > Categories...). I think it would be useful. I dunno, haven't looked at the format recently. I think the new format is on the wiki but I have no idea where; otherwise look at the source. :) It's XML so it's easy to extend... > > > application/x-freenet-blob = .fblob > > > > A Freenet "binary blob" file. This is a collection of node-level keys, > > and possibly some user-level keys. It allows us to export a freesite to > > a file, then reinsert it on another node, without having to know the > > private SSK key. > > If the purpose of the blob is that specific, perhaps it would be better to > reflect it in the MIME type. x-freenet-fsite i.e. It doesn't have to be a freesite, it could be a single file, for example. > > > (The x- indicates that these are unofficial MIME types; if they were > > ever registered they would lose the x-). -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/bb282e31/attachment.pgp From alejandro at mosteo.com Thu Dec 7 15:59:49 2006 From: alejandro at mosteo.com (Jano) Date: Thu, 07 Dec 2006 16:59:49 +0100 Subject: [Tech] The effect of slow nodes References: <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> <20061205165721.GA17212@amphibian.dyndns.org> <20061207155152.GA29152@amphibian.dyndns.org> Message-ID: toad wrote: > What's the difference between search failure, RNF and DNF? Search > failure is a timeout? If so, this is with throttling? There shouldn't be > _that_ many timeouts... I think two messages I sent yesterday have been lost, at least I don't see them through Gmane. This run is without any throttling mechanism. In one of the lost messages it seemed that the collapse point was in the 6-8 range. About the failure meanings, I leave this to Michael to answer. I'll repost the messages now. > > On Wed, Dec 06, 2006 at 09:26:01PM +0100, Jano wrote: >> > Jano wrote: >> > >> >> About the purple line: these simulations die by OOM errors in the >> >> simulator. In particular, I guess you're concerned by the drop. I'd >> >> not give it any meaning, it's almost surely a partial execution. I'll >> >> take care to remove these wrong results next time. >> >> >> >> Nonetheless, I'm going to check it and come back to confirm. >> > >> > Mmmmm I was wrong. Some of these simulations are ending properly and >> > most request are failing with RNF. Even the ones dying because of OOM >> > show prevalence of RNF. I'm going to run a more detailed series in the >> > critical range where absence of balancing collapses. >> >> This is yet another run in the range 6..8/0.2, now without OOMs... From alejandro at mosteo.com Thu Dec 7 16:03:23 2006 From: alejandro at mosteo.com (Jano) Date: Thu, 07 Dec 2006 17:03:23 +0100 Subject: [Tech] Proposed Freenet MIME types References: <20061207151921.GA7635@amphibian.dyndns.org> <20061207155324.GB29152@amphibian.dyndns.org> Message-ID: toad wrote: >> > application/x-freenet-blob = .fblob >> > >> > A Freenet "binary blob" file. This is a collection of node-level keys, >> > and possibly some user-level keys. It allows us to export a freesite to >> > a file, then reinsert it on another node, without having to know the >> > private SSK key. >> >> If the purpose of the blob is that specific, perhaps it would be better >> to reflect it in the MIME type. x-freenet-fsite i.e. > > It doesn't have to be a freesite, it could be a single file, for example. Ok. I mean, a blob can be anything, while it seems you have a definite purpose for the blob in mind. Perhaps there will be other uses for blobs in the future that will need discrimination. From alejandro at mosteo.com Thu Dec 7 16:12:34 2006 From: alejandro at mosteo.com (Jano) Date: Thu, 07 Dec 2006 17:12:34 +0100 Subject: [Tech] The effect of slow nodes References: <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> <20061205165721.GA17212@amphibian.dyndns.org> <20061207155152.GA29152@amphibian.dyndns.org> Message-ID: Michael Rogers wrote: > Jano wrote: >> Mmmmm I was wrong. Some of these simulations are ending properly and most >> request are failing with RNF. Even the ones dying because of OOM show >> prevalence of RNF. I'm going to run a more detailed series in the >> critical range where absence of balancing collapses. > > I would guess the RNFs are probably happening because of timeouts - when > a search is not accepted by a peer within a certain amount of time the > handler moves on to the next peer, and returns RNF when it runs out of > peers. Perhaps you could enable logging of timeouts before you run your > series? Trop tard! Meanwhile, here are some results. This is a run without balancing mechanism, in the range 6..12/0.5. Only simulations for 6, 6.5 and 7.5 finish without OOM, and they're certainly different than the others, so... In any case the first hour is always discarded, so the ones finishing early are doing it after the first hour. I need Michael to confirm that all successes not marked as local are indeed remote. I'll try to narrow to the 6..8 interval, giving some more memory... -------------- next part -------------- A non-text attachment was scrubbed... Name: detail.png Type: image/png Size: 7637 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/308f1d7d/attachment.png From toad at amphibian.dyndns.org Thu Dec 7 16:15:16 2006 From: toad at amphibian.dyndns.org (toad) Date: Thu, 7 Dec 2006 16:15:16 +0000 Subject: [Tech] Proposed Freenet MIME types In-Reply-To: References: <20061207151921.GA7635@amphibian.dyndns.org> <20061207155324.GB29152@amphibian.dyndns.org> Message-ID: <20061207161516.GA11385@amphibian.dyndns.org> On Thu, Dec 07, 2006 at 05:03:23PM +0100, Jano wrote: > toad wrote: > > >> > application/x-freenet-blob = .fblob > >> > > >> > A Freenet "binary blob" file. This is a collection of node-level keys, > >> > and possibly some user-level keys. It allows us to export a freesite to > >> > a file, then reinsert it on another node, without having to know the > >> > private SSK key. > >> > >> If the purpose of the blob is that specific, perhaps it would be better > >> to reflect it in the MIME type. x-freenet-fsite i.e. > > > > It doesn't have to be a freesite, it could be a single file, for example. > > Ok. I mean, a blob can be anything, while it seems you have a definite > purpose for the blob in mind. Perhaps there will be other uses for blobs in > the future that will need discrimination. "Binary blobs" was the original proposal's name. What it is is a bunch of node-level keys exported to the client for transport between networks (insert your favourite site to a disconnected darknet), or for verifiability (verify that a freenet update, or a revocation cert, really is signed by the SSK it's supposed to be signed by; this is why binary blobs are needed for update over mandatory). -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/2d77bc6d/attachment.pgp From toad at amphibian.dyndns.org Thu Dec 7 16:21:47 2006 From: toad at amphibian.dyndns.org (toad) Date: Thu, 7 Dec 2006 16:21:47 +0000 Subject: [Tech] What I don't understand... (simulations) In-Reply-To: <457690F9.1050008@cs.ucl.ac.uk> References: <20061206020023.GA6987@amphibian.dyndns.org> <457690F9.1050008@cs.ucl.ac.uk> Message-ID: <20061207162147.GB17913@amphibian.dyndns.org> On Wed, Dec 06, 2006 at 09:44:25AM +0000, Michael Rogers wrote: > toad wrote: > >I don't understand why slow nodes don't pull down the throttle. Is the > >throttle using the same parameters as on the network proper? > > No, the simulated throttle is much simpler - no RTT measurement, one > throttle for all inserts and requests rather than 4 separate throttles. > To be honest I'm not sure RTT measurement makes much sense when each > search follows a different path, but I'm willing to emulate the current > behaviour more closely if it would be useful. If there is no RTT measurement, then how does it work? It only does the additive increase multiplicative decrease part, on the rate of sending requests? Or decrease/increase with the interval between requests? Can you compare this with the current algorithm, which includes the RTT? It may be that your simplified algorithm is better than the strict copy of TCP which is currently deployed (which measures the RTT). > > >The thing is, it seems to me that this is exactly what is happening on > >the main network - average bandwidth usage is very low, precisely > >because the throttles aren't starting many requests. > > I don't think the simulations show low bandwidth usage, but I'll check. The real network does. What we want to discover is why... > > >Do the simulations approximate the current shouldRejectRequest() > >behaviour reasonably well? In particular do they emulate the predictive > >bandwidth allocation code? > > Bandwidth prediction and ping time aren't simulated, I'm afraid - > shouldRejectRequest() is just based on a running average of the > bandwidth/congestion delay (ie how long messages are delayed beyond > their coalescing deadlines). > > I'm happy to make the simulations more realistic, but there's a tradeoff > between testing new features (eg token passing) and simulating existing > features accurately - I'm happy to do either, so let me know which would > be more useful. Please simulate the current algorithm with RTT measurement. This was put in to closely match TCP. If it doesn't work very well we can replace it. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/a5347b28/attachment.pgp From alejandro at mosteo.com Thu Dec 7 16:16:00 2006 From: alejandro at mosteo.com (Jano) Date: Thu, 07 Dec 2006 17:16:00 +0100 Subject: [Tech] LIFO simulations Message-ID: I have some preliminary results using LIFO queues. What I've done is changing all queue insertions so request are put at head instead of at tail. There's one queue in Node.java and three in Peer.java; I've changed all. Michael could comment on some brokeness this could introduce, albeit simulations seem to run correctly. There are three sets of results each with ratio/total graphs. All are in the 6-12 range; graphs with only lifo queues use 0.5 as increment, while where are lifo and fifo results increment is 1 (I was reusing results from other runs). First set is LIFO alone and with the balancing mechanisms (though I don't know if they make sense in this case). Second set is LIFO alone against FIFO with balancing mechanisms. Third set is the three balancing mechanisms with both LIFO and FIFO. Results seem promising, although it seems LIFO alone will also collapse. I'll try next with a larger load range and the eight combinations in a single graph. -------------- next part -------------- A non-text attachment was scrubbed... Name: lifo-fifo-balancing-ratio-6-12.png Type: image/png Size: 5352 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/c7cc77e8/attachment.png -------------- next part -------------- A non-text attachment was scrubbed... Name: lifo-fifo-balancing-total-6-12.png Type: image/png Size: 7136 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/c7cc77e8/attachment-0001.png -------------- next part -------------- A non-text attachment was scrubbed... Name: lifo-fifo-ratio-6-12.png Type: image/png Size: 4981 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/c7cc77e8/attachment-0002.png -------------- next part -------------- A non-text attachment was scrubbed... Name: lifo-fifo-total-6-12.png Type: image/png Size: 5209 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/c7cc77e8/attachment-0003.png -------------- next part -------------- A non-text attachment was scrubbed... Name: lifo-lifo-ratio-6-12.png Type: image/png Size: 5315 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/c7cc77e8/attachment-0004.png -------------- next part -------------- A non-text attachment was scrubbed... Name: lifo-lifo-total-6-12.png Type: image/png Size: 5627 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/c7cc77e8/attachment-0005.png From toad at amphibian.dyndns.org Thu Dec 7 16:19:18 2006 From: toad at amphibian.dyndns.org (toad) Date: Thu, 7 Dec 2006 16:19:18 +0000 Subject: [Tech] The effect of slow nodes In-Reply-To: <457692B4.40601@cs.ucl.ac.uk> References: <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> <20061205165721.GA17212@amphibian.dyndns.org> <457692B4.40601@cs.ucl.ac.uk> Message-ID: <20061207161918.GA17913@amphibian.dyndns.org> On Wed, Dec 06, 2006 at 09:51:48AM +0000, Michael Rogers wrote: > Jano wrote: > >Mmmmm I was wrong. Some of these simulations are ending properly and most > >request are failing with RNF. Even the ones dying because of OOM show > >prevalence of RNF. I'm going to run a more detailed series in the critical > >range where absence of balancing collapses. > > I would guess the RNFs are probably happening because of timeouts - when > a search is not accepted by a peer within a certain amount of time the > handler moves on to the next peer, and returns RNF when it runs out of > peers. Perhaps you could enable logging of timeouts before you run your > series? Timeouts should not happen with the current code. They do, but they should be rare; pre-emptive reject should generally avoid them. What I want to know is why the bandwidth usage is so low on the network at present. Probably this is some interaction between pre-emptive rejection and sender side load limiting (and maybe backoff). As I've explained I'm skeptical about the results appearing to show that backoff is always bad... > > Cheers, > Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/aeead916/attachment.pgp From freenetwork at web.de Thu Dec 7 18:06:26 2006 From: freenetwork at web.de (freenetwork at web.de) Date: Thu, 07 Dec 2006 19:06:26 +0100 Subject: [Tech] Proposed Freenet MIME types In-Reply-To: <20061207151921.GA7635@amphibian.dyndns.org> Message-ID: >I propose the following MIME types for Freenet: >application/x-freenet-reference = .fref >A Freenet darknet reference. okay >application/x-freenet-index = .fridx shouldn't this be .fidx !!? >A Freenet index file. There are two formats, we can detect which is in >use reasonably easily. The old format will be deprecated when Librarian >supports the new (XML-based) format (which is used by Thaw for indexes). i'd say to only use this if it can handle almost all indexing structures (folders, files, comments, ...) mime-types are quasi-standards >application/x-freenet-blob = .fblob >A Freenet "binary blob" file. This is a collection of node-level keys, >and possibly some user-level keys. It allows us to export a freesite to >a file, then reinsert it on another node, without having to know the >private SSK key. 1) what exactly does this file contain? manifests? data too? 2) please explain the different key levels! 3) why the hell "blob"?? blob is *void so use x-binary or something like that - if you want a "real" mimetype, let it have a descriptive name! bad suggestions from me: .fce "x-freenet-contentexport", .flle "-lowlevelexport", .fdp "-datapackage", .fcnc "- cannednetworkcontent", a wild combination of these or something else, but "blob" it unnecessarily weird >(The x- indicates that these are unofficial MIME types; if they were >ever registered they would lose the x-). >--bp/iNruPH9dso1Pn >Content-Type: application/pgp-signature; name="signature.asc" >Content-Description: Digital signature >Content-Disposition: inline >-----BEGIN PGP SIGNATURE----- >Version: GnuPG v1.4.5 (GNU/Linux) >iD8DBQFFeDD5A9rUluQ9pFARAmo/AJ9A0O88mkrulthho2u4DabdHvgLDACfcphL >YvK+Ywgpn7p4FXuMzumXGUw= >=JUwS >-----END PGP SIGNATURE----- >--bp/iNruPH9dso1Pn-- >--===============0275209801== >Content-Type: text/plain; charset="us-ascii" >MIME-Version: 1.0 >Content-Transfer-Encoding: 7bit >Content-Disposition: inline >_______________________________________________ >Tech mailing list >Tech at freenetproject.org >http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech >--===============0275209801==-- From toad at amphibian.dyndns.org Thu Dec 7 18:39:42 2006 From: toad at amphibian.dyndns.org (toad) Date: Thu, 7 Dec 2006 18:39:42 +0000 Subject: [Tech] Proposed Freenet MIME types In-Reply-To: References: <20061207151921.GA7635@amphibian.dyndns.org> Message-ID: <20061207183942.GA14621@amphibian.dyndns.org> On Thu, Dec 07, 2006 at 07:06:26PM +0100, freenetwork at web.de wrote: > >I propose the following MIME types for Freenet: > > >application/x-freenet-reference = .fref > > >A Freenet darknet reference. > > okay > > >application/x-freenet-index = .fridx > > shouldn't this be .fidx !!? It should, but .fidx is taken, several times over. > > >A Freenet index file. There are two formats, we can detect which is in > >use reasonably easily. The old format will be deprecated when Librarian > >supports the new (XML-based) format (which is used by Thaw for indexes). > > i'd say to only use this if it can handle almost all indexing structures (folders, files, comments, ...) > mime-types are quasi-standards I don't see what the problem is. Tags we don't understand we just ignore in XML... > > >application/x-freenet-blob = .fblob > > >A Freenet "binary blob" file. This is a collection of node-level keys, > >and possibly some user-level keys. It allows us to export a freesite to > >a file, then reinsert it on another node, without having to know the > >private SSK key. > > 1) what exactly does this file contain? manifests? data too? Node-level keys. All those involved in fetching or inserting the file. Both CHKs and SSKs. > 2) please explain the different key levels! These are the actual network keys. Not the file you get when you download them via fproxy. The point is, they include the original SSK, so you can a) reinsert a freesite despite not having the private key, b) verify that it was created using the private key. > 3) why the hell "blob"?? blob is *void so use x-binary or something like that - if you want a "real" mimetype, let it have a descriptive name! bad suggestions from me: .fce "x-freenet-contentexport", .flle "-lowlevelexport", .fdp "-datapackage", .fcnc "- > cannednetworkcontent", a wild combination of these or something else, but "blob" it unnecessarily weird The original proposal was for "binary blobs". Maybe it should be a different name. For example? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/048b03ae/attachment.pgp From m.rogers at cs.ucl.ac.uk Thu Dec 7 20:24:13 2006 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Thu, 07 Dec 2006 20:24:13 +0000 Subject: [Tech] The effect of slow nodes In-Reply-To: <20061207155152.GA29152@amphibian.dyndns.org> References: <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> <20061205165721.GA17212@amphibian.dyndns.org> <20061207155152.GA29152@amphibian.dyndns.org> Message-ID: <4578786D.4030704@cs.ucl.ac.uk> toad wrote: > What's the difference between search failure, RNF and DNF? Search > failure is a timeout? "failed (search)" means the search timed out at the originating node - a peer accepted the search but then didn't return a result. "failed (rnf)" means the search ran out of nodes before running of hops. However, this could be caused by timeouts, because if we time out waiting for a peer to accept the search we move on to another peer. "failed (dnf)" means the search ran out of hops before running out of nodes. This only applies to requests, not inserts, and doesn't indicate a timeout. > If so, this is with throttling? There shouldn't be > _that_ many timeouts... Agreed, timeouts should be rare with any kind of flow control. I believe the results Jano posted were without flow control. It seems that RNFs can indicate overload, not just bad connectivity, but when we get an RNF we *increase* the throttle. I might not have understood this correctly - after rereading your message (copied below) it looks like maybe this should only apply to inserts, not requests? > On Sat, Nov 18, 2006 at 12:10:39PM +0000, Michael Rogers wrote: >> > If an insert generates a RouteNotFound but we successfully sent the data >> > to at least one node, should we count it as a success or a failure from >> > the point of view of throttling? > > Success. From the point of view of throttling, it's only a failure if we > get a RejectedOverload or a timeout. Cheers, Michael From m.rogers at cs.ucl.ac.uk Thu Dec 7 20:39:29 2006 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Thu, 07 Dec 2006 20:39:29 +0000 Subject: [Tech] What I don't understand... (simulations) In-Reply-To: <20061207162147.GB17913@amphibian.dyndns.org> References: <20061206020023.GA6987@amphibian.dyndns.org> <457690F9.1050008@cs.ucl.ac.uk> <20061207162147.GB17913@amphibian.dyndns.org> Message-ID: <45787C01.9030803@cs.ucl.ac.uk> toad wrote: > If there is no RTT measurement, then how does it work? It only does the > additive increase multiplicative decrease part, on the rate of sending > requests? Right. The node keeps locally generated searches in a queue. There's no limit on how quickly searches can be added to the queue, but there's an enforced delay between releasing searches from the queue. The enforced delay is 1/r, where r (the search rate) is additively increased whenever a search succeeds and multiplicatively decreased whenever a search fails. When the throttle's disabled, locally generated searches are processed immediately. > Can > you compare this with the current algorithm, which includes the RTT? It > may be that your simplified algorithm is better than the strict copy of > TCP which is currently deployed (which measures the RTT). Will do. Could you give me a quick idea of what RTT means in this context? Do you measure the time between generating the search and receiving a reply, or between sending the search and receiving a reply? Do searches that time out or otherwise fail count towards the RTT? Thanks, Michael From m.rogers at cs.ucl.ac.uk Thu Dec 7 20:48:20 2006 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Thu, 07 Dec 2006 20:48:20 +0000 Subject: [Tech] The effect of slow nodes In-Reply-To: <20061207161918.GA17913@amphibian.dyndns.org> References: <20061201172437.GA18289@amphibian.dyndns.org> <20061201190008.GA27488@amphibian.dyndns.org> <20061205165721.GA17212@amphibian.dyndns.org> <457692B4.40601@cs.ucl.ac.uk> <20061207161918.GA17913@amphibian.dyndns.org> Message-ID: <45787E14.3030304@cs.ucl.ac.uk> toad wrote: > As I've > explained I'm skeptical about the results appearing to show that backoff > is always bad... Agreed, advisory backoff seems to do a pretty good job. However, without throttling or token passing to limit the number of searches entering the network it seems like it ought to collapse *eventually*, albeit at a much higher load than without backoff. I've run sims up to a load of 50 with no problems - at 55 I started to get OOM errors for backoff alone, but I haven't looked into the causes yet. Cheers, Michael From m.rogers at cs.ucl.ac.uk Thu Dec 7 21:18:24 2006 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Thu, 07 Dec 2006 21:18:24 +0000 Subject: [Tech] LIFO simulations In-Reply-To: References: Message-ID: <45788520.4010306@cs.ucl.ac.uk> Jano wrote: > I have some preliminary results using LIFO queues. What I've done is > changing all queue insertions so request are put at head instead of at > tail. There's one queue in Node.java and three in Peer.java; I've changed > all. Michael could comment on some brokeness this could introduce, albeit > simulations seem to run correctly. It shouldn't break anything, but it seems like it might be inefficient to allow existing searches to time out while new searches take priority. On the other hand the same argument would seem to apply to LIFO router queues, so my intuition is probably wrong. By the way, reversing the node's queue shouldn't make any difference - no timers are started until the search leaves the queue. > Results seem promising, although it seems LIFO alone will also collapse. > I'll try next with a larger load range and the eight combinations in a > single graph. Looking forward to it! Backoff alone seems to get good throughput at high loads, but with a poor success rate (see the attached graphs, averaged over three runs) - I'll be interested to see whether LIFO's high success rate can be combined with backoff's high throughput. Cheers, Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: success-rate.png Type: image/png Size: 4694 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/a59b743a/attachment.png -------------- next part -------------- A non-text attachment was scrubbed... Name: throughput.png Type: image/png Size: 5574 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061207/a59b743a/attachment-0001.png From toad at amphibian.dyndns.org Fri Dec 8 01:59:23 2006 From: toad at amphibian.dyndns.org (toad) Date: Fri, 8 Dec 2006 01:59:23 +0000 Subject: [Tech] The effect of slow nodes In-Reply-To: <45787E14.3030304@cs.ucl.ac.uk> References: <20061201190008.GA27488@amphibian.dyndns.org> <20061205165721.GA17212@amphibian.dyndns.org> <457692B4.40601@cs.ucl.ac.uk> <20061207161918.GA17913@amphibian.dyndns.org> <45787E14.3030304@cs.ucl.ac.uk> Message-ID: <20061208015923.GA15016@amphibian.dyndns.org> On Thu, Dec 07, 2006 at 08:48:20PM +0000, Michael Rogers wrote: > toad wrote: > > As I've > > explained I'm skeptical about the results appearing to show that backoff > > is always bad... > > Agreed, advisory backoff seems to do a pretty good job. However, without > throttling or token passing to limit the number of searches entering the > network it seems like it ought to collapse *eventually*, albeit at a > much higher load than without backoff. I've run sims up to a load of 50 > with no problems - at 55 I started to get OOM errors for backoff alone, > but I haven't looked into the causes yet. Well there are two issues here: 1. Simulations don't seem to show a consistent lead for backoff plus throttling over just throttling. At least, recent ones - do yours? If this is true, then why? 2. Why is the real network so slow? It uses far less bandwidth than the specified limit; this suggests that the throttling has gone mad, or something (pre-emptive rejection) has broken it. Can we reproduce this in the simulation, and show why it happens / that it doesn't happen if we slightly change things? > > Cheers, > Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061208/f65deb61/attachment.pgp From toad at amphibian.dyndns.org Fri Dec 8 02:05:35 2006 From: toad at amphibian.dyndns.org (toad) Date: Fri, 8 Dec 2006 02:05:35 +0000 Subject: [Tech] The effect of slow nodes In-Reply-To: <4578786D.4030704@cs.ucl.ac.uk> References: <20061201190008.GA27488@amphibian.dyndns.org> <20061205165721.GA17212@amphibian.dyndns.org> <20061207155152.GA29152@amphibian.dyndns.org> <4578786D.4030704@cs.ucl.ac.uk> Message-ID: <20061208020535.GB15016@amphibian.dyndns.org> On Thu, Dec 07, 2006 at 08:24:13PM +0000, Michael Rogers wrote: > toad wrote: > > What's the difference between search failure, RNF and DNF? Search > > failure is a timeout? > > "failed (search)" means the search timed out at the originating node - a > peer accepted the search but then didn't return a result. Okay, so it's a timeout. > > "failed (rnf)" means the search ran out of nodes before running of hops. > However, this could be caused by timeouts, because if we time out > waiting for a peer to accept the search we move on to another peer. Right. But the timeout is fixed on each hop, so very often the first node will timeout by the time the later node times out. > > "failed (dnf)" means the search ran out of hops before running out of > nodes. This only applies to requests, not inserts, and doesn't indicate > a timeout. > > > If so, this is with throttling? There shouldn't be > > _that_ many timeouts... > > Agreed, timeouts should be rare with any kind of flow control. I believe > the results Jano posted were without flow control. > > It seems that RNFs can indicate overload, not just bad connectivity, but > when we get an RNF we *increase* the throttle. I might not have > understood this correctly - after rereading your message (copied below) > it looks like maybe this should only apply to inserts, not requests? The implemented algorithm is that we increase the rate of sending requests if we complete without timing out ourselves, or receiving a timeout notification from another node involved in the request (these are propagated back to source). By the way, did you try implementing the throttle with RTT, and comparing to throttle without? > > > On Sat, Nov 18, 2006 at 12:10:39PM +0000, Michael Rogers wrote: > >> > If an insert generates a RouteNotFound but we successfully sent the data > >> > to at least one node, should we count it as a success or a failure from > >> > the point of view of throttling? > > > > Success. From the point of view of throttling, it's only a failure if we > > get a RejectedOverload or a timeout. > > Cheers, > Michael > _______________________________________________ > Tech mailing list > Tech at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061208/1116f0e6/attachment.pgp From toad at amphibian.dyndns.org Fri Dec 8 02:10:52 2006 From: toad at amphibian.dyndns.org (toad) Date: Fri, 8 Dec 2006 02:10:52 +0000 Subject: [Tech] What I don't understand... (simulations) In-Reply-To: <45787C01.9030803@cs.ucl.ac.uk> References: <20061206020023.GA6987@amphibian.dyndns.org> <457690F9.1050008@cs.ucl.ac.uk> <20061207162147.GB17913@amphibian.dyndns.org> <45787C01.9030803@cs.ucl.ac.uk> Message-ID: <20061208021052.GC15016@amphibian.dyndns.org> On Thu, Dec 07, 2006 at 08:39:29PM +0000, Michael Rogers wrote: > toad wrote: > > If there is no RTT measurement, then how does it work? It only does the > > additive increase multiplicative decrease part, on the rate of sending > > requests? > > Right. The node keeps locally generated searches in a queue. There's no > limit on how quickly searches can be added to the queue, but there's an > enforced delay between releasing searches from the queue. The enforced > delay is 1/r, where r (the search rate) is additively increased whenever > a search succeeds and multiplicatively decreased whenever a search fails. > > When the throttle's disabled, locally generated searches are processed > immediately. > > > Can > > you compare this with the current algorithm, which includes the RTT? It > > may be that your simplified algorithm is better than the strict copy of > > TCP which is currently deployed (which measures the RTT). > > Will do. Could you give me a quick idea of what RTT means in this > context? Do you measure the time between generating the search and > receiving a reply, or between sending the search and receiving a reply? > Do searches that time out or otherwise fail count towards the RTT? Time between sending the search and completing the request. Success, DNF, RNF, verify failure all count for requests. For inserts, we wait for the all transfers completed notification. > > Thanks, > Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061208/0f3084c1/attachment.pgp From toad at amphibian.dyndns.org Fri Dec 8 02:12:25 2006 From: toad at amphibian.dyndns.org (toad) Date: Fri, 8 Dec 2006 02:12:25 +0000 Subject: [Tech] LIFO simulations In-Reply-To: <45788520.4010306@cs.ucl.ac.uk> References: <45788520.4010306@cs.ucl.ac.uk> Message-ID: <20061208021225.GD15016@amphibian.dyndns.org> On Thu, Dec 07, 2006 at 09:18:24PM +0000, Michael Rogers wrote: > Jano wrote: > >I have some preliminary results using LIFO queues. What I've done is > >changing all queue insertions so request are put at head instead of at > >tail. There's one queue in Node.java and three in Peer.java; I've changed > >all. Michael could comment on some brokeness this could introduce, albeit > >simulations seem to run correctly. > > It shouldn't break anything, but it seems like it might be inefficient > to allow existing searches to time out while new searches take priority. > On the other hand the same argument would seem to apply to LIFO router > queues, so my intuition is probably wrong. LIFO router queues? > > By the way, reversing the node's queue shouldn't make any difference - > no timers are started until the search leaves the queue. > > >Results seem promising, although it seems LIFO alone will also collapse. > >I'll try next with a larger load range and the eight combinations in a > >single graph. > > Looking forward to it! Backoff alone seems to get good throughput at > high loads, but with a poor success rate (see the attached graphs, > averaged over three runs) - I'll be interested to see whether LIFO's > high success rate can be combined with backoff's high throughput. > > Cheers, > Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20061208/743f0562/attachment.pgp From jflesch at gmail.com Thu Dec 7 19:35:17 2006 From: jflesch at gmail.com (Jerome Flesch) Date: Thu, 7 Dec 2006 20:35:17 +0100 Subject: [Tech] Proposed Freenet MIME types In-Reply-To: References: <20061207151921.GA7635@amphibian.dyndns.org> Message-ID: <200612072035.18063.jflesch@gmail.com> > toad wrote: > > I propose the following MIME types for Freenet: > > > > application/x-freenet-reference = .fref > > > > A Freenet darknet reference. > > > > application/x-freenet-index = .fridx > > > > A Freenet index file. There are two formats, we can detect which is in > > use reasonably easily. The old format will be deprecated when Librarian > > supports the new (XML-based) format (which is used by Thaw for indexes). > > Correct me if I'm mistaken; Thaw indexes are a list of files. Are there > provisions for hierarchies of files within a single index? (Folders, > Categories...). I think it would be useful. > http://wiki.freenetproject.org/AnotherFreenetIndexFormat It doesn't allow to do a complete hierarchie, but you can use the tag