From nextgens at freenetproject.org Tue Jul 1 04:02:06 2008 From: nextgens at freenetproject.org (NextGen$) Date: Tue, 1 Jul 2008 04:02:06 +0000 Subject: [Tech] Switch Repository to Git In-Reply-To: <200806300031.03920.bombe@pterodactylus.net> References: <200806300031.03920.bombe@pterodactylus.net> Message-ID: <20080701040206.GB26659@freenetproject.org> * David ???Bombe??? Roden [2008-06-30 00:31:03]: > Hey folks, > > I have been looking at and working with Git for the last four or five weeks > and I must say that I???m very impressed by it. It???s easy to use, We must have different definitions of the word "easy" :) > it???s extremely powerful, granted > it enables developers to perform the most tedious tasks > in a very short time and in general simply wipes the floor with > subversion. :) > Here I don't agree; they are targetting a different audience, hence they provide a different set of functionnalities... and have different problems. > It is possible to import subversion repositories into Git, keeping the > complete history and even branches, and it???s even possible to commit changes > made in a local repository back to subversion; I just tested that tonight > when I fixed nextgens??? backport of toad???s BucketChainBucket???it even made > finding the bug simple. > We all agree that that merging things from branches back and forth is not easy with svn 1.4... but it will be easier when we will upgrade to a merge-tracking enabled version. > Git repositories can be served by a special git-daemon or by a normal, > run-off-the-mill HTTP server without any special modules. No DAV, no CGI, no > nothing, just plain HTTP. > > I???d suggest that we move to Git rather sooner than later; What feature does it have we would *need* ? Since 0.7 started we had something like 6 branches... Even though I agree that the merging tool is under-optimized, I think we can live with it. First of all I am not convinced that we want to use a DSCM... and as for the choice of GIT, well, I don't think it's appropriate in our usecase for the following reasons: - Its user interface sucks (even the git guys acknowledge that and that's why they developped many front-ends to git like cogito) - Its documentation is inexistent - Its integration to IDEs sucks - Its not cross-platform, contrary to SVN - As far as I know it's not possible to integrate it with mantis, nor to auto-link svn revision numbers filled into tickets to commit identifiers git uses. - It's a DSCM ... DSCMs concepts are more complicated to apprehend than SCMs concepts for the average guy; We try to lower the fence the average contributor has to pass in order to be able to contribute; not to higher it. - DSCMs encourage forking; that's definitly not something we want to do. To be fair, on the assets side we have: - makes anonymous development easier (but we didn't have many anon. contributors... and I doubt it's related to the tools) - Working copies are smaller as they aren't per-branch. - It's probably faster than SVN when you work locally and don't push your changes. > most of the trouble > we had with bug #2440 (which resulted from the faulty backport) would never > have occured in the first place with Git. (Sure, of course you can make > changes in the wrong branch but Git will happily apply a commit from one > branch to another so the backport could have been done in two minutes.) > Well, I was aware it was broken but didn't take the time to fix it before I left... > Anybody in favor? Anybody opposed? > I'm opposed but I guess you already know it if you read that email up to here :) Florent From j16sdiz+freenet at gmail.com Tue Jul 1 06:45:04 2008 From: j16sdiz+freenet at gmail.com (Daniel Cheng) Date: Tue, 1 Jul 2008 14:45:04 +0800 Subject: [Tech] Switch Repository to Git In-Reply-To: <20080701040206.GB26659@freenetproject.org> References: <200806300031.03920.bombe@pterodactylus.net> <20080701040206.GB26659@freenetproject.org> Message-ID: On Tue, Jul 1, 2008 at 12:02 PM, NextGen$ wrote: > * David ???Bombe??? Roden [2008-06-30 00:31:03]: > >> Hey folks, >> >> I have been looking at and working with Git for the last four or five weeks >> and I must say that I???m very impressed by it. It???s easy to use, > > We must have different definitions of the word "easy" :) > >> it???s extremely powerful, > > granted > >> it enables developers to perform the most tedious tasks >> in a very short time and in general simply wipes the floor with >> subversion. :) >> > > Here I don't agree; they are targetting a different audience, hence they > provide a different set of functionnalities... and have different problems. > >> It is possible to import subversion repositories into Git, keeping the >> complete history and even branches, and it???s even possible to commit changes >> made in a local repository back to subversion; I just tested that tonight >> when I fixed nextgens??? backport of toad???s BucketChainBucket???it even made >> finding the bug simple. >> > > We all agree that that merging things from branches back and forth is not > easy with svn 1.4... but it will be easier when we will upgrade to a > merge-tracking enabled version. > >> Git repositories can be served by a special git-daemon or by a normal, >> run-off-the-mill HTTP server without any special modules. No DAV, no CGI, no >> nothing, just plain HTTP. >> >> I???d suggest that we move to Git rather sooner than later; > > What feature does it have we would *need* ? > Since 0.7 started we had something like 6 branches... Even though I agree > that the merging tool is under-optimized, I think we can live with it. > > First of all I am not convinced that we want to use a DSCM... and as for > the choice of GIT, well, I don't think it's appropriate in our usecase > for the following reasons: > - Its user interface sucks (even the git guys acknowledge that and > that's why they developped many front-ends to git like cogito) That's many years ago. The cogito project have ended, and some of the feature have been merged to the mainline. Lots of the frontend and documentation have been rewritten dueing the process. > - Its documentation is inexistent Same as above. > - Its integration to IDEs sucks > - Its not cross-platform, contrary to SVN It is cross-platform. It work well on mingw32 and cygwin. > - As far as I know it's not possible to integrate it with mantis, nor > to auto-link svn revision numbers filled into tickets to commit > identifiers git uses. This is a general problem for DSCM. > - It's a DSCM ... DSCMs concepts are more complicated to apprehend > than SCMs concepts for the average guy; We try to lower the fence > the average contributor has to pass in order to be able to > contribute; not to higher it. average contributor just send in small, self contained patch. > - DSCMs encourage forking; that's definitly not something we want to > do. for anonymous development, we *need* private branch and forking. > To be fair, on the assets side we have: > - makes anonymous development easier (but we didn't have many anon. > contributors... and I doubt it's related to the tools) If we want to encourage anonymous contribution, a DSCM is preferred.. This allow the contributor keep a private branch while waiting for review / merge. I know we don't want to accept anonymous contribution now (for legal reason), but we have to do this sooner or later due to the nature of freenet. (development of freenet *will* be outlawed in most country in a few years..) > - Working copies are smaller as they aren't per-branch. > - It's probably faster than SVN when you work locally and don't push > your changes. > >> most of the trouble >> we had with bug #2440 (which resulted from the faulty backport) would never >> have occured in the first place with Git. (Sure, of course you can make >> changes in the wrong branch but Git will happily apply a commit from one >> branch to another so the backport could have been done in two minutes.) >> > > Well, I was aware it was broken but didn't take the time to fix it before I > left... > >> Anybody in favor? Anybody opposed? >> > > I'm opposed but I guess you already know it if you read that email up to > here :) How about Mercurial? Will you support migrating to it? It have much better IDE support and cross platform support. (while keeping the other "disadvantages") Plus, there are three different freenet+hg integrations. We need a DSCM that work over freenet before it is outlawed in most country. (I prefer git) -- From bombe at pterodactylus.net Tue Jul 1 17:23:16 2008 From: bombe at pterodactylus.net (David =?utf-8?q?=E2=80=98Bombe=E2=80=99_Roden?=) Date: Tue, 1 Jul 2008 19:23:16 +0200 Subject: [Tech] Switch Repository to Git In-Reply-To: <20080701040206.GB26659@freenetproject.org> References: <200806300031.03920.bombe@pterodactylus.net> <20080701040206.GB26659@freenetproject.org> Message-ID: <200807011923.20243.bombe@pterodactylus.net> On Tuesday 01 July 2008 06:02:06 NextGen$ wrote: >> I have been looking at and working with Git for the last four or five >> weeks and I must say that I???m very impressed by it. It???s easy to use, > We must have different definitions of the word "easy" :) I don?t know about your definition of ?easy? but mine matches what it says in dictionaries. ;) Anything subversion can do Git can do as well and most of it is simpler. Granted, what takes a while to grasp is the concept of begin commit-based instead of revision-based. But once you got that you _have_ to marvel at its beauty. :) > Here I don't agree; they are targetting a different audience, hence they > provide a different set of functionnalities... and have different problems. I?m not so sure about that. Sure, when Git was conceived being distributed was a key requirement. But saying that DSCMs are only something for a small part of all developers it wrong, IMHO. SCM is for every developer?whether it?s distributed or not is a minor concern. > We all agree that that merging things from branches back and forth is not > easy with svn 1.4... but it will be easier when we will upgrade to a > merge-tracking enabled version. In subversion it will always be a cludge, though, because subversion wasn?t designed with repeated merges in mind. > What feature does it have we would *need* ? > Since 0.7 started we had something like 6 branches... Actually, the six branches we have are all there have been since 0.4.0 (at least that?s the first commit in the subversion repository says). :) But I think that having six branches is a symptom of using subversion because heavy branching and merging is basically impossible in subversion without a lot of manual work. We not using subversion because we don?t branch a lot but we don?t branch a lot because we are using subversion. I never used to branch much when I used subversion; now with Git I branch a whole lot more. It?s easy, it keeps your construction sites apart, and merging the branches back into some mainline branch is a breeze. With subversion I was always very hesitant with merging branches back because I had to keep track manually of when I merged what into where. > First of all I am not convinced that we want to use a DSCM... Why not? Why is a DSCM so fundamentally different? Why would it not be appropriate for freenet? (You can still have a central repository that all main developers keep pushing their work into and that acts as a source of reference for other. Plus, because of all that cryptographic stuff we wouldn?t have to host it ourselfs. If somebody messes with it, we?d notice.) > as for the choice of GIT, well, I don't think it's appropriate in our > usecase for the following reasons: > - Its user interface sucks (even the git guys acknowledge that and > that's why they developped many front-ends to git like cogito) > - Its documentation is inexistent As Daniel already pointed out, these points are no longer valid. A lot of work has been done and is still being done. > - As far as I know it's not possible to integrate it with mantis, nor > to auto-link svn revision numbers filled into tickets to commit > identifiers git uses. XMMS2?s mantis[1] is somehow tied into their Git repository. For example [2] shows that an issue was automatically resolved when the fix was committed. > - It's a DSCM ... DSCMs concepts are more complicated to apprehend > than SCMs concepts for the average guy; We try to lower the fence > the average contributor has to pass in order to be able to > contribute; not to higher it. The average contributor is as capable to follow instructions for a Git repository as he is capable to follow instructions for a subversion repository. You don?t to know much about the SCM in use to create a patch. You can even get by completely without using any SCM. > - DSCMs encourage forking; that's definitly not something we want to > do. That nobody forked freenet (0.7 at least) is not a matter of the SCM we use; it?s a matter of nobody understanding the ultra-convoluted codebase. :) > - It's probably faster than SVN when you work locally and don't push > your changes. Even when pushing over the network it?s a lot faster. The transfer itself is tiny and as the repository that does the integration is locally fast as well, it?s faster overall. I can check out the linux Git repository in about 8 minutes and that carries the _complete_ history since 2.6.13. A clone of freenet?s subversion repository with all commits takes several _hours_, and it?s a lot smaller. > Well, I was aware it was broken but didn't take the time to fix it before I > left... Yes, but that?s not the point. With Git, toad (or anybody else) could have git-cherry-pick?d the three (or so) commits and applied them to trunk in a matter of minutes. Okay, subversion can merge a revision range into another branch as well but it will seriously barf when you try to merge the two complete branches together later. Git would happily skip those commits as they already have been applied. Also, it?s not the point here that toad should have done the fix on trunk to begin with. ;) > I'm opposed but I guess you already know it if you read that email up to > here :) Yeah, well, I gathered that. I still hope that Git is not completely out of the window yet. It is a great system which I think would make our lifes at least a bit easier. > Florent David [1] http://bugs.xmms2.xmms.se/ [2] http://bugs.xmms2.xmms.se/view.php?id=1994 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080701/6544e401/attachment.pgp From alejandro at mosteo.com Wed Jul 2 08:02:32 2008 From: alejandro at mosteo.com (Jano) Date: Wed, 02 Jul 2008 10:02:32 +0200 Subject: [Tech] Switch Repository to Git References: <200806300031.03920.bombe@pterodactylus.net> <20080701040206.GB26659@freenetproject.org> <200807011923.20243.bombe@pterodactylus.net> Message-ID: David ?Bombe? Roden wrote: > On Tuesday 01 July 2008 06:02:06 NextGen$ wrote: > >> We all agree that that merging things from branches back and forth is not >> easy with svn 1.4... but it will be easier when we will upgrade to a >> merge-tracking enabled version. > > In subversion it will always be a cludge, though, because subversion wasn?t > designed with repeated merges in mind. Have you read the svn 1.5 release notes? You'd still call it a kludge? I'm honestly asking, I'd like to know from someone used to git. >> Florent > > David From nextgens at freenetproject.org Wed Jul 2 14:03:31 2008 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Wed, 2 Jul 2008 16:03:31 +0200 Subject: [Tech] Switch Repository to Git In-Reply-To: <200807011923.20243.bombe@pterodactylus.net> References: <200806300031.03920.bombe@pterodactylus.net> <20080701040206.GB26659@freenetproject.org> <200807011923.20243.bombe@pterodactylus.net> Message-ID: <20080702140329.GB3513@freenetproject.org> * David ???Bombe??? Roden [2008-07-01 19:23:16]: > On Tuesday 01 July 2008 06:02:06 NextGen$ wrote: > > >> I have been looking at and working with Git for the last four or five > >> weeks and I must say that I???m very impressed by it. It???s easy to use, > > We must have different definitions of the word "easy" :) > > I don???t know about your definition of ???easy??? but mine matches what it says in > dictionaries. ;) > > Anything subversion can do Git can do as well and most of it is simpler. > Granted, what takes a while to grasp is the concept of begin commit-based > instead of revision-based. But once you got that you _have_ to marvel at its > beauty. :) > > > > Here I don't agree; they are targetting a different audience, hence they > > provide a different set of functionnalities... and have different problems. > > I???m not so sure about that. Sure, when Git was conceived being distributed was > a key requirement. But saying that DSCMs are only something for a small part > of all developers it wrong, IMHO. SCM is for every developer???whether it???s > distributed or not is a minor concern. > No, it's not a minor concern; it affects the team-work and more generally speaking the way the application is developed. > > > We all agree that that merging things from branches back and forth is not > > easy with svn 1.4... but it will be easier when we will upgrade to a > > merge-tracking enabled version. > > In subversion it will always be a cludge, though, because subversion wasn???t > designed with repeated merges in mind. > Could you elaborate please ? Starting with svn 1.5 each commit has both a revision number and a "merge identifier" (I'm not sure about the terminology here). Git has only one of those. > > > What feature does it have we would *need* ? > > Since 0.7 started we had something like 6 branches... > > Actually, the six branches we have are all there have been since 0.4.0 (at > least that???s the first commit in the subversion repository says). :) > > But I think that having six branches is a symptom of using subversion because > heavy branching and merging is basically impossible in subversion without a > lot of manual work. We not using subversion because we don???t branch a lot but > we don???t branch a lot because we are using subversion. > Blaming the tools is easy. That's also what we did regarding "why the website isn't updated" and "why aren't people reviewing commits consistently"; In both occurrences experience has shown that changing the tools didn't help. > I never used to branch much when I used subversion; now with Git I branch a > whole lot more. It???s easy, it keeps your construction sites apart, and > merging the branches back into some mainline branch is a breeze. huh ? to me it's hard to make simpler than "svn cp" > With > subversion I was always very hesitant with merging branches back because I > had to keep track manually of when I merged what into where. > That's not a concern anymore with merge-tracking as implemented in svn 1.5 > > > First of all I am not convinced that we want to use a DSCM... > > Why not? Why is a DSCM so fundamentally different? Why would it not be > appropriate for freenet? (You can still have a central repository that all > main developers keep pushing their work into and that acts as a source of > reference for other. Plus, because of all that cryptographic stuff we > wouldn???t have to host it ourselfs. If somebody messes with it, we???d notice.) > > > > as for the choice of GIT, well, I don't think it's appropriate in our > > usecase for the following reasons: > > - Its user interface sucks (even the git guys acknowledge that and > > that's why they developped many front-ends to git like cogito) > > - Its documentation is inexistent > > As Daniel already pointed out, these points are no longer valid. A lot of work > has been done and is still being done. > I recon. that I did look into it a while ago and that it may have changed :) > > > - As far as I know it's not possible to integrate it with mantis, nor > > to auto-link svn revision numbers filled into tickets to commit > > identifiers git uses. > > XMMS2???s mantis[1] is somehow tied into their Git repository. For example [2] shows that an issue was automatically resolved when the fix was committed. > > > > - It's a DSCM ... DSCMs concepts are more complicated to apprehend > > than SCMs concepts for the average guy; We try to lower the fence > > the average contributor has to pass in order to be able to > > contribute; not to higher it. > > The average contributor is as capable to follow instructions for a Git > repository as he is capable to follow instructions for a subversion > repository. You don???t to know much about the SCM in use to create a patch. > You can even get by completely without using any SCM. > That's exactly what we want to prevent. If the tools are too complicated people won't use them and we will end up limiting the number of "commiters"... The asset of SCMs is traceability; we want to know to who belong each single line of code we host; That might be useful in case we want to re-license the code or something. > > > - DSCMs encourage forking; that's definitly not something we want to > > do. > > That nobody forked freenet (0.7 at least) is not a matter of the SCM we use; > it???s a matter of nobody understanding the ultra-convoluted codebase. :) > > > > - It's probably faster than SVN when you work locally and don't push > > your changes. > > Even when pushing over the network it???s a lot faster. The transfer itself is > tiny and as the repository that does the integration is locally fast as well, > it???s faster overall. I can check out the linux Git repository in about 8 > minutes and that carries the _complete_ history since 2.6.13. A clone of > freenet???s subversion repository with all commits takes several _hours_, and > it???s a lot smaller. > As far as I know git doesn't handle sparse checkouts. That alone is a good reason why not to use it :) I do not have a full checkout of the trunk here... and I see no point in having one; I don't buy the "it will take less space" argument. > > > Well, I was aware it was broken but didn't take the time to fix it before I > > left... > > Yes, but that???s not the point. With Git, toad (or anybody else) could have > git-cherry-pick???d the three (or so) commits and applied them to trunk in a > matter of minutes. Okay, subversion can merge a revision range into another > branch as well but it will seriously barf when you try to merge the two > complete branches together later. Git would happily skip those commits as > they already have been applied. Also, it???s not the point here that toad > should have done the fix on trunk to begin with. ;) > svn 1.5's merge tracking is solving that problem > > > I'm opposed but I guess you already know it if you read that email up to > > here :) > > Yeah, well, I gathered that. I still hope that Git is not completely out of > the window yet. It is a great system which I think would make our lifes at > least a bit easier. No one prevent you from using it; we might even consider publishing an officially converted, read-only tree... but for now, using it as our main SCM is out of question: Quote from IRC: " [13:33] <@ nextgens> | toad_> any views regarding a switch to a dscm ? [13:34] < toad_> | nextgens: broadly in favour but sceptical in the short run, we have limited time [13:34] <@ nextgens> | okay so we agree :) " Florent -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080702/d3510953/attachment.pgp From j16sdiz+freenet at gmail.com Wed Jul 2 15:21:45 2008 From: j16sdiz+freenet at gmail.com (Daniel Cheng) Date: Wed, 2 Jul 2008 23:21:45 +0800 Subject: [Tech] Switch Repository to Git In-Reply-To: References: <200806300031.03920.bombe@pterodactylus.net> <20080701040206.GB26659@freenetproject.org> <200807011923.20243.bombe@pterodactylus.net> Message-ID: On Wed, Jul 2, 2008 at 4:02 PM, Jano wrote: > David 'Bombe' Roden wrote: > >> On Tuesday 01 July 2008 06:02:06 NextGen$ wrote: >> >>> We all agree that that merging things from branches back and forth is not >>> easy with svn 1.4... but it will be easier when we will upgrade to a >>> merge-tracking enabled version. >> >> In subversion it will always be a cludge, though, because subversion wasn't >> designed with repeated merges in mind. > > Have you read the svn 1.5 release notes? You'd still call it a kludge? I'm > honestly asking, I'd like to know from someone used to git. While I love using git, I am more interesting in DSCM then any specific SCM. The ability of commit locally and work offline allow me to test out my idea more freely. You know, everybody have some crazy idea that he want to test without letting others to see. For instance, I have yet another experimental database and an alternative location swapping code locally. A (good) side effect is the ability to work offline. Easy merge management is just a side effect of DSCM -- DSCM encourage branching and forking. That's why every DSCM comes with a set of good merging tools. From nextgens at freenetproject.org Wed Jul 2 15:24:35 2008 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Wed, 2 Jul 2008 17:24:35 +0200 Subject: [Tech] Switch Repository to Git In-Reply-To: References: <200806300031.03920.bombe@pterodactylus.net> <20080701040206.GB26659@freenetproject.org> <200807011923.20243.bombe@pterodactylus.net> Message-ID: <20080702152433.GD3513@freenetproject.org> * Daniel Cheng [2008-07-02 23:21:45]: > On Wed, Jul 2, 2008 at 4:02 PM, Jano wrote: > > David 'Bombe' Roden wrote: > > > >> On Tuesday 01 July 2008 06:02:06 NextGen$ wrote: > >> > >>> We all agree that that merging things from branches back and forth is not > >>> easy with svn 1.4... but it will be easier when we will upgrade to a > >>> merge-tracking enabled version. > >> > >> In subversion it will always be a cludge, though, because subversion wasn't > >> designed with repeated merges in mind. > > > > Have you read the svn 1.5 release notes? You'd still call it a kludge? I'm > > honestly asking, I'd like to know from someone used to git. > > While I love using git, I am more interesting in DSCM then any > specific SCM. The ability of commit locally and work offline allow me > to test out my idea more freely. > You know, everybody have some crazy idea that he want to test without > letting others to see. > For instance, I have yet another experimental database and an > alternative location swapping code locally. > > A (good) side effect is the ability to work offline. > > Easy merge management is just a side effect of DSCM -- DSCM encourage > branching and forking. That's why every DSCM comes with a set of good > merging tools. That's my point: at this stage we don't want to encourage branching nor forking. We need the network to be homogeneous version and bug-wise to be debuggable and to move forward. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080702/1cb5f506/attachment.pgp From j16sdiz+freenet at gmail.com Wed Jul 2 15:34:29 2008 From: j16sdiz+freenet at gmail.com (Daniel Cheng) Date: Wed, 2 Jul 2008 23:34:29 +0800 Subject: [Tech] Switch Repository to Git In-Reply-To: <20080702152433.GD3513@freenetproject.org> References: <200806300031.03920.bombe@pterodactylus.net> <20080701040206.GB26659@freenetproject.org> <200807011923.20243.bombe@pterodactylus.net> <20080702152433.GD3513@freenetproject.org> Message-ID: 2008/7/2 Florent Daigni?re : > * Daniel Cheng [2008-07-02 23:21:45]: > >> On Wed, Jul 2, 2008 at 4:02 PM, Jano wrote: >> > David 'Bombe' Roden wrote: >> > >> >> On Tuesday 01 July 2008 06:02:06 NextGen$ wrote: >> >> >> >>> We all agree that that merging things from branches back and forth is not >> >>> easy with svn 1.4... but it will be easier when we will upgrade to a >> >>> merge-tracking enabled version. >> >> >> >> In subversion it will always be a cludge, though, because subversion wasn't >> >> designed with repeated merges in mind. >> > >> > Have you read the svn 1.5 release notes? You'd still call it a kludge? I'm >> > honestly asking, I'd like to know from someone used to git. >> >> While I love using git, I am more interesting in DSCM then any >> specific SCM. The ability of commit locally and work offline allow me >> to test out my idea more freely. >> You know, everybody have some crazy idea that he want to test without >> letting others to see. >> For instance, I have yet another experimental database and an >> alternative location swapping code locally. >> >> A (good) side effect is the ability to work offline. >> >> Easy merge management is just a side effect of DSCM -- DSCM encourage >> branching and forking. That's why every DSCM comes with a set of good >> merging tools. > > That's my point: at this stage we don't want to encourage branching nor > forking. The tools make this easier does not means we have to encourage it. > We need the network to be homogeneous version and bug-wise to be debuggable > and to move forward. Anyway, you can't stop me (or anybody else) from using git-svn locally (and I am using it). -- From alejandro at mosteo.com Wed Jul 2 16:08:37 2008 From: alejandro at mosteo.com (Jano) Date: Wed, 02 Jul 2008 18:08:37 +0200 Subject: [Tech] Switch Repository to Git References: <200806300031.03920.bombe@pterodactylus.net> <20080701040206.GB26659@freenetproject.org> <200807011923.20243.bombe@pterodactylus.net> Message-ID: Daniel Cheng wrote: > On Wed, Jul 2, 2008 at 4:02 PM, Jano > wrote: >> David 'Bombe' Roden wrote: >> >>> On Tuesday 01 July 2008 06:02:06 NextGen$ wrote: >>> >>>> We all agree that that merging things from branches back and forth is not >>>> easy with svn 1.4... but it will be easier when we will upgrade to a >>>> merge-tracking enabled version. >>> >>> In subversion it will always be a cludge, though, because subversion wasn't >>> designed with repeated merges in mind. >> >> Have you read the svn 1.5 release notes? You'd still call it a kludge? I'm >> honestly asking, I'd like to know from someone used to git. > > While I love using git, I am more interesting in DSCM then any > specific SCM. The ability of commit locally and work offline allow me > to test out my idea more freely. > You know, everybody have some crazy idea that he want to test without > letting others to see. > For instance, I have yet another experimental database and an > alternative location swapping code locally. > > A (good) side effect is the ability to work offline. > > Easy merge management is just a side effect of DSCM -- DSCM encourage > branching and forking. That's why every DSCM comes with a set of good > merging tools. Well -- (leaving aside my unanswered question ;) ) -- I'm asking because for my own data (note -- not only code) I use svn, and merge tracking will be welcome. But I wonder what you (or `Bombe') consider a kludge and what not. Certainly I can see advantages to DSCM, but I don't feel for now a pressing need to move from svn for my non-collaborative needs. From nextgens at freenetproject.org Wed Jul 2 16:11:17 2008 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Wed, 2 Jul 2008 18:11:17 +0200 Subject: [Tech] Switch Repository to Git In-Reply-To: References: <200806300031.03920.bombe@pterodactylus.net> <20080701040206.GB26659@freenetproject.org> <200807011923.20243.bombe@pterodactylus.net> <20080702152433.GD3513@freenetproject.org> Message-ID: <20080702161116.GE3513@freenetproject.org> * Daniel Cheng [2008-07-02 23:34:29]: > 2008/7/2 Florent Daigni?re : > > * Daniel Cheng [2008-07-02 23:21:45]: > > > >> On Wed, Jul 2, 2008 at 4:02 PM, Jano wrote: > >> > David 'Bombe' Roden wrote: > >> > > >> >> On Tuesday 01 July 2008 06:02:06 NextGen$ wrote: > >> >> > >> >>> We all agree that that merging things from branches back and forth is not > >> >>> easy with svn 1.4... but it will be easier when we will upgrade to a > >> >>> merge-tracking enabled version. > >> >> > >> >> In subversion it will always be a cludge, though, because subversion wasn't > >> >> designed with repeated merges in mind. > >> > > >> > Have you read the svn 1.5 release notes? You'd still call it a kludge? I'm > >> > honestly asking, I'd like to know from someone used to git. > >> > >> While I love using git, I am more interesting in DSCM then any > >> specific SCM. The ability of commit locally and work offline allow me > >> to test out my idea more freely. > >> You know, everybody have some crazy idea that he want to test without > >> letting others to see. > >> For instance, I have yet another experimental database and an > >> alternative location swapping code locally. > >> > >> A (good) side effect is the ability to work offline. > >> > >> Easy merge management is just a side effect of DSCM -- DSCM encourage > >> branching and forking. That's why every DSCM comes with a set of good > >> merging tools. > > > > That's my point: at this stage we don't want to encourage branching nor > > forking. > > The tools make this easier does not means we have to encourage it. > Sure but why should we make it easier if it's not to encourage it then ? :) > > We need the network to be homogeneous version and bug-wise to be debuggable > > and to move forward. > > Anyway, you can't stop me (or anybody else) from using git-svn locally > (and I am using it). > I have no problem with you (or anybody else) using the tools they want. I just don't want brazillion of incompatible/forked clients to spawn up in the wild and to connect to the main network. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080702/8e6f6d9d/attachment.pgp From thelema314 at gmail.com Thu Jul 3 00:46:34 2008 From: thelema314 at gmail.com (Edgar Friendly) Date: Wed, 02 Jul 2008 19:46:34 -0500 Subject: [Tech] Switch Repository to Git In-Reply-To: <20080702161116.GE3513@freenetproject.org> References: <200806300031.03920.bombe@pterodactylus.net> <20080701040206.GB26659@freenetproject.org> <200807011923.20243.bombe@pterodactylus.net> <20080702152433.GD3513@freenetproject.org> <20080702161116.GE3513@freenetproject.org> Message-ID: <486C216A.8050007@gmail.com> Florent Daigni?re wrote: > I have no problem with you (or anybody else) using the tools they want. > I just don't want brazillion of incompatible/forked clients to spawn up > in the wild and to connect to the main network. Reflecting on this idea, I think it'd be good (in the long run) if this did happen. Of course the network would get disrupted, but I thought that the idea of Freenet was to have a disruption-resistant network. And a brazillion forks means many new developers trying changes. If the project doesn't reject their changes, they would become a great force for building a better Fred. OTOH, without DSCM, forked clients will still exist, it'll just be tougher to keep them up to date and to merge any improvements back to the official client. $0.02 from an oldie, E. From nextgens at freenetproject.org Thu Jul 3 00:50:15 2008 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Thu, 3 Jul 2008 02:50:15 +0200 Subject: [Tech] Switch Repository to Git In-Reply-To: <486C216A.8050007@gmail.com> References: <200806300031.03920.bombe@pterodactylus.net> <20080701040206.GB26659@freenetproject.org> <200807011923.20243.bombe@pterodactylus.net> <20080702152433.GD3513@freenetproject.org> <20080702161116.GE3513@freenetproject.org> <486C216A.8050007@gmail.com> Message-ID: <20080703005014.GB3519@freenetproject.org> * Edgar Friendly [2008-07-02 19:46:34]: > Florent Daigni?re wrote: > > I have no problem with you (or anybody else) using the tools they want. > > I just don't want brazillion of incompatible/forked clients to spawn up > > in the wild and to connect to the main network. > > Reflecting on this idea, I think it'd be good (in the long run) if this > did happen. Of course the network would get disrupted, but I thought > that the idea of Freenet was to have a disruption-resistant network. > And a brazillion forks means many new developers trying changes. If the > project doesn't reject their changes, they would become a great force > for building a better Fred. > > OTOH, without DSCM, forked clients will still exist, it'll just be > tougher to keep them up to date and to merge any improvements back to > the official client. > > $0.02 from an oldie, > > E. Heh, I'm not against the idea here; I'm just being pragmatic: we don't have the resources to deal with forked clients. NextGen$ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080703/21a25ac9/attachment.pgp From ajft at ajft.org Fri Jul 4 01:44:32 2008 From: ajft at ajft.org (Adrian Tritschler) Date: Fri, 04 Jul 2008 11:44:32 +1000 Subject: [Tech] Switch Repository to Git In-Reply-To: <20080702161116.GE3513@freenetproject.org> References: <200806300031.03920.bombe@pterodactylus.net> <20080701040206.GB26659@freenetproject.org> <200807011923.20243.bombe@pterodactylus.net> <20080702152433.GD3513@freenetproject.org> <20080702161116.GE3513@freenetproject.org> Message-ID: <486D8080.1000505@ajft.org> Florent Daigni?re wrote: > * Daniel Cheng [2008-07-02 23:34:29]: > >> 2008/7/2 Florent Daigni?re : >>> * Daniel Cheng [2008-07-02 23:21:45]: >>> ...snip... discussion on SCM du jour > Sure but why should we make it easier if it's not to encourage it then ? > :) > >>> We need the network to be homogeneous version and bug-wise to be debuggable >>> and to move forward. >> Anyway, you can't stop me (or anybody else) from using git-svn locally >> (and I am using it). >> > > I have no problem with you (or anybody else) using the tools they want. > I just don't want brazillion of incompatible/forked clients to spawn up > in the wild and to connect to the main network. That's a little bit like a web server developer saying they only want a specific web browser out there connecting "to the main network". Or a XyzBSD protocol guy saying they only want XyzBSD TCP/IP implementations out there connecting to "the internet". You build a distributed network, you have to make it robust for anything and everything connecting to it. Adrain -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080704/6417ccd1/attachment.pgp From nextgens at freenetproject.org Fri Jul 4 04:08:20 2008 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Fri, 4 Jul 2008 06:08:20 +0200 Subject: [Tech] Switch Repository to Git In-Reply-To: <486D8080.1000505@ajft.org> References: <200806300031.03920.bombe@pterodactylus.net> <20080701040206.GB26659@freenetproject.org> <200807011923.20243.bombe@pterodactylus.net> <20080702152433.GD3513@freenetproject.org> <20080702161116.GE3513@freenetproject.org> <486D8080.1000505@ajft.org> Message-ID: <20080704040818.GA3521@freenetproject.org> * Adrian Tritschler [2008-07-04 11:44:32]: > Florent Daigni?re wrote: >> * Daniel Cheng [2008-07-02 23:34:29]: >> >>> 2008/7/2 Florent Daigni?re : >>>> * Daniel Cheng [2008-07-02 23:21:45]: >>>> > > ...snip... discussion on SCM du jour > That's *the* topic of this thread. >> Sure but why should we make it easier if it's not to encourage it then ? >> :) >> >>>> We need the network to be homogeneous version and bug-wise to be debuggable >>>> and to move forward. >>> Anyway, you can't stop me (or anybody else) from using git-svn locally >>> (and I am using it). >>> >> >> I have no problem with you (or anybody else) using the tools they want. >> I just don't want brazillion of incompatible/forked clients to spawn up >> in the wild and to connect to the main network. > > That's a little bit like a web server developer saying they only want a > specific web browser out there connecting "to the main network". Except that web-technologies are specified and not in constant evolution. Freenet's inter-node protocol is not specified yet and that's on purpose. > Or a XyzBSD protocol guy saying they only want XyzBSD TCP/IP implementations > out there connecting to "the internet". > What's your problem exactly? I can always say that I don't *want* other implementations; that won't prevent them from appearing... and I am aware of that. The standard corporate response to that kind of concerns is to make every new version of the official client incompatible with previous. The original debate was about "why should we switch to git": I have just explained why I don't receive the "makes branching/forking easier" as an important point for now. > You build a distributed network, you have to make it robust for anything > and everything connecting to it. > I'm fed up with people telling me what I have to do. If you can't understand that we have only a limited amount of resources and time, well... it's not even worth arguing. For your information, "robust for anything and everything" doesn't exist. NextGen$ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080704/31a549b4/attachment.pgp From thelema314 at gmail.com Fri Jul 4 17:46:01 2008 From: thelema314 at gmail.com (Edgar Friendly) Date: Fri, 04 Jul 2008 12:46:01 -0500 Subject: [Tech] Switch Repository to Git In-Reply-To: <20080704040818.GA3521@freenetproject.org> References: <200806300031.03920.bombe@pterodactylus.net> <20080701040206.GB26659@freenetproject.org> <200807011923.20243.bombe@pterodactylus.net> <20080702152433.GD3513@freenetproject.org> <20080702161116.GE3513@freenetproject.org> <486D8080.1000505@ajft.org> <20080704040818.GA3521@freenetproject.org> Message-ID: <486E61D9.7060304@gmail.com> Florent Daigni?re wrote: > The original debate was about "why should we switch to git": I have just > explained why I don't receive the "makes branching/forking easier" as an > important point for now. > And I'd like to chime in that easier branching/forking doesn't mean only more incompatible clients, it also means a different type of interaction with your SCM, where checking in happens nearly as often as saving your work. This may have positive effects on development because the more history is kept in the SCM, the easier it is to revert to a previous version on failure, or pull from somewhere else in the tree of code if you realize you didn't want to get rid of something. A single line of development results in less of this because people generally commit to the SCM only top-quality, fully working and tested code, so more of the work happens outside the SCM, losing the above benefits to some extent. E. From toad at amphibian.dyndns.org Fri Jul 4 22:00:11 2008 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Fri, 4 Jul 2008 23:00:11 +0100 Subject: [Tech] Switch Repository to Git In-Reply-To: <20080703005014.GB3519@freenetproject.org> References: <200806300031.03920.bombe@pterodactylus.net> <486C216A.8050007@gmail.com> <20080703005014.GB3519@freenetproject.org> Message-ID: <200807042300.11475.toad@amphibian.dyndns.org> On Thursday 03 July 2008 01:50, Florent Daigni?re wrote: > * Edgar Friendly [2008-07-02 19:46:34]: > > > Florent Daigni?re wrote: > > > I have no problem with you (or anybody else) using the tools they want. > > > I just don't want brazillion of incompatible/forked clients to spawn up > > > in the wild and to connect to the main network. > > > > Reflecting on this idea, I think it'd be good (in the long run) if this > > did happen. Of course the network would get disrupted, but I thought > > that the idea of Freenet was to have a disruption-resistant network. > > And a brazillion forks means many new developers trying changes. If the > > project doesn't reject their changes, they would become a great force > > for building a better Fred. > > > > OTOH, without DSCM, forked clients will still exist, it'll just be > > tougher to keep them up to date and to merge any improvements back to > > the official client. > > > > $0.02 from an oldie, > > > > E. > > Heh, I'm not against the idea here; I'm just being pragmatic: we don't > have the resources to deal with forked clients. It's an emergent system, 100 clients each with slightly different behaviour will kill it. > > NextGen$ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080704/a9d6880c/attachment.pgp From nextgens at freenetproject.org Sat Jul 5 01:16:52 2008 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Sat, 5 Jul 2008 03:16:52 +0200 Subject: [Tech] Switch Repository to Git In-Reply-To: <486E61D9.7060304@gmail.com> References: <20080701040206.GB26659@freenetproject.org> <200807011923.20243.bombe@pterodactylus.net> <20080702152433.GD3513@freenetproject.org> <20080702161116.GE3513@freenetproject.org> <486D8080.1000505@ajft.org> <20080704040818.GA3521@freenetproject.org> <486E61D9.7060304@gmail.com> Message-ID: <20080705011650.GA3525@freenetproject.org> * Edgar Friendly [2008-07-04 12:46:01]: > Florent Daigni?re wrote: > > The original debate was about "why should we switch to git": I have just > > explained why I don't receive the "makes branching/forking easier" as an > > important point for now. > > > And I'd like to chime in that easier branching/forking doesn't mean only > more incompatible clients, it also means a different type of interaction > with your SCM, where checking in happens nearly as often as saving your > work. This may have positive effects on development because the more > history is kept in the SCM, the easier it is to revert to a previous > version on failure, or pull from somewhere else in the tree of code if > you realize you didn't want to get rid of something. > Two points here: 1) if we ever use a DSCM it *will* be in a centralized manner: there will be a public tree devs will have to push to; I do not think that the full history of each dev's individual "working-copy" will be pushed there. 2) How the SCM is used depends on people more than the tools; see below > A single line of development results in less of this because people > generally commit to the SCM only top-quality, fully working and tested > code, Do you read the commits diffs? That's for sure, we only commit top-quality, fully working and tested code... and that's why we don't need a bug-tracker and we have released freenet 1.0 since years now ;) > so more of the work happens outside the SCM, losing the above > benefits to some extent. > As said previously it depends more on people than on the tools you use; Say we switch to git tomorrow... someone is willing to work on a new feature or a bugfix... he will create a branch locally... work on it... probably do some stupid mistakes (as we all do)... and when he has something working he will push that back to the official trunk. At that moment he will be given the choice in between "merging incrementally" and "merging fully". The choice is still up to him and I bet that people not committing often right now won't merge incrementally their work either. Experience has shown that long-standing coders are *not* numerous; if we ever have to look for the history of the merge, it will be present only on the dev's working copy who is likely to have disappeared since a while. NextGen$ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080705/becf1df1/attachment.pgp From thelema314 at gmail.com Mon Jul 7 11:52:44 2008 From: thelema314 at gmail.com (Edgar Friendly) Date: Mon, 07 Jul 2008 06:52:44 -0500 Subject: [Tech] Switch Repository to Git In-Reply-To: <20080705011650.GA3525@freenetproject.org> References: <20080701040206.GB26659@freenetproject.org> <200807011923.20243.bombe@pterodactylus.net> <20080702152433.GD3513@freenetproject.org> <20080702161116.GE3513@freenetproject.org> <486D8080.1000505@ajft.org> <20080704040818.GA3521@freenetproject.org> <486E61D9.7060304@gmail.com> <20080705011650.GA3525@freenetproject.org> Message-ID: <4872038C.5030303@gmail.com> Florent Daigni?re wrote: > Two points here: > 1) if we ever use a DSCM it *will* be in a centralized manner: > there will be a public tree devs will have to push to; I do not > think that the full history of each dev's individual > "working-copy" will be pushed there. I expect any branches that multiple devs work on will end up in the central server. > 2) How the SCM is used depends on people more than the tools; > see below The tools influence how people use them. > > As said previously it depends more on people than on the tools you use; > Say we switch to git tomorrow... someone is willing to work on a new > feature or a bugfix... he will create a branch locally... work on it... > probably do some stupid mistakes (as we all do)... and when he has > something working he will push that back to the official trunk. At that > moment he will be given the choice in between "merging incrementally" > and "merging fully". The choice is still up to him and I bet that people > not committing often right now won't merge incrementally their work > either. Experience has shown that long-standing coders are *not* > numerous; if we ever have to look for the history of the merge, it will > be present only on the dev's working copy who is likely to have > disappeared since a while. > > NextGen$ > With git, both can exist - the full branch with all changes (in a developer-specific branch name) and the "fully merged" code in the trunk. E. From nextgens at freenetproject.org Mon Jul 7 12:33:09 2008 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Mon, 7 Jul 2008 14:33:09 +0200 Subject: [Tech] Switch Repository to Git In-Reply-To: <4872038C.5030303@gmail.com> References: <20080702152433.GD3513@freenetproject.org> <20080702161116.GE3513@freenetproject.org> <486D8080.1000505@ajft.org> <20080704040818.GA3521@freenetproject.org> <486E61D9.7060304@gmail.com> <20080705011650.GA3525@freenetproject.org> <4872038C.5030303@gmail.com> Message-ID: <20080707123308.GC3534@freenetproject.org> * Edgar Friendly [2008-07-07 06:52:44]: > Florent Daigni?re wrote: > > Two points here: > > 1) if we ever use a DSCM it *will* be in a centralized manner: > > there will be a public tree devs will have to push to; I do not > > think that the full history of each dev's individual > > "working-copy" will be pushed there. > I expect any branches that multiple devs work on will end up in the > central server. ... Read the commits and you will realize that *all* our branches have been a one-man-at-a-time job so far. > > 2) How the SCM is used depends on people more than the tools; > > see below > The tools influence how people use them. > Sure they do but that's not my point here. > > > > As said previously it depends more on people than on the tools you use; > > Say we switch to git tomorrow... someone is willing to work on a new > > feature or a bugfix... he will create a branch locally... work on it... > > probably do some stupid mistakes (as we all do)... and when he has > > something working he will push that back to the official trunk. At that > > moment he will be given the choice in between "merging incrementally" > > and "merging fully". The choice is still up to him and I bet that people > > not committing often right now won't merge incrementally their work > > either. Experience has shown that long-standing coders are *not* > > numerous; if we ever have to look for the history of the merge, it will > > be present only on the dev's working copy who is likely to have > > disappeared since a while. > > > > NextGen$ > > > With git, both can exist - the full branch with all changes (in a > developer-specific branch name) and the "fully merged" code in the trunk. > I am not saying that they can't. I am saying that developper's behaviour won't change: they will "fully merge" even if we switch to git. NextGen$ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080707/41007693/attachment.pgp From toad at amphibian.dyndns.org Mon Jul 14 18:43:44 2008 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Mon, 14 Jul 2008 19:43:44 +0100 Subject: [Tech] Darknet, uptime, and true passive requests Message-ID: <200807141943.51701.toad@amphibian.dyndns.org> 1. Many people have proposed over the years that we have a "bulk" flag which can be set when the timing of a request is less important (e.g. for splitfile fetches), or a priority class for a request which is visible at the network layer. I have always opposed this mostly because it makes traffic profiling slightly easier and any sort of priority scheme would need careful regulation to prevent race-to-the-top. 2. Long-term, and in particularly nasty places, Freenet will have to be mostly darknet, because it is much easier to attack opennet nodes, or to block them in bulk. One of the biggest practical problems with a pure darknet is the 24/7 issue: more people have laptops than have real PCs nowadays, and this trend is likely to continue and accelerate, but even if people have a desktop PC, many users won't run it 24x7 for various reasons: power consumption, noise, security (with encrypted disk, do you want to leave it unattended?), etc etc. Fanless home server appliances might be able to run 24x7, but that means additional expenditure to buy them. 3. FMS, even more than Frost, makes heavy use of SSK polling, and this is likely to expand as the network grows and FMS becomes more newbie friendly. Also various innovative applications require fast propagation of data once inserted (although there are frequently security issues with this). And widely-wanted data which is hard to find can be effectively polled by much of the network, causing excessive load. 4. The solution to SSK polling etc is some form of passive requests. In 0.7, we have ultra-lightweight passive requests, which are a very limited and unreliable mechanism but nonetheless should help significantly. The basic principle of ULPRs is that once a request completes, each node on the network remembers who wants the data and who it has asked for it, for a short time, without making any effort to reroute if connections are lost; if the data is found it is propagated quickly to everyone who wants it. 5. True passive requests (0.9) would be a mechanism whereby a node could send out a request, which once it failed would be remembered permanently, subject to a (long) timeout and/or periodic renewal from the originator. It would be automatically rerouted if the network topology changes. Passive requests would introduce a number of new technical challenges such as load management for persistent requests, evaluating a peer's competence in performing them, and so on, but they could greatly reduce the cost of SSK polling, rerequesting common but absent data, and enable such things as medium bandwidth high latency publish/subscribe for for example audio streams. Passive requests would probably have to have a priority level setting. It's a big job, but a big prize... 6. Passive requests would go a long way to solving the uptime problem. Say you have a small darknet, say 5 nodes. Its nodes are only online during evenings local time. Its only connection to the outside world is through one node which is connected to two of the small darknet, which is only online on Thursdays. Right now, except on Thursdays, the network would be essentially a leaf network: our real-time routing assumes that the network is fully connected. Most data will be very difficult to obtain. Real-time routing requires real-time load balancing, which means that all the nodes would request whatever it is they want constantly, generating load to no good purpose, except on Thursdays when the requests would get through, but severely limited by load management, and by the fact that more than one of the small darknet may be asking for the same file. So on Thursdays, some progress would be made, but often not very much. Now, with true passive requests, things can be very different. From the user's point of view the semantics are essentially the same: they click a link, it gets a DNF (fairly quickly), and they click the button to queue it to the global queue; some time later, they get a notification that the content is available. But performance could be much higher. If a node requests a block while the network is "offline", the request will propagate to all 5 nodes, and then sit there waiting for something to happen. When we connect to the wider network, the request is immediately rerouted to the node that just connected (either because it's a better route, or because there are spare hops). It propagates, fails, and is stored as a passive request on the wider network, hopefully reaching somewhere near the optimal node for the key. When the link is lost, both sides remember the other, so when/if the data is found on the wider network, it is propagated back to the originator. Furthermore, the load management would be optimised for passive requests: when the small network connects, it can immediately send a large number of passive requests for different blocks of the same file or for different files. These are not real-time requests, because they have already failed and turned into passive requests; so they can be trickled out at whatever rate the recipient sees fit. Also, they are not subject to the anti-polling measures we have introduced: Polling a key in 0.7 means requesting it 3 times, sleeping for half an hour, and repeating ad infinitum. Further similar measures may need to be introduced at the node level to try to deal with increasing load caused by FMS, but because we reroute on getting a connection, we can immediately route the requests. When we reconnect, hopefully our peer will have found most of the data we requested and can transfer it at link speed (or whatever limit may be imposed for security reasons). The transfer might take longer than the intersection, but I expect the whole system will be significantly faster than it would be now. It's even better if you have more than two network fragments: on a large darknet you might have subnetworks coming online and going offline constantly, so that you never actually have a fully online network. Passive requests would happily search out every relevant nook and cranny of the network. Note that much of this is only feasible on darknet, because of the trust connection: on opennet, passive requests probably will have to last only as long as the connection is open, and bulk transfer of passive requests is certainly not feasible on opennet. With regards to security, it may be possible to determine whether an FMS poster (for example) is on the local network, if you know when his posts come in. This is of course feasible now on such a topology, but on the other hand if nobody uses it because it's unusable, there's no threat. Passive requests would probably make it a little easier. Some form of tunneling, preferably with long client controlled delays for inserts, might help to solve it, but we would have to have a way of determining that the network is too small to provide useful anonymity. 7. Even longer term, many ISPs and countries may deploy traffic flow analysis hardware to identify and block all (unlicensed?) peer to peer networking. The only way to beat traffic flow analysis is to not send data continually over the internet. The obvious ways to do this are: 1. Parasitic transports: Steal the video stream in a VoIP call to a friend. Note that if VoIP calls are rare this won't work well, and if they are artificially common to speed downloads up that will probably be detectable. 2. Fake timing: Make the transports look like e.g. a private gaming server, and fake the timing based on statistical models. This is classic stego. It's a race between you and your opponent for whoever has the better model. Given that your opponent may store traffic data indefinitely and not act immediately, this is very dangerous... 3. Wifi etc, non-internet constant data transfer. 4. Sneakernet and physical-rendezvous-based protocols (the latter working on the Freenet threat model assumptions, so still a form of darknet, rather than Haggle's free-for-all-networking system which is also interesting but IMHO dangerous in the long term, and certainly isn't Freenet). Passive requests are again the right tool to deal with this IMHO. They lend themselves to efficient stream subscriptions, and also enable long-term downloads without assuming a fully connected network at all times. Of course for the very high latency options (sneakernet), there are other challenges, such as how to assign locations without being able to swap continually. But for medium latency caused by sparse darknets not often having many nodes online simultaneously, and for transports which have the same effect despite nodes' host computers actually being switched on, it should work well. Comments? Am I spouting nonsense? :) Apologies for the length of this mail, it's a somewhat complex subject! Given the enormous implications, maybe we should postpone passive requests to post 1.0 ... but I'm worried that FMS, and polling in general, may force our hand. It's likely that there will be different parts of it implemented over different versions of Freenet in any case... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080714/1b37a0df/attachment.pgp From j16sdiz+freenet at gmail.com Tue Jul 15 08:08:23 2008 From: j16sdiz+freenet at gmail.com (Daniel Cheng) Date: Tue, 15 Jul 2008 16:08:23 +0800 Subject: [Tech] Darknet, uptime, and true passive requests In-Reply-To: <200807141943.51701.toad@amphibian.dyndns.org> References: <200807141943.51701.toad@amphibian.dyndns.org> Message-ID: 2008/7/15 Matthew Toseland : > 1. Many people have proposed over the years that we have a "bulk" flag which > can be set when the timing of a request is less important (e.g. for splitfile > fetches), or a priority class for a request which is visible at the network > layer. I have always opposed this mostly because it makes traffic profiling > slightly easier and any sort of priority scheme would need careful regulation > to prevent race-to-the-top. > > 2. Long-term, and in particularly nasty places, Freenet will have to be mostly > darknet, because it is much easier to attack opennet nodes, or to block them > in bulk. One of the biggest practical problems with a pure darknet is the > 24/7 issue: more people have laptops than have real PCs nowadays, and this > trend is likely to continue and accelerate, but even if people have a desktop > PC, many users won't run it 24x7 for various reasons: power consumption, > noise, security (with encrypted disk, do you want to leave it unattended?), > etc etc. Fanless home server appliances might be able to run 24x7, but that > means additional expenditure to buy them. > > 3. FMS, even more than Frost, makes heavy use of SSK polling, and this is > likely to expand as the network grows and FMS becomes more newbie friendly. > Also various innovative applications require fast propagation of data once > inserted (although there are frequently security issues with this). And > widely-wanted data which is hard to find can be effectively polled by much of > the network, causing excessive load. > > 4. The solution to SSK polling etc is some form of passive requests. In 0.7, > we have ultra-lightweight passive requests, which are a very limited and > unreliable mechanism but nonetheless should help significantly. The basic > principle of ULPRs is that once a request completes, each node on the network > remembers who wants the data and who it has asked for it, for a short time, > without making any effort to reroute if connections are lost; if the data is > found it is propagated quickly to everyone who wants it. > > 5. True passive requests (0.9) would be a mechanism whereby a node could send > out a request, which once it failed would be remembered permanently, subject > to a (long) timeout and/or periodic renewal from the originator. It would be > automatically rerouted if the network topology changes. Passive requests > would introduce a number of new technical challenges such as load management > for persistent requests, evaluating a peer's competence in performing them, > and so on, but they could greatly reduce the cost of SSK polling, > rerequesting common but absent data, and enable such things as medium > bandwidth high latency publish/subscribe for for example audio streams. > Passive requests would probably have to have a priority level setting. It's a > big job, but a big prize... > > 6. Passive requests would go a long way to solving the uptime problem. Say you > have a small darknet, say 5 nodes. Its nodes are only online during evenings > local time. Its only connection to the outside world is through one node > which is connected to two of the small darknet, which is only online on > Thursdays. Right now, except on Thursdays, the network would be essentially a > leaf network: our real-time routing assumes that the network is fully > connected. Most data will be very difficult to obtain. Real-time routing > requires real-time load balancing, which means that all the nodes would > request whatever it is they want constantly, generating load to no good > purpose, except on Thursdays when the requests would get through, but > severely limited by load management, and by the fact that more than one of > the small darknet may be asking for the same file. So on Thursdays, some > progress would be made, but often not very much. > > Now, with true passive requests, things can be very different. From the user's > point of view the semantics are essentially the same: they click a link, it > gets a DNF (fairly quickly), and they click the button to queue it to the > global queue; some time later, they get a notification that the content is > available. But performance could be much higher. If a node requests a block > while the network is "offline", the request will propagate to all 5 nodes, > and then sit there waiting for something to happen. When we connect to the > wider network, the request is immediately rerouted to the node that just > connected (either because it's a better route, or because there are spare > hops). It propagates, fails, and is stored as a passive request on the wider > network, hopefully reaching somewhere near the optimal node for the key. When > the link is lost, both sides remember the other, so when/if the data is found > on the wider network, it is propagated back to the originator. Furthermore, > the load management would be optimised for passive requests: when the small > network connects, it can immediately send a large number of passive requests > for different blocks of the same file or for different files. These are not > real-time requests, because they have already failed and turned into passive > requests; so they can be trickled out at whatever rate the recipient sees > fit. Also, they are not subject to the anti-polling measures we have > introduced: Polling a key in 0.7 means requesting it 3 times, sleeping for > half an hour, and repeating ad infinitum. Further similar measures may need > to be introduced at the node level to try to deal with increasing load caused > by FMS, but because we reroute on getting a connection, we can immediately > route the requests. When we reconnect, hopefully our peer will have found > most of the data we requested and can transfer it at link speed (or whatever > limit may be imposed for security reasons). The transfer might take longer > than the intersection, but I expect the whole system will be significantly > faster than it would be now. It's even better if you have more than two > network fragments: on a large darknet you might have subnetworks coming > online and going offline constantly, so that you never actually have a fully > online network. Passive requests would happily search out every relevant nook > and cranny of the network. > > Note that much of this is only feasible on darknet, because of the trust > connection: on opennet, passive requests probably will have to last only as > long as the connection is open, and bulk transfer of passive requests is > certainly not feasible on opennet. > > With regards to security, it may be possible to determine whether an FMS > poster (for example) is on the local network, if you know when his posts come > in. This is of course feasible now on such a topology, but on the other hand > if nobody uses it because it's unusable, there's no threat. Passive requests > would probably make it a little easier. Some form of tunneling, preferably > with long client controlled delays for inserts, might help to solve it, but > we would have to have a way of determining that the network is too small to > provide useful anonymity. Is DoS possible under this scheme? let's say.... (1) a attacker introduce lots of "imaginary" nodes. (just some imaginary location and announcements ...) (2) put lots of request from these imaginary locations wait for the passive request propagate to the whole network, (3) upload the actual content..... (4) all node will then try to send the content to those "imaginary" nodes, as these location never exist, all nodes will have to busy routing to some non-exist location.. Passive request make this attack much less resources intensive. Or did I missed anything here? I think DoS is a much easier attack then determining the poster using this. > 7. Even longer term, many ISPs and countries may deploy traffic flow analysis > hardware to identify and block all (unlicensed?) peer to peer networking. The > only way to beat traffic flow analysis is to not send data continually over > the internet. The obvious ways to do this are: > 1. Parasitic transports: Steal the video stream in a VoIP call to a friend. > Note that if VoIP calls are rare this won't work well, and if they are > artificially common to speed downloads up that will probably be detectable. > 2. Fake timing: Make the transports look like e.g. a private gaming server, > and fake the timing based on statistical models. This is classic stego. It's > a race between you and your opponent for whoever has the better model. Given > that your opponent may store traffic data indefinitely and not act > immediately, this is very dangerous... > 3. Wifi etc, non-internet constant data transfer. > 4. Sneakernet and physical-rendezvous-based protocols (the latter working on > the Freenet threat model assumptions, so still a form of darknet, rather than > Haggle's free-for-all-networking system which is also interesting but IMHO > dangerous in the long term, and certainly isn't Freenet). > > Passive requests are again the right tool to deal with this IMHO. They lend > themselves to efficient stream subscriptions, and also enable long-term > downloads without assuming a fully connected network at all times. Of course > for the very high latency options (sneakernet), there are other challenges, > such as how to assign locations without being able to swap continually. But > for medium latency caused by sparse darknets not often having many nodes > online simultaneously, and for transports which have the same effect despite > nodes' host computers actually being switched on, it should work well. > > Comments? Am I spouting nonsense? :) Apologies for the length of this mail, > it's a somewhat complex subject! > > Given the enormous implications, maybe we should postpone passive requests to > post 1.0 ... but I'm worried that FMS, and polling in general, may force our > hand. It's likely that there will be different parts of it implemented over > different versions of Freenet in any case... > From theodore.hong at cl.cam.ac.uk Tue Jul 15 10:12:39 2008 From: theodore.hong at cl.cam.ac.uk (Theodore Hong) Date: Tue, 15 Jul 2008 11:12:39 +0100 Subject: [Tech] Darknet, uptime, and true passive requests In-Reply-To: <200807141943.51701.toad@amphibian.dyndns.org> References: <200807141943.51701.toad@amphibian.dyndns.org> Message-ID: <179050340807150312q752ea9f1qb7efbb20bddbf7e4@mail.gmail.com> I think that there are actually two distinct though related issues going on here, which are downtime and polling. Downtime means that the data you want is in the network, but you can't get to it right now because one of the nodes that the request needs to go through isn't up right now (e.g. a gateway out of a darknet). Polling means that the data you want isn't in the network yet, but you'd like to get it sometime in the future when it is inserted. DOWNTIME The downtime problem is related to delay-tolerant networking - http://en.wikipedia.org/wiki/Delay_Tolerant_Networking http://www.dtnrg.org/wiki One way to think about it is to think about routing in time as well as space. Suppose you want to send a message along the route A-B-C, but A-B is only up on Monday and B-C is only up on Tuesday. You can complete the route by holding the message at B for a day - effectively 'routing' between B_on_Monday and B_on_Tuesday. The practical implication is that if your first-choice destination is down, sometimes it may be better to hold the request for a while and wait for it to come back up, rather than immediately rerouting to your second-choice destination. Or, you route to the second-choice destination, but if it fails, you hold onto it. When the first-choice destination comes back up, it triggers a retry - a sort of "active" passive request. (A terrible term - perhaps "persistent request" or "delay-tolerant request"?) POLLING Polling, on the other hand, is related to publish-subscribe: http://en.wikipedia.org/wiki/Publish/subscribe You set up a subscription that registers your interest in some key(s), together with a route showing where to find you later on. When the data is later inserted, it gets pushed to you. Here, the trigger event is data arriving, rather than a node coming up. There is no further forward routing, only sending data back along an already-existing route. If the route to the subscriber no longer exists, that's another problem, but in this case I think we should just drop it rather than try to find the subscriber. If s/he really wants it, s/he can just request it again. Thoughts? theo 2008/7/14 Matthew Toseland : > 1. Many people have proposed over the years that we have a "bulk" flag which > can be set when the timing of a request is less important (e.g. for splitfile > fetches), or a priority class for a request which is visible at the network > layer. I have always opposed this mostly because it makes traffic profiling > slightly easier and any sort of priority scheme would need careful regulation > to prevent race-to-the-top. > > 2. Long-term, and in particularly nasty places, Freenet will have to be mostly > darknet, because it is much easier to attack opennet nodes, or to block them > in bulk. One of the biggest practical problems with a pure darknet is the > 24/7 issue: more people have laptops than have real PCs nowadays, and this > trend is likely to continue and accelerate, but even if people have a desktop > PC, many users won't run it 24x7 for various reasons: power consumption, > noise, security (with encrypted disk, do you want to leave it unattended?), > etc etc. Fanless home server appliances might be able to run 24x7, but that > means additional expenditure to buy them. > > 3. FMS, even more than Frost, makes heavy use of SSK polling, and this is > likely to expand as the network grows and FMS becomes more newbie friendly. > Also various innovative applications require fast propagation of data once > inserted (although there are frequently security issues with this). And > widely-wanted data which is hard to find can be effectively polled by much of > the network, causing excessive load. > > 4. The solution to SSK polling etc is some form of passive requests. In 0.7, > we have ultra-lightweight passive requests, which are a very limited and > unreliable mechanism but nonetheless should help significantly. The basic > principle of ULPRs is that once a request completes, each node on the network > remembers who wants the data and who it has asked for it, for a short time, > without making any effort to reroute if connections are lost; if the data is > found it is propagated quickly to everyone who wants it. > > 5. True passive requests (0.9) would be a mechanism whereby a node could send > out a request, which once it failed would be remembered permanently, subject > to a (long) timeout and/or periodic renewal from the originator. It would be > automatically rerouted if the network topology changes. Passive requests > would introduce a number of new technical challenges such as load management > for persistent requests, evaluating a peer's competence in performing them, > and so on, but they could greatly reduce the cost of SSK polling, > rerequesting common but absent data, and enable such things as medium > bandwidth high latency publish/subscribe for for example audio streams. > Passive requests would probably have to have a priority level setting. It's a > big job, but a big prize... > > 6. Passive requests would go a long way to solving the uptime problem. Say you > have a small darknet, say 5 nodes. Its nodes are only online during evenings > local time. Its only connection to the outside world is through one node > which is connected to two of the small darknet, which is only online on > Thursdays. Right now, except on Thursdays, the network would be essentially a > leaf network: our real-time routing assumes that the network is fully > connected. Most data will be very difficult to obtain. Real-time routing > requires real-time load balancing, which means that all the nodes would > request whatever it is they want constantly, generating load to no good > purpose, except on Thursdays when the requests would get through, but > severely limited by load management, and by the fact that more than one of > the small darknet may be asking for the same file. So on Thursdays, some > progress would be made, but often not very much. > > Now, with true passive requests, things can be very different. From the user's > point of view the semantics are essentially the same: they click a link, it > gets a DNF (fairly quickly), and they click the button to queue it to the > global queue; some time later, they get a notification that the content is > available. But performance could be much higher. If a node requests a block > while the network is "offline", the request will propagate to all 5 nodes, > and then sit there waiting for something to happen. When we connect to the > wider network, the request is immediately rerouted to the node that just > connected (either because it's a better route, or because there are spare > hops). It propagates, fails, and is stored as a passive request on the wider > network, hopefully reaching somewhere near the optimal node for the key. When > the link is lost, both sides remember the other, so when/if the data is found > on the wider network, it is propagated back to the originator. Furthermore, > the load management would be optimised for passive requests: when the small > network connects, it can immediately send a large number of passive requests > for different blocks of the same file or for different files. These are not > real-time requests, because they have already failed and turned into passive > requests; so they can be trickled out at whatever rate the recipient sees > fit. Also, they are not subject to the anti-polling measures we have > introduced: Polling a key in 0.7 means requesting it 3 times, sleeping for > half an hour, and repeating ad infinitum. Further similar measures may need > to be introduced at the node level to try to deal with increasing load caused > by FMS, but because we reroute on getting a connection, we can immediately > route the requests. When we reconnect, hopefully our peer will have found > most of the data we requested and can transfer it at link speed (or whatever > limit may be imposed for security reasons). The transfer might take longer > than the intersection, but I expect the whole system will be significantly > faster than it would be now. It's even better if you have more than two > network fragments: on a large darknet you might have subnetworks coming > online and going offline constantly, so that you never actually have a fully > online network. Passive requests would happily search out every relevant nook > and cranny of the network. > > Note that much of this is only feasible on darknet, because of the trust > connection: on opennet, passive requests probably will have to last only as > long as the connection is open, and bulk transfer of passive requests is > certainly not feasible on opennet. > > With regards to security, it may be possible to determine whether an FMS > poster (for example) is on the local network, if you know when his posts come > in. This is of course feasible now on such a topology, but on the other hand > if nobody uses it because it's unusable, there's no threat. Passive requests > would probably make it a little easier. Some form of tunneling, preferably > with long client controlled delays for inserts, might help to solve it, but > we would have to have a way of determining that the network is too small to > provide useful anonymity. > > 7. Even longer term, many ISPs and countries may deploy traffic flow analysis > hardware to identify and block all (unlicensed?) peer to peer networking. The > only way to beat traffic flow analysis is to not send data continually over > the internet. The obvious ways to do this are: > 1. Parasitic transports: Steal the video stream in a VoIP call to a friend. > Note that if VoIP calls are rare this won't work well, and if they are > artificially common to speed downloads up that will probably be detectable. > 2. Fake timing: Make the transports look like e.g. a private gaming server, > and fake the timing based on statistical models. This is classic stego. It's > a race between you and your opponent for whoever has the better model. Given > that your opponent may store traffic data indefinitely and not act > immediately, this is very dangerous... > 3. Wifi etc, non-internet constant data transfer. > 4. Sneakernet and physical-rendezvous-based protocols (the latter working on > the Freenet threat model assumptions, so still a form of darknet, rather than > Haggle's free-for-all-networking system which is also interesting but IMHO > dangerous in the long term, and certainly isn't Freenet). > > Passive requests are again the right tool to deal with this IMHO. They lend > themselves to efficient stream subscriptions, and also enable long-term > downloads without assuming a fully connected network at all times. Of course > for the very high latency options (sneakernet), there are other challenges, > such as how to assign locations without being able to swap continually. But > for medium latency caused by sparse darknets not often having many nodes > online simultaneously, and for transports which have the same effect despite > nodes' host computers actually being switched on, it should work well. > > Comments? Am I spouting nonsense? :) Apologies for the length of this mail, > it's a somewhat complex subject! > > Given the enormous implications, maybe we should postpone passive requests to > post 1.0 ... but I'm worried that FMS, and polling in general, may force our > hand. It's likely that there will be different parts of it implemented over > different versions of Freenet in any case... > > _______________________________________________ > Tech mailing list > Tech at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > From theodore.hong at cl.cam.ac.uk Tue Jul 15 10:17:02 2008 From: theodore.hong at cl.cam.ac.uk (Theodore Hong) Date: Tue, 15 Jul 2008 11:17:02 +0100 Subject: [Tech] Darknet, uptime, and true passive requests In-Reply-To: References: <200807141943.51701.toad@amphibian.dyndns.org> Message-ID: <179050340807150317o137cbdf0lad2c69a83964f0ff@mail.gmail.com> Daniel Cheng wrote: > Is DoS possible under this scheme? > > let's say.... > (1) a attacker introduce lots of "imaginary" nodes. > (just some imaginary location and announcements ...) > (2) put lots of request from these imaginary locations > wait for the passive request propagate to the whole network, > (3) upload the actual content..... > (4) all node will then try to send the content to those "imaginary" nodes, > as these location never exist, all nodes will have to busy routing to some > non-exist location.. The answer I think is that (2) the subscription should not propagate to the whole network, just to the nodes responsible for those keys. Then (4) the nodes holding those keys (not 'all nodes') will send to the imaginary nodes. That's ok, it's no worse than if you were to request a key and then immediately disconnect. theo From toad at amphibian.dyndns.org Tue Jul 15 13:04:07 2008 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Tue, 15 Jul 2008 14:04:07 +0100 Subject: [Tech] Darknet, uptime, and true passive requests In-Reply-To: <179050340807150312q752ea9f1qb7efbb20bddbf7e4@mail.gmail.com> References: <200807141943.51701.toad@amphibian.dyndns.org> <179050340807150312q752ea9f1qb7efbb20bddbf7e4@mail.gmail.com> Message-ID: <200807151404.13389.toad@amphibian.dyndns.org> On Tuesday 15 July 2008 11:12, Theodore Hong wrote: > I think that there are actually two distinct though related issues > going on here, which are downtime and polling. > > Downtime means that the data you want is in the network, but you can't > get to it right now because one of the nodes that the request needs to > go through isn't up right now (e.g. a gateway out of a darknet). > > Polling means that the data you want isn't in the network yet, but > you'd like to get it sometime in the future when it is inserted. > > > DOWNTIME > > The downtime problem is related to delay-tolerant networking - > http://en.wikipedia.org/wiki/Delay_Tolerant_Networking > http://www.dtnrg.org/wiki > > One way to think about it is to think about routing in time as well as > space. Suppose you want to send a message along the route A-B-C, but > A-B is only up on Monday and B-C is only up on Tuesday. You can > complete the route by holding the message at B for a day - effectively > 'routing' between B_on_Monday and B_on_Tuesday. > > The practical implication is that if your first-choice destination is > down, sometimes it may be better to hold the request for a while and > wait for it to come back up, rather than immediately rerouting to your > second-choice destination. Or, you route to the second-choice > destination, but if it fails, you hold onto it. When the first-choice > destination comes back up, it triggers a retry - a sort of "active" > passive request. (A terrible term - perhaps "persistent request" or > "delay-tolerant request"?) > > > POLLING > > Polling, on the other hand, is related to publish-subscribe: > http://en.wikipedia.org/wiki/Publish/subscribe > > You set up a subscription that registers your interest in some key(s), > together with a route showing where to find you later on. When the > data is later inserted, it gets pushed to you. > > Here, the trigger event is data arriving, rather than a node coming > up. There is no further forward routing, only sending data back along > an already-existing route. If the route to the subscriber no longer > exists, that's another problem, but in this case I think we should > just drop it rather than try to find the subscriber. If s/he really > wants it, s/he can just request it again. > > Thoughts? > theo Okay, there are two problems, but there may be some overlap in the solutions. First I will explain why I think we need persistent requests for polling, I would very much appreciate some input on this. CURRENT SITUATION: We already have a form of unreliable polling. Ultra-lightweight persistent requests last for 1 hour, are dropped if the connection is lost, are established automatically after any request, and propagate the data back to the originator and any other subscribers when it is found. This was implemented shortly before 0.7. The main motivation was increasing worry about the load that FMS is likely to put on the network. FMS is a usenet-style chat system which uses outbox polling to prevent spam. An earlier system, Frost, uses a global outbox and has been thoroughly DoS'ed. An important side-issue is that of many nodes constantly polling for big files that have fallen out of the network. At the same time as implementing ULPRs, we (I) also implemented per-node failure tables, so that every request for the same key (within some period) will be routed slightly differently (within some period). Before this, we were constantly sending requests for data we can't find; now we send 3 requests every half hour, on the hope that 1) the 3 requests will go to slightly different paths, and 2) if the data is found during the half hour pause, it *may* be propagated back to us via ULPRs. Both big file downloads and FMS's polling use the same mechanism. Other polling apps are under development: batosai's web of trust plugin works on the same principles as FMS but is incompatible and may see wider use for filesharing apps and so on. And there is a real-time chat client which seems to work (for small numbers so far), and which presumably polls one or more queues constantly. The second phase for ULPRs was going to be, and still may be, to limit the number of requests going through any single node for any given key, on the basis that the data will be propagated back to the requestors via ULPRs when it is found. This should have a significant impact on network load/efficiency. PASSIVE REQUESTS: Polling is inefficient and causes a lot of network load, especially if a large proportion of the network are polling for an overlapping large number of keys (as with FMS). Also, many applications will not be happy when a ULPR fails because a connection is lost, and they aren't told about it (because ULPRs don't do that), and their data is delayed by half an hour. This is almost tolerable for FMS but even there it's annoying. For audio streaming and certainly for real time chat it's not tolerable (admittedly real time chat probably isn't something we want to encourage, since it will likely be very inefficient in any case as even SSKs are 1KB payload). Full passive requests would enable a node to subscribe to a key, and not have to worry about polling it. The network would be responsible for polling, and it could do it far more efficiently than the client could, at least for keys being requested by many nodes. Passive requests would not arbitrarily fail and not tell the originator: they would be automatically rerouted when a connection is lost or a better route is found. This could be reasonably efficient bandwidth-wise, since we have a large number of passive requests effectively queued, we can send a whole bunch of them in a single packet when something changes. Passive requests would include some form of coalescing, of course, forming a big tree if a key is popular. When the data is inserted, it can be propagated to all subscribers very fast. All this would of course require different load management: since a passive request has ongoing overhead, each node's subscriptions would have to be limited, probably in relation to its connectedness to the rest of the network to prevent an attacker from doing DoS attacks. There would probably also be network level priorities, which again would present some load management issues. Also, if the capacity is available, we would be able to have a lot more passive requests than we can currently have ULPRs, possibly persisting them on disk. The end user impact of all of this is that apps such as FMS should both have much lower latency and much less impact on the network, even as it scales up, and applications such as audio streaming will be feasible without having to have a 30-60 minutes delay. Some open questions: 1. How many passive requests can we afford? Will churn make them very costly for non-popular keys? Maybe... 2. Can we mitigate this, perhaps by not always rerouting immediately? What would the performance cost be? It could be significant... maybe make it related to the priority/the number of subscribers/etc? 3. What is the impact on the other main source of network load/inefficiency: Clients polling big files that have mostly fallen out of the network ? ULPRs throttle this to some degree, although for big files ULPRs won't be much help as a single block will be rerequested less than once every hour; per-node failure tables mean that every retry will go somewhere different, improving the chances of finding data that isn't where it should be. Clients may want to subscribe to the keys, but at a lower priority than their FMS traffic, and explicitly poll to try to do an exhaustive search as well - it's a question of how much we limit both of these strategies. If we could assume that routing works and that data is always where it should be, and that passive requests are low overhad, we'd simply subscribe to the key (at least for popular keys), and block all further requests for it on the grounds that when we find it we'll return it to all requestors. But this may not be a feasible assumption, unless we automatically migrate data when locations are swapped, which might cause a lot of load... > > > 2008/7/14 Matthew Toseland : > > 1. Many people have proposed over the years that we have a "bulk" flag which > > can be set when the timing of a request is less important (e.g. for splitfile > > fetches), or a priority class for a request which is visible at the network > > layer. I have always opposed this mostly because it makes traffic profiling > > slightly easier and any sort of priority scheme would need careful regulation > > to prevent race-to-the-top. > > > > 2. Long-term, and in particularly nasty places, Freenet will have to be mostly > > darknet, because it is much easier to attack opennet nodes, or to block them > > in bulk. One of the biggest practical problems with a pure darknet is the > > 24/7 issue: more people have laptops than have real PCs nowadays, and this > > trend is likely to continue and accelerate, but even if people have a desktop > > PC, many users won't run it 24x7 for various reasons: power consumption, > > noise, security (with encrypted disk, do you want to leave it unattended?), > > etc etc. Fanless home server appliances might be able to run 24x7, but that > > means additional expenditure to buy them. > > > > 3. FMS, even more than Frost, makes heavy use of SSK polling, and this is > > likely to expand as the network grows and FMS becomes more newbie friendly. > > Also various innovative applications require fast propagation of data once > > inserted (although there are frequently security issues with this). And > > widely-wanted data which is hard to find can be effectively polled by much of > > the network, causing excessive load. > > > > 4. The solution to SSK polling etc is some form of passive requests. In 0.7, > > we have ultra-lightweight passive requests, which are a very limited and > > unreliable mechanism but nonetheless should help significantly. The basic > > principle of ULPRs is that once a request completes, each node on the network > > remembers who wants the data and who it has asked for it, for a short time, > > without making any effort to reroute if connections are lost; if the data is > > found it is propagated quickly to everyone who wants it. > > > > 5. True passive requests (0.9) would be a mechanism whereby a node could send > > out a request, which once it failed would be remembered permanently, subject > > to a (long) timeout and/or periodic renewal from the originator. It would be > > automatically rerouted if the network topology changes. Passive requests > > would introduce a number of new technical challenges such as load management > > for persistent requests, evaluating a peer's competence in performing them, > > and so on, but they could greatly reduce the cost of SSK polling, > > rerequesting common but absent data, and enable such things as medium > > bandwidth high latency publish/subscribe for for example audio streams. > > Passive requests would probably have to have a priority level setting. It's a > > big job, but a big prize... > > > > 6. Passive requests would go a long way to solving the uptime problem. Say you > > have a small darknet, say 5 nodes. Its nodes are only online during evenings > > local time. Its only connection to the outside world is through one node > > which is connected to two of the small darknet, which is only online on > > Thursdays. Right now, except on Thursdays, the network would be essentially a > > leaf network: our real-time routing assumes that the network is fully > > connected. Most data will be very difficult to obtain. Real-time routing > > requires real-time load balancing, which means that all the nodes would > > request whatever it is they want constantly, generating load to no good > > purpose, except on Thursdays when the requests would get through, but > > severely limited by load management, and by the fact that more than one of > > the small darknet may be asking for the same file. So on Thursdays, some > > progress would be made, but often not very much. > > > > Now, with true passive requests, things can be very different. From the user's > > point of view the semantics are essentially the same: they click a link, it > > gets a DNF (fairly quickly), and they click the button to queue it to the > > global queue; some time later, they get a notification that the content is > > available. But performance could be much higher. If a node requests a block > > while the network is "offline", the request will propagate to all 5 nodes, > > and then sit there waiting for something to happen. When we connect to the > > wider network, the request is immediately rerouted to the node that just > > connected (either because it's a better route, or because there are spare > > hops). It propagates, fails, and is stored as a passive request on the wider > > network, hopefully reaching somewhere near the optimal node for the key. When > > the link is lost, both sides remember the other, so when/if the data is found > > on the wider network, it is propagated back to the originator. Furthermore, > > the load management would be optimised for passive requests: when the small > > network connects, it can immediately send a large number of passive requests > > for different blocks of the same file or for different files. These are not > > real-time requests, because they have already failed and turned into passive > > requests; so they can be trickled out at whatever rate the recipient sees > > fit. Also, they are not subject to the anti-polling measures we have > > introduced: Polling a key in 0.7 means requesting it 3 times, sleeping for > > half an hour, and repeating ad infinitum. Further similar measures may need > > to be introduced at the node level to try to deal with increasing load caused > > by FMS, but because we reroute on getting a connection, we can immediately > > route the requests. When we reconnect, hopefully our peer will have found > > most of the data we requested and can transfer it at link speed (or whatever > > limit may be imposed for security reasons). The transfer might take longer > > than the intersection, but I expect the whole system will be significantly > > faster than it would be now. It's even better if you have more than two > > network fragments: on a large darknet you might have subnetworks coming > > online and going offline constantly, so that you never actually have a fully > > online network. Passive requests would happily search out every relevant nook > > and cranny of the network. > > > > Note that much of this is only feasible on darknet, because of the trust > > connection: on opennet, passive requests probably will have to last only as > > long as the connection is open, and bulk transfer of passive requests is > > certainly not feasible on opennet. > > > > With regards to security, it may be possible to determine whether an FMS > > poster (for example) is on the local network, if you know when his posts come > > in. This is of course feasible now on such a topology, but on the other hand > > if nobody uses it because it's unusable, there's no threat. Passive requests > > would probably make it a little easier. Some form of tunneling, preferably > > with long client controlled delays for inserts, might help to solve it, but > > we would have to have a way of determining that the network is too small to > > provide useful anonymity. > > > > 7. Even longer term, many ISPs and countries may deploy traffic flow analysis > > hardware to identify and block all (unlicensed?) peer to peer networking. The > > only way to beat traffic flow analysis is to not send data continually over > > the internet. The obvious ways to do this are: > > 1. Parasitic transports: Steal the video stream in a VoIP call to a friend. > > Note that if VoIP calls are rare this won't work well, and if they are > > artificially common to speed downloads up that will probably be detectable. > > 2. Fake timing: Make the transports look like e.g. a private gaming server, > > and fake the timing based on statistical models. This is classic stego. It's > > a race between you and your opponent for whoever has the better model. Given > > that your opponent may store traffic data indefinitely and not act > > immediately, this is very dangerous... > > 3. Wifi etc, non-internet constant data transfer. > > 4. Sneakernet and physical-rendezvous-based protocols (the latter working on > > the Freenet threat model assumptions, so still a form of darknet, rather than > > Haggle's free-for-all-networking system which is also interesting but IMHO > > dangerous in the long term, and certainly isn't Freenet). > > > > Passive requests are again the right tool to deal with this IMHO. They lend > > themselves to efficient stream subscriptions, and also enable long-term > > downloads without assuming a fully connected network at all times. Of course > > for the very high latency options (sneakernet), there are other challenges, > > such as how to assign locations without being able to swap continually. But > > for medium latency caused by sparse darknets not often having many nodes > > online simultaneously, and for transports which have the same effect despite > > nodes' host computers actually being switched on, it should work well. > > > > Comments? Am I spouting nonsense? :) Apologies for the length of this mail, > > it's a somewhat complex subject! > > > > Given the enormous implications, maybe we should postpone passive requests to > > post 1.0 ... but I'm worried that FMS, and polling in general, may force our > > hand. It's likely that there will be different parts of it implemented over > > different versions of Freenet in any case... > > > > _______________________________________________ > > Tech mailing list > > Tech at freenetproject.org > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > > > _______________________________________________ > Tech mailing list > Tech at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080715/9ee832f5/attachment.pgp From toad at amphibian.dyndns.org Tue Jul 15 13:15:03 2008 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Tue, 15 Jul 2008 14:15:03 +0100 Subject: [Tech] Darknet, uptime, and true passive requests In-Reply-To: References: <200807141943.51701.toad@amphibian.dyndns.org> Message-ID: <200807151415.09001.toad@amphibian.dyndns.org> On Tuesday 15 July 2008 09:08, Daniel Cheng wrote: > 2008/7/15 Matthew Toseland : > > 1. Many people have proposed over the years that we have a "bulk" flag which > > can be set when the timing of a request is less important (e.g. for splitfile > > fetches), or a priority class for a request which is visible at the network > > layer. I have always opposed this mostly because it makes traffic profiling > > slightly easier and any sort of priority scheme would need careful regulation > > to prevent race-to-the-top. > > > > 2. Long-term, and in particularly nasty places, Freenet will have to be mostly > > darknet, because it is much easier to attack opennet nodes, or to block them > > in bulk. One of the biggest practical problems with a pure darknet is the > > 24/7 issue: more people have laptops than have real PCs nowadays, and this > > trend is likely to continue and accelerate, but even if people have a desktop > > PC, many users won't run it 24x7 for various reasons: power consumption, > > noise, security (with encrypted disk, do you want to leave it unattended?), > > etc etc. Fanless home server appliances might be able to run 24x7, but that > > means additional expenditure to buy them. > > > > 3. FMS, even more than Frost, makes heavy use of SSK polling, and this is > > likely to expand as the network grows and FMS becomes more newbie friendly. > > Also various innovative applications require fast propagation of data once > > inserted (although there are frequently security issues with this). And > > widely-wanted data which is hard to find can be effectively polled by much of > > the network, causing excessive load. > > > > 4. The solution to SSK polling etc is some form of passive requests. In 0.7, > > we have ultra-lightweight passive requests, which are a very limited and > > unreliable mechanism but nonetheless should help significantly. The basic > > principle of ULPRs is that once a request completes, each node on the network > > remembers who wants the data and who it has asked for it, for a short time, > > without making any effort to reroute if connections are lost; if the data is > > found it is propagated quickly to everyone who wants it. > > > > 5. True passive requests (0.9) would be a mechanism whereby a node could send > > out a request, which once it failed would be remembered permanently, subject > > to a (long) timeout and/or periodic renewal from the originator. It would be > > automatically rerouted if the network topology changes. Passive requests > > would introduce a number of new technical challenges such as load management > > for persistent requests, evaluating a peer's competence in performing them, > > and so on, but they could greatly reduce the cost of SSK polling, > > rerequesting common but absent data, and enable such things as medium > > bandwidth high latency publish/subscribe for for example audio streams. > > Passive requests would probably have to have a priority level setting. It's a > > big job, but a big prize... > > > > 6. Passive requests would go a long way to solving the uptime problem. Say you > > have a small darknet, say 5 nodes. Its nodes are only online during evenings > > local time. Its only connection to the outside world is through one node > > which is connected to two of the small darknet, which is only online on > > Thursdays. Right now, except on Thursdays, the network would be essentially a > > leaf network: our real-time routing assumes that the network is fully > > connected. Most data will be very difficult to obtain. Real-time routing > > requires real-time load balancing, which means that all the nodes would > > request whatever it is they want constantly, generating load to no good > > purpose, except on Thursdays when the requests would get through, but > > severely limited by load management, and by the fact that more than one of > > the small darknet may be asking for the same file. So on Thursdays, some > > progress would be made, but often not very much. > > > > Now, with true passive requests, things can be very different. From the user's > > point of view the semantics are essentially the same: they click a link, it > > gets a DNF (fairly quickly), and they click the button to queue it to the > > global queue; some time later, they get a notification that the content is > > available. But performance could be much higher. If a node requests a block > > while the network is "offline", the request will propagate to all 5 nodes, > > and then sit there waiting for something to happen. When we connect to the > > wider network, the request is immediately rerouted to the node that just > > connected (either because it's a better route, or because there are spare > > hops). It propagates, fails, and is stored as a passive request on the wider > > network, hopefully reaching somewhere near the optimal node for the key. When > > the link is lost, both sides remember the other, so when/if the data is found > > on the wider network, it is propagated back to the originator. Furthermore, > > the load management would be optimised for passive requests: when the small > > network connects, it can immediately send a large number of passive requests > > for different blocks of the same file or for different files. These are not > > real-time requests, because they have already failed and turned into passive > > requests; so they can be trickled out at whatever rate the recipient sees > > fit. Also, they are not subject to the anti-polling measures we have > > introduced: Polling a key in 0.7 means requesting it 3 times, sleeping for > > half an hour, and repeating ad infinitum. Further similar measures may need > > to be introduced at the node level to try to deal with increasing load caused > > by FMS, but because we reroute on getting a connection, we can immediately > > route the requests. When we reconnect, hopefully our peer will have found > > most of the data we requested and can transfer it at link speed (or whatever > > limit may be imposed for security reasons). The transfer might take longer > > than the intersection, but I expect the whole system will be significantly > > faster than it would be now. It's even better if you have more than two > > network fragments: on a large darknet you might have subnetworks coming > > online and going offline constantly, so that you never actually have a fully > > online network. Passive requests would happily search out every relevant nook > > and cranny of the network. > > > > Note that much of this is only feasible on darknet, because of the trust > > connection: on opennet, passive requests probably will have to last only as > > long as the connection is open, and bulk transfer of passive requests is > > certainly not feasible on opennet. > > > > With regards to security, it may be possible to determine whether an FMS > > poster (for example) is on the local network, if you know when his posts come > > in. This is of course feasible now on such a topology, but on the other hand > > if nobody uses it because it's unusable, there's no threat. Passive requests > > would probably make it a little easier. Some form of tunneling, preferably > > with long client controlled delays for inserts, might help to solve it, but > > we would have to have a way of determining that the network is too small to > > provide useful anonymity. > > Is DoS possible under this scheme? > > let's say.... > (1) a attacker introduce lots of "imaginary" nodes. > (just some imaginary location and announcements ...) > (2) put lots of request from these imaginary locations > wait for the passive request propagate to the whole network, > (3) upload the actual content..... > (4) all node will then try to send the content to those "imaginary" nodes, > as these location never exist, all nodes will have to busy routing to some > non-exist location.. > > Passive request make this attack much less resources intensive. > Or did I missed anything here? > > I think DoS is a much easier attack then determining the poster using this. > Passive request data goes back along the current path; it doesn't get routed to a location. However, you do have a point in that if we use a different load management scheme for passive requests, and cache the data on each node, often more quickly than we can relay it to the next, it might be exploitable. We will need to have some sort of limit on the amount of damage that any given node can cause, such that the number of passive requests you can have pending is limited by the number of edges you have connecting to the wider network. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080715/99104ea6/attachment.pgp From toad at amphibian.dyndns.org Tue Jul 15 13:22:08 2008 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Tue, 15 Jul 2008 14:22:08 +0100 Subject: [Tech] Darknet, uptime, and true passive requests In-Reply-To: <200807151404.13389.toad@amphibian.dyndns.org> References: <200807141943.51701.toad@amphibian.dyndns.org> <179050340807150312q752ea9f1qb7efbb20bddbf7e4@mail.gmail.com> <200807151404.13389.toad@amphibian.dyndns.org> Message-ID: <200807151422.09192.toad@amphibian.dyndns.org> On Tuesday 15 July 2008 14:04, Matthew Toseland wrote: > On Tuesday 15 July 2008 11:12, Theodore Hong wrote: > > I think that there are actually two distinct though related issues > > going on here, which are downtime and polling. > > > > Downtime means that the data you want is in the network, but you can't > > get to it right now because one of the nodes that the request needs to > > go through isn't up right now (e.g. a gateway out of a darknet). > > > > Polling means that the data you want isn't in the network yet, but > > you'd like to get it sometime in the future when it is inserted. > > > > > > DOWNTIME > > > > The downtime problem is related to delay-tolerant networking - > > http://en.wikipedia.org/wiki/Delay_Tolerant_Networking > > http://www.dtnrg.org/wiki > > > > One way to think about it is to think about routing in time as well as > > space. Suppose you want to send a message along the route A-B-C, but > > A-B is only up on Monday and B-C is only up on Tuesday. You can > > complete the route by holding the message at B for a day - effectively > > 'routing' between B_on_Monday and B_on_Tuesday. > > > > The practical implication is that if your first-choice destination is > > down, sometimes it may be better to hold the request for a while and > > wait for it to come back up, rather than immediately rerouting to your > > second-choice destination. Or, you route to the second-choice > > destination, but if it fails, you hold onto it. When the first-choice > > destination comes back up, it triggers a retry - a sort of "active" > > passive request. (A terrible term - perhaps "persistent request" or > > "delay-tolerant request"?) > > > > > > POLLING > > > > Polling, on the other hand, is related to publish-subscribe: > > http://en.wikipedia.org/wiki/Publish/subscribe > > > > You set up a subscription that registers your interest in some key(s), > > together with a route showing where to find you later on. When the > > data is later inserted, it gets pushed to you. > > > > Here, the trigger event is data arriving, rather than a node coming > > up. There is no further forward routing, only sending data back along > > an already-existing route. If the route to the subscriber no longer > > exists, that's another problem, but in this case I think we should > > just drop it rather than try to find the subscriber. If s/he really > > wants it, s/he can just request it again. > > > > Thoughts? > > theo > > Okay, there are two problems, but there may be some overlap in the solutions. > First I will explain why I think we need persistent requests for polling, I > would very much appreciate some input on this. > > CURRENT SITUATION: > > We already have a form of unreliable polling. Ultra-lightweight persistent > requests last for 1 hour, are dropped if the connection is lost, are > established automatically after any request, and propagate the data back to > the originator and any other subscribers when it is found. This was > implemented shortly before 0.7. The main motivation was increasing worry > about the load that FMS is likely to put on the network. FMS is a > usenet-style chat system which uses outbox polling to prevent spam. An > earlier system, Frost, uses a global outbox and has been thoroughly DoS'ed. > > An important side-issue is that of many nodes constantly polling for big files > that have fallen out of the network. At the same time as implementing ULPRs, > we (I) also implemented per-node failure tables, so that every request for > the same key (within some period) will be routed slightly differently (within > some period). Before this, we were constantly sending requests for data we > can't find; now we send 3 requests every half hour, on the hope that 1) the 3 > requests will go to slightly different paths, and 2) if the data is found > during the half hour pause, it *may* be propagated back to us via ULPRs. Both > big file downloads and FMS's polling use the same mechanism. > > Other polling apps are under development: batosai's web of trust plugin works > on the same principles as FMS but is incompatible and may see wider use for > filesharing apps and so on. And there is a real-time chat client which seems > to work (for small numbers so far), and which presumably polls one or more > queues constantly. > > The second phase for ULPRs was going to be, and still may be, to limit the > number of requests going through any single node for any given key, on the > basis that the data will be propagated back to the requestors via ULPRs when > it is found. This should have a significant impact on network > load/efficiency. > > PASSIVE REQUESTS: > > Polling is inefficient and causes a lot of network load, especially if a large > proportion of the network are polling for an overlapping large number of keys > (as with FMS). Also, many applications will not be happy when a ULPR fails > because a connection is lost, and they aren't told about it (because ULPRs > don't do that), and their data is delayed by half an hour. This is almost > tolerable for FMS but even there it's annoying. For audio streaming and > certainly for real time chat it's not tolerable (admittedly real time chat > probably isn't something we want to encourage, since it will likely be very > inefficient in any case as even SSKs are 1KB payload). > > Full passive requests would enable a node to subscribe to a key, and not have > to worry about polling it. The network would be responsible for polling, and > it could do it far more efficiently than the client could, at least for keys > being requested by many nodes. Passive requests would not arbitrarily fail > and not tell the originator: they would be automatically rerouted when a > connection is lost or a better route is found. This could be reasonably > efficient bandwidth-wise, since we have a large number of passive requests > effectively queued, we can send a whole bunch of them in a single packet when > something changes. Passive requests would include some form of coalescing, of > course, forming a big tree if a key is popular. When the data is inserted, it > can be propagated to all subscribers very fast. All this would of course > require different load management: since a passive request has ongoing > overhead, each node's subscriptions would have to be limited, probably in > relation to its connectedness to the rest of the network to prevent an > attacker from doing DoS attacks. There would probably also be network level > priorities, which again would present some load management issues. Also, if > the capacity is available, we would be able to have a lot more passive > requests than we can currently have ULPRs, possibly persisting them on disk. > > The end user impact of all of this is that apps such as FMS should both have > much lower latency and much less impact on the network, even as it scales up, > and applications such as audio streaming will be feasible without having to > have a 30-60 minutes delay. > > Some open questions: > > 1. How many passive requests can we afford? Will churn make them very costly > for non-popular keys? Maybe... > > 2. Can we mitigate this, perhaps by not always rerouting immediately? What > would the performance cost be? It could be significant... maybe make it > related to the priority/the number of subscribers/etc? > > 3. What is the impact on the other main source of network load/inefficiency: > Clients polling big files that have mostly fallen out of the network ? ULPRs > throttle this to some degree, although for big files ULPRs won't be much help > as a single block will be rerequested less than once every hour; per-node > failure tables mean that every retry will go somewhere different, improving > the chances of finding data that isn't where it should be. Clients may want > to subscribe to the keys, but at a lower priority than their FMS traffic, and > explicitly poll to try to do an exhaustive search as well - it's a question > of how much we limit both of these strategies. If we could assume that > routing works and that data is always where it should be, and that passive > requests are low overhad, we'd simply subscribe to the key (at least for > popular keys), and block all further requests for it on the grounds that when > we find it we'll return it to all requestors. But this may not be a feasible > assumption, unless we automatically migrate data when locations are swapped, > which might cause a lot of load... Okay, now, lets look at downtime in this context: At first glance, passive requests would seem to closely match your "delay tolerant requests". However, for it to really fly, we would probably want the ability to subscribe through a node whose connection then goes offline, and have it, and the rest of the network, do the work of keeping the subscription up, and transferring the data, so that when we come back up we can have it fast. (Obviously the first implementation of passive requests shouldn't do this, and it shouldn't happen on opennet). The problem with this is what happens if a new connection comes up, which is a better target than our existing subscription? We can't abandon our old subscription unless we can get a message to the involved nodes to that effect. We could attempt a provisional rerouting, which is only completed if we manage to hook onto the old chain and cancel it, whether or not some nodes on the chain are offline... What are the advantages to having non-persistent delay tolerant requests, which seem to be the other option? Lower network resources in exchange for having to poll, I suppose is the answer... Two other big questions: Inserts: Passive inserts probably would be a bad idea. We need an insert to complete at some stage; requests complete when they find the data, or the originator gets bored of them and wants to reuse the quota / disconnects (either at all or permanently or for some period). But inserts are only limited by HTL. Also, it would leave a trail of breadcrumbs which a hideously powerful attacker (able to compromize nodes at will) would be able to follow; passive requests do this too to some degree, of course, but it's more important for inserts. Delay tolerant inserts? Might be an option... we know where we want to go, we know when it's likely to be online again... but when we've inserted, the insert is finished. Darknet merging: Not all connection issues are caused by existing connections going down and coming back up regularly. Adding new connections to connect two darknets together also has many issues. Passive requests would be perfectly happy with such a situation, but inserts might not be; and there are likely to be opportunities for attack: if you know that you are the only link to the other darknet, you can probe all their datastores on connection for any keys you are interested in... > > > > > > 2008/7/14 Matthew Toseland : > > > 1. Many people have proposed over the years that we have a "bulk" flag > which > > > can be set when the timing of a request is less important (e.g. for > splitfile > > > fetches), or a priority class for a request which is visible at the > network > > > layer. I have always opposed this mostly because it makes traffic > profiling > > > slightly easier and any sort of priority scheme would need careful > regulation > > > to prevent race-to-the-top. > > > > > > 2. Long-term, and in particularly nasty places, Freenet will have to be > mostly > > > darknet, because it is much easier to attack opennet nodes, or to block > them > > > in bulk. One of the biggest practical problems with a pure darknet is the > > > 24/7 issue: more people have laptops than have real PCs nowadays, and this > > > trend is likely to continue and accelerate, but even if people have a > desktop > > > PC, many users won't run it 24x7 for various reasons: power consumption, > > > noise, security (with encrypted disk, do you want to leave it > unattended?), > > > etc etc. Fanless home server appliances might be able to run 24x7, but > that > > > means additional expenditure to buy them. > > > > > > 3. FMS, even more than Frost, makes heavy use of SSK polling, and this is > > > likely to expand as the network grows and FMS becomes more newbie > friendly. > > > Also various innovative applications require fast propagation of data once > > > inserted (although there are frequently security issues with this). And > > > widely-wanted data which is hard to find can be effectively polled by much > of > > > the network, causing excessive load. > > > > > > 4. The solution to SSK polling etc is some form of passive requests. In > 0.7, > > > we have ultra-lightweight passive requests, which are a very limited and > > > unreliable mechanism but nonetheless should help significantly. The basic > > > principle of ULPRs is that once a request completes, each node on the > network > > > remembers who wants the data and who it has asked for it, for a short > time, > > > without making any effort to reroute if connections are lost; if the data > is > > > found it is propagated quickly to everyone who wants it. > > > > > > 5. True passive requests (0.9) would be a mechanism whereby a node could > send > > > out a request, which once it failed would be remembered permanently, > subject > > > to a (long) timeout and/or periodic renewal from the originator. It would > be > > > automatically rerouted if the network topology changes. Passive requests > > > would introduce a number of new technical challenges such as load > management > > > for persistent requests, evaluating a peer's competence in performing > them, > > > and so on, but they could greatly reduce the cost of SSK polling, > > > rerequesting common but absent data, and enable such things as medium > > > bandwidth high latency publish/subscribe for for example audio streams. > > > Passive requests would probably have to have a priority level setting. > It's a > > > big job, but a big prize... > > > > > > 6. Passive requests would go a long way to solving the uptime problem. Say > you > > > have a small darknet, say 5 nodes. Its nodes are only online during > evenings > > > local time. Its only connection to the outside world is through one node > > > which is connected to two of the small darknet, which is only online on > > > Thursdays. Right now, except on Thursdays, the network would be > essentially a > > > leaf network: our real-time routing assumes that the network is fully > > > connected. Most data will be very difficult to obtain. Real-time routing > > > requires real-time load balancing, which means that all the nodes would > > > request whatever it is they want constantly, generating load to no good > > > purpose, except on Thursdays when the requests would get through, but > > > severely limited by load management, and by the fact that more than one of > > > the small darknet may be asking for the same file. So on Thursdays, some > > > progress would be made, but often not very much. > > > > > > Now, with true passive requests, things can be very different. From the > user's > > > point of view the semantics are essentially the same: they click a link, > it > > > gets a DNF (fairly quickly), and they click the button to queue it to the > > > global queue; some time later, they get a notification that the content is > > > available. But performance could be much higher. If a node requests a > block > > > while the network is "offline", the request will propagate to all 5 nodes, > > > and then sit there waiting for something to happen. When we connect to the > > > wider network, the request is immediately rerouted to the node that just > > > connected (either because it's a better route, or because there are spare > > > hops). It propagates, fails, and is stored as a passive request on the > wider > > > network, hopefully reaching somewhere near the optimal node for the key. > When > > > the link is lost, both sides remember the other, so when/if the data is > found > > > on the wider network, it is propagated back to the originator. > Furthermore, > > > the load management would be optimised for passive requests: when the > small > > > network connects, it can immediately send a large number of passive > requests > > > for different blocks of the same file or for different files. These are > not > > > real-time requests, because they have already failed and turned into > passive > > > requests; so they can be trickled out at whatever rate the recipient sees > > > fit. Also, they are not subject to the anti-polling measures we have > > > introduced: Polling a key in 0.7 means requesting it 3 times, sleeping for > > > half an hour, and repeating ad infinitum. Further similar measures may > need > > > to be introduced at the node level to try to deal with increasing load > caused > > > by FMS, but because we reroute on getting a connection, we can immediately > > > route the requests. When we reconnect, hopefully our peer will have found > > > most of the data we requested and can transfer it at link speed (or > whatever > > > limit may be imposed for security reasons). The transfer might take longer > > > than the intersection, but I expect the whole system will be significantly > > > faster than it would be now. It's even better if you have more than two > > > network fragments: on a large darknet you might have subnetworks coming > > > online and going offline constantly, so that you never actually have a > fully > > > online network. Passive requests would happily search out every relevant > nook > > > and cranny of the network. > > > > > > Note that much of this is only feasible on darknet, because of the trust > > > connection: on opennet, passive requests probably will have to last only > as > > > long as the connection is open, and bulk transfer of passive requests is > > > certainly not feasible on opennet. > > > > > > With regards to security, it may be possible to determine whether an FMS > > > poster (for example) is on the local network, if you know when his posts > come > > > in. This is of course feasible now on such a topology, but on the other > hand > > > if nobody uses it because it's unusable, there's no threat. Passive > requests > > > would probably make it a little easier. Some form of tunneling, preferably > > > with long client controlled delays for inserts, might help to solve it, > but > > > we would have to have a way of determining that the network is too small > to > > > provide useful anonymity. > > > > > > 7. Even longer term, many ISPs and countries may deploy traffic flow > analysis > > > hardware to identify and block all (unlicensed?) peer to peer networking. > The > > > only way to beat traffic flow analysis is to not send data continually > over > > > the internet. The obvious ways to do this are: > > > 1. Parasitic transports: Steal the video stream in a VoIP call to a > friend. > > > Note that if VoIP calls are rare this won't work well, and if they are > > > artificially common to speed downloads up that will probably be > detectable. > > > 2. Fake timing: Make the transports look like e.g. a private gaming > server, > > > and fake the timing based on statistical models. This is classic stego. > It's > > > a race between you and your opponent for whoever has the better model. > Given > > > that your opponent may store traffic data indefinitely and not act > > > immediately, this is very dangerous... > > > 3. Wifi etc, non-internet constant data transfer. > > > 4. Sneakernet and physical-rendezvous-based protocols (the latter working > on > > > the Freenet threat model assumptions, so still a form of darknet, rather > than > > > Haggle's free-for-all-networking system which is also interesting but IMHO > > > dangerous in the long term, and certainly isn't Freenet). > > > > > > Passive requests are again the right tool to deal with this IMHO. They > lend > > > themselves to efficient stream subscriptions, and also enable long-term > > > downloads without assuming a fully connected network at all times. Of > course > > > for the very high latency options (sneakernet), there are other > challenges, > > > such as how to assign locations without being able to swap continually. > But > > > for medium latency caused by sparse darknets not often having many nodes > > > online simultaneously, and for transports which have the same effect > despite > > > nodes' host computers actually being switched on, it should work well. > > > > > > Comments? Am I spouting nonsense? :) Apologies for the length of this > mail, > > > it's a somewhat complex subject! > > > > > > Given the enormous implications, maybe we should postpone passive requests > to > > > post 1.0 ... but I'm worried that FMS, and polling in general, may force > our > > > hand. It's likely that there will be different parts of it implemented > over > > > different versions of Freenet in any case... > > > > > > _______________________________________________ > > > Tech mailing list > > > Tech at freenetproject.org > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > > > > > _______________________________________________ > > Tech mailing list > > Tech at freenetproject.org > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > > > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080715/a92b4f6c/attachment.pgp From m.rogers at cs.ucl.ac.uk Tue Jul 15 14:00:10 2008 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Tue, 15 Jul 2008 15:00:10 +0100 Subject: [Tech] Darknet, uptime, and true passive requests In-Reply-To: <179050340807150312q752ea9f1qb7efbb20bddbf7e4@mail.gmail.com> References: <200807141943.51701.toad@amphibian.dyndns.org> <179050340807150312q752ea9f1qb7efbb20bddbf7e4@mail.gmail.com> Message-ID: <487CAD6A.70701@cs.ucl.ac.uk> Theodore Hong wrote: > I think that there are actually two distinct though related issues > going on here, which are downtime and polling. > > Downtime means that the data you want is in the network, but you can't > get to it right now because one of the nodes that the request needs to > go through isn't up right now (e.g. a gateway out of a darknet). > > Polling means that the data you want isn't in the network yet, but > you'd like to get it sometime in the future when it is inserted. This seems like a useful distinction. > DOWNTIME > > The downtime problem is related to delay-tolerant networking - > http://en.wikipedia.org/wiki/Delay_Tolerant_Networking > http://www.dtnrg.org/wiki In my opinion we shouldn't over-complicate the design for the sake of nodes that are only online once a week - maybe we should avoid sending them inserts, but a DTN layer would be overkill. Routing, caching and swapping are all based on the assumption of a single well-connected network. If people can't remain connected to the network then maybe they should use a DTN instead of Freenet, but Freenet shouldn't try to be a DTN *and* a scalable anonymous DHT. There's no such thing as an egg-laying woolmilkpig. :-) > POLLING > > Polling, on the other hand, is related to publish-subscribe: > http://en.wikipedia.org/wiki/Publish/subscribe > > You set up a subscription that registers your interest in some key(s), > together with a route showing where to find you later on. When the > data is later inserted, it gets pushed to you. > > Here, the trigger event is data arriving, rather than a node coming > up. There is no further forward routing, only sending data back along > an already-existing route. If the route to the subscriber no longer > exists, that's another problem, but in this case I think we should > just drop it rather than try to find the subscriber. If s/he really > wants it, s/he can just request it again. Again, I would argue against adding complexity unless we definitely need it. Is FMS currently the only use case for pub-sub? Is there any other way to decrease the cost of outbox polling? Here's a suggestion: each FMS identity already publishes a list of trusted identities and polls their outboxes. When you discover that a trusted identity has updated its outbox, add the key to your own outbox (unless you've already seen the key). This doesn't harm your privacy because you've already revealed that you trust that identity. After polling each outbox, wait for a random period before polling again. The combination of reposting and random polling creates a gossip protocol, which should allow nodes to discover new messages relatively quickly with a limited amount of polling per node. This can all be done at the application layer without pub-sub. Cheers, Michael From toad at amphibian.dyndns.org Wed Jul 16 10:55:36 2008 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Wed, 16 Jul 2008 11:55:36 +0100 Subject: [Tech] Darknet, uptime, and true passive requests In-Reply-To: <487CAD6A.70701@cs.ucl.ac.uk> References: <200807141943.51701.toad@amphibian.dyndns.org> <179050340807150312q752ea9f1qb7efbb20bddbf7e4@mail.gmail.com> <487CAD6A.70701@cs.ucl.ac.uk> Message-ID: <200807161155.50418.toad@amphibian.dyndns.org> On Tuesday 15 July 2008 15:00, Michael Rogers wrote: > Theodore Hong wrote: > > I think that there are actually two distinct though related issues > > going on here, which are downtime and polling. > > > > Downtime means that the data you want is in the network, but you can't > > get to it right now because one of the nodes that the request needs to > > go through isn't up right now (e.g. a gateway out of a darknet). > > > > Polling means that the data you want isn't in the network yet, but > > you'd like to get it sometime in the future when it is inserted. > > This seems like a useful distinction. > > > DOWNTIME > > > > The downtime problem is related to delay-tolerant networking - > > http://en.wikipedia.org/wiki/Delay_Tolerant_Networking > > http://www.dtnrg.org/wiki > > In my opinion we shouldn't over-complicate the design for the sake of > nodes that are only online once a week - maybe we should avoid sending > them inserts, but a DTN layer would be overkill. IMHO nodes that are not online all the time - as opposed to nodes whose downtime is so low that it's hardly feasible to route, I was taking an extreme example to try to make things clearer - are likely to be the majority of nodes either on an opennet or on a darknet. Most Freenet users are likely to be non-geeks who won't run their computers 24x7 for various good reasons: - Noise. - Pollution. - Energy bills. - Other people. - Security. (The paranoid will encrypt their disk and therefore only run it when they are there). On an opennet we can deal with them by reconnecting if our old peers have slots and are port forwarded, otherwise reannouncing. On a darknet, it's a lot more difficult. IMHO Freenet will not be widely deployable or usable, in the long term, unless it can be reasonably happy with say 10x7. > > Routing, caching and swapping are all based on the assumption of a > single well-connected network. If people can't remain connected to the > network then maybe they should use a DTN instead of Freenet, but Freenet > shouldn't try to be a DTN *and* a scalable anonymous DHT. There's no > such thing as an egg-laying woolmilkpig. :-) I'm not convinced. Freenet is not defined by its real-time behaviour, so much as by its goals (censorship resistant datastore), and its need to build a global f2f darknet to support those goals. > > > POLLING > > > > Polling, on the other hand, is related to publish-subscribe: > > http://en.wikipedia.org/wiki/Publish/subscribe > > > > You set up a subscription that registers your interest in some key(s), > > together with a route showing where to find you later on. When the > > data is later inserted, it gets pushed to you. > > > > Here, the trigger event is data arriving, rather than a node coming > > up. There is no further forward routing, only sending data back along > > an already-existing route. If the route to the subscriber no longer > > exists, that's another problem, but in this case I think we should > > just drop it rather than try to find the subscriber. If s/he really > > wants it, s/he can just request it again. > > Again, I would argue against adding complexity unless we definitely need > it. Is FMS currently the only use case for pub-sub? Is there any other > way to decrease the cost of outbox polling? We already have ULPRs, which do decrease the cost somewhat, and have other beneficial effects. > > Here's a suggestion: each FMS identity already publishes a list of > trusted identities and polls their outboxes. When you discover that a > trusted identity has updated its outbox, add the key to your own outbox > (unless you've already seen the key). This doesn't harm your privacy > because you've already revealed that you trust that identity. After > polling each outbox, wait for a random period before polling again. The > combination of reposting and random polling creates a gossip protocol, > which should allow nodes to discover new messages relatively quickly > with a limited amount of polling per node. This can all be done at the > application layer without pub-sub. This has been proposed more than once, I don't know if it has been implemented yet. Nonetheless there is going to be a hell of a lot of polling going on from FMS. Other apps may use batosai's web of trust plugin. There is a real time chat client which some people are using, which no doubt does evil levels of polling. And longer term it'd be great if we could have e.g. audio streams. Apart from that, a lot of traffic is nodes rerequesting stuff that has fallen out of the network, or has moved to places where it shouldn't be, I'm not sure how that relates to this though, what I do know is that some mechanism to limit it will work much better if it has some form of subscription associated with it (that being more or less the principle behind ULPRs). > > Cheers, > Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080716/76bbd87e/attachment.pgp From m.rogers at cs.ucl.ac.uk Wed Jul 16 18:45:53 2008 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Wed, 16 Jul 2008 19:45:53 +0100 Subject: [Tech] Darknet, uptime, and true passive requests In-Reply-To: <200807161155.50418.toad@amphibian.dyndns.org> References: <200807141943.51701.toad@amphibian.dyndns.org> <179050340807150312q752ea9f1qb7efbb20bddbf7e4@mail.gmail.com> <487CAD6A.70701@cs.ucl.ac.uk> <200807161155.50418.toad@amphibian.dyndns.org> Message-ID: <487E41E1.2090405@cs.ucl.ac.uk> Matthew Toseland wrote: > Most Freenet users are likely > to be non-geeks who won't run their computers 24x7 for various good reasons: You might well be right, but in that case we need to ask some difficult questions: * Is it even possible to build a scalable, robust, distributed cache where most of the nodes are offline most of the time and some of them only connect to their friends? * If it is possible, do we have a clear idea of how to build it, or are we just piling systems on top of each other? * What kind of applications can such a network support? What kind of applications should it not even try to support? > I'm not convinced. Freenet is not defined by its real-time behaviour, so much > as by its goals (censorship resistant datastore), and its need to build a > global f2f darknet to support those goals. I think you've hit the nail on the head - Freenet has always been defined by the broad goal of censorship resistance rather than by concrete use cases, and as a result the means of achieving that goal keep changing. For example, should Freenet support real-time communication? Long-term storage? Small, isolated darknets connected to the main network by camel caravan? A large darknet in China and a large darknet in the West joined by a handful of cross-border links? > And longer term it'd be great if we could have e.g. audio > streams. And a pony. ;-) Cheers, Michael From toad at amphibian.dyndns.org Wed Jul 16 19:15:20 2008 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Wed, 16 Jul 2008 20:15:20 +0100 Subject: [Tech] Defining Freenet was Re: Darknet, uptime, and true passive requests In-Reply-To: <487E41E1.2090405@cs.ucl.ac.uk> References: <200807141943.51701.toad@amphibian.dyndns.org> <200807161155.50418.toad@amphibian.dyndns.org> <487E41E1.2090405@cs.ucl.ac.uk> Message-ID: <200807162015.30242.toad@amphibian.dyndns.org> On Wednesday 16 July 2008 19:45, Michael Rogers wrote: > Matthew Toseland wrote: > > Most Freenet users are likely > > to be non-geeks who won't run their computers 24x7 for various good reasons: > > You might well be right, but in that case we need to ask some difficult > questions: > > * Is it even possible to build a scalable, robust, distributed cache > where most of the nodes are offline most of the time Yes. Opennet can do this. Although performance will start very slow after startup after a long downtime. > and some of them > only connect to their friends? Good question. > > * If it is possible, do we have a clear idea of how to build it, or are > we just piling systems on top of each other? Well... I have some ideas, that's all. Nothing concrete, simulated in detail, mathematically modelled and guaranteed to work. :) > > * What kind of applications can such a network support? Downloads. Uploads. Non-realtime chat. Web sites with offline fallback, similar to offline browsing: this page is not currently available, click here to queue it, we will tell you when it is found. (We already do this, it is useful for not-so-popular freesites that need some retrying to fetch). > What kind of > applications should it not even try to support? Real time chat. But even there, a large minority of the network - or some separate darknets - may have high enough uptimes. And opennet can support this. > > > I'm not convinced. Freenet is not defined by its real-time behaviour, so much > > as by its goals (censorship resistant datastore), and its need to build a > > global f2f darknet to support those goals. > > I think you've hit the nail on the head - Freenet has always been > defined by the broad goal of censorship resistance rather than by > concrete use cases, and as a result the means of achieving that goal > keep changing. Okay... > > For example, should Freenet support real-time communication? To the extent that that is possible. IMHO Freenet's mission is to provide an uncensorable network that provides as much of the services which are no longer usable to those subject to censorship as possible. However, on a low-uptime darknet, real time communication is going to be difficult. But on an opennet, or even a high-uptime darknet, should we actively discourage real-time communication? I'm not convinced, I don't see any problem with offering services which won't be feasible longer term, if they get people onto the network now. :) > Long-term storage? If we could, it'd be great, wouldn't it? But we can't afaics. Well, there are a few ideas floating about ("inverse passive requests"), but that's really long term stuff and likely a big security risk, as well as of course not being completely reliable. I do think that there is useful work to be done on improving data reachability however, the current network has some issues with this. > Small, isolated darknets connected to the main network by camel > caravan? I would certainly like to support this if it is possible, and if the characteristics of the resulting network don't make it pointless. The main concerns afaik: - The single connection is a point of attack. If the recipient is evil, he can both observe and control all traffic out of the small darknet. - The single connection has limited capacity and may have high latency. It's essential that the two networks are aware of their separatedness, and that requests are tried on the small darknet before being forwarded to the wider network. Rationing of limited capacity may also be a big issue. > A large darknet in China and a large darknet in the West joined > by a handful of cross-border links? This is very similar to the last question. Given the limited interconnect capacity, and to some degree given the likely differences in content/interests between the networks, it is essential that requests be tried on the "local" network before the "remote" one. It's a question of (automatic? manual?) partitioning and then using that information. Both of these last two cases, but especially the latter, follow directly from the darknet concept; IMHO they are important long-term. However, there is much to do in the short term, we need to be careful with allocating resources given the pressures on funds (immediate problem) and time (external/legal/legislative problem). > > > And longer term it'd be great if we could have e.g. audio > > streams. > > And a pony. ;-) On opennet, or even on a high uptime darknet, low bitrate audio streaming implemented via passive requests should be feasible, with lag hopefully in minutes rather than tens of minutes. Of course this depends very much on load management; in theory it should have been feasible on 0.5, but I couldn't get sufficient performance out of it. > > Cheers, > Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080716/a24c9839/attachment.pgp From m.rogers at cs.ucl.ac.uk Thu Jul 17 21:15:13 2008 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Thu, 17 Jul 2008 22:15:13 +0100 Subject: [Tech] Defining Freenet was Re: Darknet, uptime, and true passive requests In-Reply-To: <200807162015.30242.toad@amphibian.dyndns.org> References: <200807141943.51701.toad@amphibian.dyndns.org> <200807161155.50418.toad@amphibian.dyndns.org> <487E41E1.2090405@cs.ucl.ac.uk> <200807162015.30242.toad@amphibian.dyndns.org> Message-ID: <487FB661.1070307@cs.ucl.ac.uk> Matthew Toseland wrote: >> Small, isolated darknets connected to the main network by camel >> caravan? > > I would certainly like to support this if it is possible, and if the > characteristics of the resulting network don't make it pointless. >>> And longer term it'd be great if we could have e.g. audio >>> streams. I've got this mental image of RealPlayer "buffering" while a camel plods across the Sahara with the next block of data. :-) Cheers, Michael From toad at amphibian.dyndns.org Thu Jul 17 22:38:34 2008 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Thu, 17 Jul 2008 23:38:34 +0100 Subject: [Tech] Defining Freenet was Re: Darknet, uptime, and true passive requests In-Reply-To: <487FB661.1070307@cs.ucl.ac.uk> References: <200807141943.51701.toad@amphibian.dyndns.org> <200807162015.30242.toad@amphibian.dyndns.org> <487FB661.1070307@cs.ucl.ac.uk> Message-ID: <200807172338.42175.toad@amphibian.dyndns.org> On Thursday 17 July 2008 22:15, Michael Rogers wrote: > Matthew Toseland wrote: > >> Small, isolated darknets connected to the main network by camel > >> caravan? > > > > I would certainly like to support this if it is possible, and if the > > characteristics of the resulting network don't make it pointless. > > >>> And longer term it'd be great if we could have e.g. audio > >>> streams. > > I've got this mental image of RealPlayer "buffering" while a camel plods > across the Sahara with the next block of data. :-) LOL. :) IMHO real time and non-real-time are not mutually exclusive. > > Cheers, > Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20080717/d0fda52e/attachment.pgp