From toad at amphibian.dyndns.org Fri Sep 7 13:02:08 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Fri, 7 Sep 2007 14:02:08 +0100 Subject: [Tech] Social networking sites interop vs lock-in Message-ID: <200709071402.13103.toad@amphibian.dyndns.org> Worth a read: http://bradfitz.com/social-graph-problem/ Ideally Freenet could get your friends off the social networking sites then send them IMs or emails to get them peered. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070907/5fd4c605/attachment.pgp From bruthoma at student.ethz.ch Tue Sep 18 07:31:58 2007 From: bruthoma at student.ethz.ch (Thomas Bruderer) Date: Tue, 18 Sep 2007 07:31:58 +0000 (UTC) Subject: [Tech] Theoretical Master Thesis on sybil attacks on freenet Message-ID: Hello everyone I announced a half year ago that I write a master thesis over possible sibyl attacks on freenet. I don't want to spam the devl list because the info of the paper is already known, however not in these clear detail. So I post it here on the technical list, since the information of the paper won't help much in developing. The paper gives some theoretical background of how safe you might or might not be in freenet, however the network topology is very very simplified throughout the paper to make it possible to analyze it theoretically. For those who won't believe that darknet and trusted peers are a necessity the paper will open your eyes. Freenet 0.5 and Opennet are wide open to attacks described in the paper. A small step for freenet a big step for me, I finally have some spare time :) Anyone interested can read the paper here, its quite mathematical, but I hope easy to read to any Undergraduate CS Student. http://download.apophis.ch/paper/MA.pdf cheers Thomas Bruderer aka. Apophis From sback at sback.it Tue Sep 18 11:11:35 2007 From: sback at sback.it (Alberto Bacchelli) Date: Tue, 18 Sep 2007 13:11:35 +0200 Subject: [Tech] Theoretical Master Thesis on sybil attacks on freenet In-Reply-To: References: Message-ID: Thank you for this reference, I'll surely read it Sback On Sep 18, 2007, at 9:31 AM, Thomas Bruderer wrote: > Hello everyone > > I announced a half year ago that I write a master thesis over > possible sibyl > attacks on freenet. I don't want to spam the devl list because the > info of the > paper is already known, however not in these clear detail. So I > post it here on > the technical list, since the information of the paper won't help > much in > developing. > > The paper gives some theoretical background of how safe you might > or might not > be in freenet, however the network topology is very very simplified > throughout > the paper to make it possible to analyze it theoretically. > > For those who won't believe that darknet and trusted peers are a > necessity the > paper will open your eyes. Freenet 0.5 and Opennet are wide open to > attacks > described in the paper. A small step for freenet a big step for me, > I finally > have some spare time :) > > Anyone interested can read the paper here, its quite mathematical, > but I hope > easy to read to any Undergraduate CS Student. > > http://download.apophis.ch/paper/MA.pdf > > cheers > Thomas Bruderer aka. Apophis > > _______________________________________________ > Tech mailing list > Tech at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech From toad at amphibian.dyndns.org Tue Sep 18 17:27:31 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Tue, 18 Sep 2007 18:27:31 +0100 Subject: [Tech] Theoretical Master Thesis on sybil attacks on freenet In-Reply-To: References: Message-ID: <200709181827.42029.toad@amphibian.dyndns.org> Cool! Any validation of our fundamental design principles is welcome. I will link the paper from the website soon, and try to get around to reading it. On Tuesday 18 September 2007 08:31, Thomas Bruderer wrote: > Hello everyone > > I announced a half year ago that I write a master thesis over possible sibyl > attacks on freenet. I don't want to spam the devl list because the info of the > paper is already known, however not in these clear detail. So I post it here on > the technical list, since the information of the paper won't help much in > developing. > > The paper gives some theoretical background of how safe you might or might not > be in freenet, however the network topology is very very simplified throughout > the paper to make it possible to analyze it theoretically. > > For those who won't believe that darknet and trusted peers are a necessity the > paper will open your eyes. Freenet 0.5 and Opennet are wide open to attacks > described in the paper. A small step for freenet a big step for me, I finally > have some spare time :) > > Anyone interested can read the paper here, its quite mathematical, but I hope > easy to read to any Undergraduate CS Student. > > http://download.apophis.ch/paper/MA.pdf > > cheers > Thomas Bruderer aka. Apophis > > _______________________________________________ > Tech mailing list > Tech at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070918/251c3462/attachment.pgp From toad at amphibian.dyndns.org Tue Sep 18 22:47:00 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Tue, 18 Sep 2007 23:47:00 +0100 Subject: [Tech] Theoretical Master Thesis on sybil attacks on freenet In-Reply-To: <46F035EA.7030600@student.ethz.ch> References: <200709181827.42029.toad@amphibian.dyndns.org> <46F035EA.7030600@student.ethz.ch> Message-ID: <200709182347.09435.toad@amphibian.dyndns.org> :| Will read it anyway. On Tuesday 18 September 2007 21:32, you wrote: > Matthew Toseland wrote: > > Cool! Any validation of our fundamental design principles is welcome. I will > > link the paper from the website soon, and try to get around to reading it. > > > > On Tuesday 18 September 2007 08:31, Thomas Bruderer wrote: > > > >> Hello everyone > >> > >> I announced a half year ago that I write a master thesis over possible sibyl > >> attacks on freenet. I don't want to spam the devl list because the info of > >> > > the > > > >> paper is already known, however not in these clear detail. So I post it here > >> > > on > > > >> the technical list, since the information of the paper won't help much in > >> developing. > >> > >> The paper gives some theoretical background of how safe you might or might > >> > > not > > > >> be in freenet, however the network topology is very very simplified > >> > > throughout > > > >> the paper to make it possible to analyze it theoretically. > >> > >> For those who won't believe that darknet and trusted peers are a necessity > >> > > the > > > >> paper will open your eyes. Freenet 0.5 and Opennet are wide open to attacks > >> described in the paper. A small step for freenet a big step for me, I > >> > > finally > > > >> have some spare time :) > >> > >> Anyone interested can read the paper here, its quite mathematical, but I > >> > > hope > > > >> easy to read to any Undergraduate CS Student. > >> > >> http://download.apophis.ch/paper/MA.pdf > >> > >> cheers > >> Thomas Bruderer aka. Apophis > >> > >> _______________________________________________ > >> Tech mailing list > >> Tech at freenetproject.org > >> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > >> > >> > >> > Well, thanks but obviously I have to revise it, got the grading today > and obviously there are some major flaws in the paper. > > Greets Thomas > > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070918/b9ea349f/attachment.pgp From jargonautti at hotmail.com Thu Sep 20 16:37:38 2007 From: jargonautti at hotmail.com (Jusa Saari) Date: Thu, 20 Sep 2007 19:37:38 +0300 Subject: [Tech] Swapping and datastore Message-ID: What happens to the datastore when a node swaps its position? From what I've understood, the requests are routed based on node locations, which causes the contents of a node's datastore to reflect its position; if a node changes its position to the other side of the position ring, it will receive very few requests relevant to its datastore, which means that the store will effectively get wiped clean every time the node swaps. I tried to look at freenetproject.org for this, but the documentation there (http://freenetproject.org/understand.html) seems to be describing the routing algorithm which predated NGRouting. So, given this, would it make sense to simply delete from the store any keys not near the current location after a swap? They can't be accessed anyway except by pure luck, and will simply waste store space which could otherwise be used to fill the store up with relevant keys. From toad at amphibian.dyndns.org Thu Sep 20 19:07:44 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Thu, 20 Sep 2007 20:07:44 +0100 Subject: [Tech] Swapping and datastore In-Reply-To: References: Message-ID: <200709202007.51985.toad@amphibian.dyndns.org> On Thursday 20 September 2007 17:37, Jusa Saari wrote: > What happens to the datastore when a node swaps its position? From what > I've understood, the requests are routed based on node locations, which > causes the contents of a node's datastore to reflect its position; if a > node changes its position to the other side of the position ring, it will > receive very few requests relevant to its datastore, which means that the > store will effectively get wiped clean every time the node swaps. > > I tried to look at freenetproject.org for this, but the documentation > there (http://freenetproject.org/understand.html) seems to be describing > the routing algorithm which predated NGRouting. > > So, given this, would it make sense to simply delete from the store any > keys not near the current location after a swap? They can't be accessed > anyway except by pure luck, and will simply waste store space which could > otherwise be used to fill the store up with relevant keys. It's not necessary. They will be automatically deleted when new keys are added, in accordance with the Least Recently Used cache replacement algorithm. Until new data comes in we may as well keep the old data. Also I'd expect *generally* that long range swaps occur mostly towards the beginning of a node's lifetime, rather than later on. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070920/842da504/attachment.pgp From toad at amphibian.dyndns.org Thu Sep 20 19:37:09 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Thu, 20 Sep 2007 20:37:09 +0100 Subject: [Tech] [freenet-support] Another Freenet client, FibonIce In-Reply-To: References: Message-ID: <200709202037.14505.toad@amphibian.dyndns.org> Replying to original author and -tech, which is more appropriate than -support for this mail. On Saturday 15 September 2007 20:40, you wrote: > Hello, I'm a spanish student that want to write a new p2p software based on > Freenet protocol, but applying another point of view. Cool. (It shows, your english is quite difficult in parts, still, I don't know any Spanish... :) ). > > In this one case the program would not serve to create a network of > appearance similar to the World Wide Web being used pages written in HTML, > but that more similar to would be known programs p2p than simply they > interchange archives (although with some differences also). You want to build a Freenet client which is closer to p2p than to www. Okay. (Check out Thaw, Frost first btw). > > The differences would be: > ? the anonymity > ? the impossibility of the censorship > ? "a new" system of searches > ? others > > The program would be focused coverall to the text document transmission, > generating more than a "signature hash" by file, could be made a signature > hash by each paragraph or each page (but applying it only to the text, not > to the format) so that whole archives like part of the chain could be used > search. It would serve this to find documents with relations to each other > (with common content). Aha, so it *is* about web pages ?? I thought you said it would be closer to traditional p2p than to web pages as currently implemented on Freenet ? > > In addition I have thought that it would be possible to create a "robot" > like an object (the implementation of a class) traveling that was dedicated > to look for information to create data bases search. The object in himself > would contain all its information ciphered with asymmetric key, and it would > only maintain direction IP of the ip adress of the robot "house" (in this > one case is only possible to know that a particular ip address is part of > that network, but nothing else), with which it would be sending data > periodically, and traveling through the nodes (that would have a mechanism > to manage the robots). So you send out some sort of request ("robot"), which is passed from node to node, until it reaches some depth, creating an index of some sort? > As soon as the robot detected that direction IP of > the "master computer" does not respond, "would die". Definitely not a good idea. Makes the robot traceable, and makes the network harvestable. Both are bad. Darknet nodes must only connect with their darknet peers; opennet nodes can connect more broadly but still we don't want to make it easy for an attacker. Explanation of "harvesting" : http://wiki.freenetproject.org/NodeHarvesting > Whenever the program > p2p began would send a robot to update its data base (if therefore it wished > the user it, he would be optional). The data base would only contain > identifiers of archives associated with its names, in no case its situation > within the network. Sounds extremely heavy. Anything you build must be scalable, if you want it to be widely used. Freenet certainly must be scalable. At the moment, text searching on Freenet is accomplished by a single node running a "spider" program/plugin (XMLSpider), which does the same thing that Google does - it fetches pages, looks for more pages from links in those its already fetched, and fetches them too, and records which keywords were in which pages ... Unlike Google, our spider publishes its keyword indexes onto Freenet, so that another plugin, the XMLLibrarian, can search them for a user, downloading only the indexes which are needed. > (A mechanism of digital signature would be used to > verify that the internal code of the object robot is not malignant). *!*!*"*!!*?$*%"&!*$%& Seriously, mobile code on a supposedly secure network? There'd have to be a *REALLY* good reason to even consider this. Creating a secure execution environment that doesn't give away stuff you don't want to give away ("hmmm, this node seems to insert this freesite, and also this frost identity..."), is a major challenge in itself. > > On the other hand, I believe that it is necessary that the network is able > to absorb information of other networks (always noticing the user that they > come from nonsafe channels). These networks would be the World Wide Web, the > network Ed2k (Edonkey) or the network Kademlia (eMule), for example. This is an interesting idea which has occasionally been proposed. The problem is that any time you go to an external, insecure network, you: a) Reveal the fact that you are running a node AND b) Potentially request content that may get you into trouble > > The data acquisition would consist of using api of finders like google, > yahoo either others to collect data of the World Wide Web and the motors > search or implemented of the networks ed2k and kad. This one single type of > searches would become at local level, not through freenet, but I believe > that equipment would be enough. It would serve mainly to contribute content > of fast form to network freenet. (I do not talk about material which it > harms the author rights, although is a collateral effect that surely would > appear). Okay, if the searches are simply part of your client, that's different - if you want to make a client which uses Freenet, edonkey, and the web, that's fine by me, let me know if you need any help e.g. the ability to get a hash of content before fetching it fully. > > The main problem is that I believe that is a loss of time to design and to > create a new protocol for this one type of network when already some > similars exist, and is by this reason why I would like to learn a little on > the operation and implementation of this one. I need to document itself on > your system and unfortunately my ISP blocks the access to your page? Your ISP sucks. :( Use Tor, or I2P, or a proxy, to get to our site. > > Of some form, although outside competing (I am sure that you surpass to me > widely in knowledge on this one subject, is difficult that I can compete), I > believe that we could collaborate to improve the freenet. > > By the way, it forgot, my project to it will be written in C# because I > believe that it is more portable than Java (verified in my they debian of 64 > bits). Not true. Sun Java admittedly will only run on a few platforms, but Kaffe and GCJ between them will run on just about anything. And since Sun released the source code this will only get better. And yes, right now Freenet only runs on Sun Java, but that's because of lack of demand rather than any serious unresolvable technical issue afaics. I personally develop Freenet on an AMD64 system using debian etch. > In addition, the platform. Net is faster than the JVM, and although > the project Monkey is something slower at the moment, I do not doubt that > someday next it will surpass in yield to the JVM. Not true either. Java is fast. Arguably it's faster than C in many areas. Not that it doesn't have other problems. Mono/C# are very similar technologies to Java. > > Thank you for your atention. > > Andreu Correa Casablanca. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070920/c92c032d/attachment.pgp From cacopatane at gmail.com Thu Sep 20 22:48:38 2007 From: cacopatane at gmail.com (Caco Patane) Date: Thu, 20 Sep 2007 19:48:38 -0300 Subject: [Tech] [freenet-support] Another Freenet client, FibonIce In-Reply-To: <200709202037.14505.toad@amphibian.dyndns.org> References: <200709202037.14505.toad@amphibian.dyndns.org> Message-ID: > > Hello, I'm a spanish student that want to write a new p2p software based on > > Freenet protocol, but applying another point of view. > > Cool. (It shows, your english is quite difficult in parts, still, I don't know > any Spanish... :) ). ... Andreu, join #freenet-es at irc.freenode.net =D Saludos, Caco_Patane From toad at amphibian.dyndns.org Fri Sep 21 00:13:55 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Fri, 21 Sep 2007 01:13:55 +0100 Subject: [Tech] Theoretical Master Thesis on sybil attacks on freenet In-Reply-To: References: Message-ID: <200709210114.00428.toad@amphibian.dyndns.org> ' The calculation of the routing table has changed several times, two of them were major changes in the protocol. After the initial Paper the first big change was called the "Next Generation Routing" [7] and the last big step was the introduction of a small world topology. [1] This newest attempt was the creation of a p2p network which relies on personal contacts of people.' Not true. We always relied on a small world topology: The network is only navigable if it is a small world network. However, the original Freenet concept was purely opennet, and therefore relied to a very large degree on "destination sampling" (connect to the data source when you get a successful request). We understand this rather better now than we did thanks to Oskar's recent work. With regards to random routing for a few steps ... unfortunately it doesn't solve the problem (it's been proposed lots of times of course): - You can only do it for a few hops before getting into real routing. Therefore your anonymity set is only those nodes which could have random routed to you. And it can be a fairly uneven set. - Much worse: If two requests can be correlated by e.g. both being part of the same splitfile, both being post inserts from the same frost ID etc ... then you can do *nasty* statistical attacks based on the proportion of that known quantity that are routed through the attacker's node. Re section 4.2: - Are the "1" occurrences leaf nodes ? - What do the "2" and "3"'s look like topologically speaking? What exactly sucks about the tree? It would be helpful to understand this so we can advise people not to abuse darknet by setting up such structures (e.g. mirroring external authority structures). Clearly if you get a request from below you in the tree, you can identify that the node is within that subtree - is that the totality of the dangers of a tree topology? Theorem 7: Cool! On the circle, if the source is just below the left attacker, the left attacker will get requests which get very close to the exact opposite of the request sender's own location. And you have the opposite with the right hand attacker. The more blocks, the closer the requests will get to the exact opposite point of the sender on the ring, and the more accurately you can identify the source... Basically the operation is this (assuming pure greedy routing and no overload etc): We have a request for location T. Our location is L. We want to find S, the location of the source node. We can reasonably conclude that S is between T' (exact opposite of T) and L (our location), within the shortest arc linking the two points. If there are enough blocks, enough different T's, the point at which we stop getting requests tells us where the source is on the circle. However: - Routing may not be purely greedy, because of e.g. backoff. - The topology may not be close enough to a ring for this to work anyway. Of course there's a question as to whether the network will work at all in that case. - Routing may be deliberately obscured by e.g. a few random hops from time to time. - Routing will not always stop when it reaches the closest node to the target. Probe requests do, but real requests and inserts have an HTL limit, currently continuing for 10 hops after the last improvement in the closeness to the target. - If the source is a long way away, we will only receive a small fraction of the requests it sends. How much does this limit accuracy? If you're too far away you may not get any requests, or not enough to connect them together. - Finding the target's rough location doesn't instantly find the target. An attacker would probably try to narrow down the search by connecting to nodes closer to the location in question. - It doesn't work on popular files which multiple requestors may be fetching simultaneously. It would be fascinating to do a more accurate attack simulation! How much information can be gained about a requestor, and to what degree is this constrained by the above factors? (Introducing them one at a time, of course...) Another attack you can do, if you are close to the target, is take into account the exact local topology and use the proportion of requests coming to your node to identify which node it is. "Beware of high degree" is an interesting message. I'm not quite sure how you support it. Generally we've assumed that reasonably high degree is a good thing - certainly it will *significantly* reduce the number of hops needed to find data, and it's quite helpful in getting small-world-like networks... (Too few connections and you probably won't have the occasional long range one). Are you saying high average degree is a problem, or that individual nodes with degree far above the average (aka ubernodes) are a problem? As far as dependant vs independant messages go ... we have a few options: - Bundle all dependant messages into huge files and transfer them all at once in a single request. This isn't very practical: transferring a 1GB file with no multi-sourcing is going to take roughly forever. Having variable block sizes up to some large limit might help a bit, but would suck for various network/practical/code reasons, as it did on 0.5, and it wouldn't solve the problem, only make correlation attacks a bit harder. - Bundle dependant requests together, and route them as a blob, keeping them bundled for a number of random hops. This has performance issues, relies on tunnels which may break if any node goes offline, doesn't work well with long lived requests ... - Premix routing. Establish an explicit anonymity set of most of the nodes within some number of hops, and use the darknet to work out which nodes are "real" versus which are bogus Sybil's, or try to plug in a network-wide mixnet layer like I2P for opennet. - Just live with it: you can identify your neighbours' traffic. This is plainly unacceptable. On Tuesday 18 September 2007 08:31, you wrote: > Hello everyone > > I announced a half year ago that I write a master thesis over possible sibyl > attacks on freenet. I don't want to spam the devl list because the info of the > paper is already known, however not in these clear detail. So I post it here on > the technical list, since the information of the paper won't help much in > developing. > > The paper gives some theoretical background of how safe you might or might not > be in freenet, however the network topology is very very simplified throughout > the paper to make it possible to analyze it theoretically. > > For those who won't believe that darknet and trusted peers are a necessity the > paper will open your eyes. Freenet 0.5 and Opennet are wide open to attacks > described in the paper. A small step for freenet a big step for me, I finally > have some spare time :) > > Anyone interested can read the paper here, its quite mathematical, but I hope > easy to read to any Undergraduate CS Student. > > http://download.apophis.ch/paper/MA.pdf > > cheers > Thomas Bruderer aka. Apophis -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070921/a8533312/attachment.pgp From jargonautti at hotmail.com Fri Sep 21 10:17:32 2007 From: jargonautti at hotmail.com (Jusa Saari) Date: Fri, 21 Sep 2007 13:17:32 +0300 Subject: [Tech] Swapping and datastore References: <200709202007.51985.toad@amphibian.dyndns.org> Message-ID: On Thu, 20 Sep 2007 20:07:44 +0100, Matthew Toseland wrote: > On Thursday 20 September 2007 17:37, Jusa Saari wrote: >> What happens to the datastore when a node swaps its position? From what >> I've understood, the requests are routed based on node locations, which >> causes the contents of a node's datastore to reflect its position; if a >> node changes its position to the other side of the position ring, it >> will receive very few requests relevant to its datastore, which means >> that the store will effectively get wiped clean every time the node >> swaps. >> >> I tried to look at freenetproject.org for this, but the documentation >> there (http://freenetproject.org/understand.html) seems to be describing >> the routing algorithm which predated NGRouting. >> >> So, given this, would it make sense to simply delete from the store any >> keys not near the current location after a swap? They can't be accessed >> anyway except by pure luck, and will simply waste store space which >> could otherwise be used to fill the store up with relevant keys. > > It's not necessary. They will be automatically deleted when new keys are > added, in accordance with the Least Recently Used cache replacement > algorithm. Until new data comes in we may as well keep the old data. Hmm... In that case, perhaps it would be worth it to remember recent swaps? Because if we get a request relevant to our new position, the node which last occupied it could still have the key in its datastore. But then again, this would mess with a pure LRU cache replacement algorithm, neccessiating adding the closeness of the key to our current position as a factor. Argh :(. > Also I'd expect *generally* that long range swaps occur mostly towards the > beginning of a node's lifetime, rather than later on. In a stable network with long-living nodes, yes. The few times I tried to get into the "darknet", I lost all my contacts (from #freenet-refs or whatever it was called) in a week or so. This would likely be true of opennet as well; in fact I'd say that at any time, most freenet nodes have only been in the network for a short time. This could potentially mean that, since those short-lived nodes are unable to do anything very useful, the semi-permanent network nodes which form the "core" of Freenet will end up serving most requests, and therefore get overloaded easily, while the nodes not directly connected to them won't be able to find anything. Is there a way to simulate the effect of constantly adding and shortly afterwards removing large amounts of node to Freenet? From toad at amphibian.dyndns.org Fri Sep 21 10:48:51 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Fri, 21 Sep 2007 11:48:51 +0100 Subject: [Tech] Swapping and datastore In-Reply-To: References: <200709202007.51985.toad@amphibian.dyndns.org> Message-ID: <200709211148.57237.toad@amphibian.dyndns.org> On Friday 21 September 2007 11:17, Jusa Saari wrote: > On Thu, 20 Sep 2007 20:07:44 +0100, Matthew Toseland wrote: > > > On Thursday 20 September 2007 17:37, Jusa Saari wrote: > >> What happens to the datastore when a node swaps its position? From what > >> I've understood, the requests are routed based on node locations, which > >> causes the contents of a node's datastore to reflect its position; if a > >> node changes its position to the other side of the position ring, it > >> will receive very few requests relevant to its datastore, which means > >> that the store will effectively get wiped clean every time the node > >> swaps. > >> > >> I tried to look at freenetproject.org for this, but the documentation > >> there (http://freenetproject.org/understand.html) seems to be describing > >> the routing algorithm which predated NGRouting. > >> > >> So, given this, would it make sense to simply delete from the store any > >> keys not near the current location after a swap? They can't be accessed > >> anyway except by pure luck, and will simply waste store space which > >> could otherwise be used to fill the store up with relevant keys. > > > > It's not necessary. They will be automatically deleted when new keys are > > added, in accordance with the Least Recently Used cache replacement > > algorithm. Until new data comes in we may as well keep the old data. > > Hmm... In that case, perhaps it would be worth it to remember recent > swaps? Because if we get a request relevant to our new position, the node > which last occupied it could still have the key in its datastore. > > But then again, this would mess with a pure LRU cache replacement > algorithm, neccessiating adding the closeness of the key to our current > position as a factor. Argh :(. Not necessarily - we could remember who we've recently swapped with and ask them for the keys as a last resort, if they're closer to the key than we are. > > > Also I'd expect *generally* that long range swaps occur mostly towards the > > beginning of a node's lifetime, rather than later on. > > In a stable network with long-living nodes, yes. The few times I tried to > get into the "darknet", I lost all my contacts (from #freenet-refs or > whatever it was called) in a week or so. This would likely be true of > opennet as well; in fact I'd say that at any time, most freenet nodes have > only been in the network for a short time. #freenet-refs sucks, this we know. :( Opennet is likely to have high join/leave churn just as #freenet-refs does, and for the same reason - lots of people join, find they don't like it, and leave. And yes, this is likely to mess with routing, especially during a slashdotting. > > This could potentially mean that, since those short-lived nodes are unable > to do anything very useful, the semi-permanent network nodes which form > the "core" of Freenet will end up serving most requests, and therefore get > overloaded easily, while the nodes not directly connected to them won't be > able to find anything. Also true. Hopefully announcements will do something to limit the damage. It might be worth considering increasing the opennet peers limit, to get more connections in case many of them are unstable, and to speed up routing generally (fewer hops) ... otoh, some attacks are easier, and very few pure darknet nodes will have 30+ connections... > > Is there a way to simulate the effect of constantly adding and shortly > afterwards removing large amounts of node to Freenet? It's been done. By the gnunet folk who wrote a paper on it, and by vive. More simulations are however always welcome. The conclusion was that it causes node locations to cluster together in a really unhelpful way (you can also achieve this by a deliberate attack); randomly resetting locations from time to time seems to resist this. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070921/fb1ef51f/attachment.pgp From vive at freenetproject.org Fri Sep 21 10:58:35 2007 From: vive at freenetproject.org (vive) Date: Fri, 21 Sep 2007 12:58:35 +0200 Subject: [Tech] Swapping and datastore In-Reply-To: References: <200709202007.51985.toad@amphibian.dyndns.org> Message-ID: <20070921105835.GA52918@tim.hack.org> On Fri, Sep 21, 2007 at 01:17:32PM +0300, Jusa Saari wrote: > On Thu, 20 Sep 2007 20:07:44 +0100, Matthew Toseland wrote: > > Also I'd expect *generally* that long range swaps occur mostly towards the > > beginning of a node's lifetime, rather than later on. > > In a stable network with long-living nodes, yes. The few times I tried to > get into the "darknet", I lost all my contacts (from #freenet-refs or > whatever it was called) in a week or so. This would likely be true of > opennet as well; in fact I'd say that at any time, most freenet nodes have > only been in the network for a short time. > > This could potentially mean that, since those short-lived nodes are unable > to do anything very useful, the semi-permanent network nodes which form > the "core" of Freenet will end up serving most requests, and therefore get > overloaded easily, while the nodes not directly connected to them won't be > able to find anything. Right, what is needed is a way for nodes to hang on to the network even if they have only been exposed to the network once. Perhaps this does not have to be a guarantee, but something that will work with high probability. My naive(?) proposal has been to remember a set of opennet nodes seen (on the order of 10-100) and attempt to get back into the network with these if they have a free slot available. I think others want to make in some sophisticatd way that I have not fully understood. > Is there a way to simulate the effect of constantly adding and shortly > afterwards removing large amounts of node to Freenet? To simulate the actual freenet topology is hard, noone knows exactly how it grows. But there are ideal models (edges added as in the Kleinberg model). You can add and remove nodes in that topology. The problem with simulations is that they typically either say something about a very good topology, or something about when its rather rather poor for what we can expect with scaling. /Vilhelm -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070921/d992ed44/attachment.pgp From edt at aei.ca Fri Sep 21 11:38:31 2007 From: edt at aei.ca (Ed Tomlinson) Date: Fri, 21 Sep 2007 07:38:31 -0400 Subject: [Tech] Swapping and datastore In-Reply-To: <200709202007.51985.toad@amphibian.dyndns.org> References: <200709202007.51985.toad@amphibian.dyndns.org> Message-ID: <200709210738.31647.edt@aei.ca> On September 20, 2007, Matthew Toseland wrote: > On Thursday 20 September 2007 17:37, Jusa Saari wrote: > > What happens to the datastore when a node swaps its position? From what > > I've understood, the requests are routed based on node locations, which > > causes the contents of a node's datastore to reflect its position; if a > > node changes its position to the other side of the position ring, it will > > receive very few requests relevant to its datastore, which means that the > > store will effectively get wiped clean every time the node swaps. > > > > I tried to look at freenetproject.org for this, but the documentation > > there (http://freenetproject.org/understand.html) seems to be describing > > the routing algorithm which predated NGRouting. > > > > So, given this, would it make sense to simply delete from the store any > > keys not near the current location after a swap? They can't be accessed > > anyway except by pure luck, and will simply waste store space which could > > otherwise be used to fill the store up with relevant keys. > > It's not necessary. They will be automatically deleted when new keys are > added, in accordance with the Least Recently Used cache replacement > algorithm. Until new data comes in we may as well keep the old data. > > Also I'd expect *generally* that long range swaps occur mostly towards the > beginning of a node's lifetime, rather than later on. I am not at all sure of that. I know that is not the case here. I have a node with dynamic address. My typical pattern is get ref to get myself 10 or more 'friends', wait 2 weeks by which time I will be back down to 5 nodes or so. When I add new 'friends' my nodes location shifts wildly... I have a 4G datastore. Question is, is this a common pattern? If it is it could easily lead to low hit ratios. I have always thought that the average location of the keys in the DS should influence the swap algorythm. Ed Tomlinson From toad at amphibian.dyndns.org Fri Sep 21 12:27:31 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Fri, 21 Sep 2007 13:27:31 +0100 Subject: [Tech] Swapping and datastore In-Reply-To: <20070921105835.GA52918@tim.hack.org> References: <20070921105835.GA52918@tim.hack.org> Message-ID: <200709211327.36790.toad@amphibian.dyndns.org> On Friday 21 September 2007 11:58, you wrote: > On Fri, Sep 21, 2007 at 01:17:32PM +0300, Jusa Saari wrote: > > On Thu, 20 Sep 2007 20:07:44 +0100, Matthew Toseland wrote: > > > Also I'd expect *generally* that long range swaps occur mostly towards the > > > beginning of a node's lifetime, rather than later on. > > > > In a stable network with long-living nodes, yes. The few times I tried to > > get into the "darknet", I lost all my contacts (from #freenet-refs or > > whatever it was called) in a week or so. This would likely be true of > > opennet as well; in fact I'd say that at any time, most freenet nodes have > > only been in the network for a short time. > > > > This could potentially mean that, since those short-lived nodes are unable > > to do anything very useful, the semi-permanent network nodes which form > > the "core" of Freenet will end up serving most requests, and therefore get > > overloaded easily, while the nodes not directly connected to them won't be > > able to find anything. > > Right, what is needed is a way for nodes to hang on to the network even if > they have only been exposed to the network once. Perhaps this does not have > to be a guarantee, but something that will work with high probability. My > naive(?) proposal has been to remember a set of opennet nodes seen (on the > order of 10-100) and attempt to get back into the network with these if they > have a free slot available. I think others want to make in some sophisticatd > way that I have not fully understood. Well, we have the opennet reconnecting problem, and we have the universal reconnecting problem... The first is that opennet auto-dumps disconnected peers after 5 minutes if there are nodes being offered. The second is that the nodes you were connected to probably aren't on the network anymore anyway because they were newbies and they got tired of it. :( > > > Is there a way to simulate the effect of constantly adding and shortly > > afterwards removing large amounts of node to Freenet? > > To simulate the actual freenet topology is hard, noone knows exactly how > it grows. But there are ideal models (edges added as in the Kleinberg > model). You can add and remove nodes in that topology. The problem with > simulations is that they typically either say something about a very good > topology, or something about when its rather rather poor for what we can > expect with scaling. > > /Vilhelm -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070921/5a12e34e/attachment.pgp From m.rogers at cs.ucl.ac.uk Fri Sep 21 13:50:28 2007 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Fri, 21 Sep 2007 14:50:28 +0100 Subject: [Tech] Swapping and datastore In-Reply-To: <200709211327.36790.toad@amphibian.dyndns.org> References: <20070921105835.GA52918@tim.hack.org> <200709211327.36790.toad@amphibian.dyndns.org> Message-ID: <46F3CC24.1070607@cs.ucl.ac.uk> Matthew Toseland wrote: > Well, we have the opennet reconnecting problem, and we have the universal > reconnecting problem... The first is that opennet auto-dumps disconnected > peers after 5 minutes if there are nodes being offered. The second is that > the nodes you were connected to probably aren't on the network anymore anyway > because they were newbies and they got tired of it. :( Gnutella uses a cache of 10,000 known peers to deal with the latter problem, and can still take up to half an hour to find a live peer if the cache is a few days old. At what point does address caching become address harvesting? :-/ Cheers, Michael From bruthoma at student.ethz.ch Fri Sep 21 19:33:04 2007 From: bruthoma at student.ethz.ch (Thomas Bruderer) Date: Fri, 21 Sep 2007 19:33:04 +0000 (UTC) Subject: [Tech] Theoretical Master Thesis on sybil attacks on freenet References: <200709210114.00428.toad@amphibian.dyndns.org> Message-ID: 12345678901234567890123456789012345678901234567890123456789012345678901234567890 I'll go through your thoughts point by point. > ' The calculation of the routing table has changed several times, two of > them were major changes in the protocol. After the initial Paper the first > big change was called the "Next Generation Routing" [7] and the last big step > was the introduction of a small world topology. [1] This newest attempt was > the creation of a p2p network which relies on personal contacts of people.' > > Not true. We always relied on a small world topology: The network is only > navigable if it is a small world network. However, the original Freenet > concept was purely opennet, and therefore relied to a very large degree > on "destination sampling" (connect to the data source when you get a > successful request). We understand this rather better now than we did thanks > to Oskar's recent work. OK, good to know. Then only the routing table calcualtion has changed, which is also a change in routing. :) And yes the work of Oskar is a real big step forward. Thats a very good base now, where the original paper had no theoretical base at all. This will definitly help, also to prove more, which I tried, but I was limited to very easy cases. > > With regards to random routing for a few steps ... unfortunately it doesn't > solve the problem (it's been proposed lots of times of course): > - You can only do it for a few hops before getting into real routing. > Therefore your anonymity set is only those nodes which could have random > routed to you. And it can be a fairly uneven set. > - Much worse: If two requests can be correlated by e.g. both being part of the > same splitfile, both being post inserts from the same frost ID etc ... then > you can do *nasty* statistical attacks based on the proportion of that known > quantity that are routed through the attacker's node. I definitly wouldn't support a random routing, but in a ring It was very easy to proof that information theoretical anonymity. > > Re section 4.2: > - Are the "1" occurrences leaf nodes ? > - What do the "2" and "3"'s look like topologically speaking? I havent really looked into detail, but the way I counstructed it, there are no nodes with low degrees at all. This was a highly connected graph. So to speak I can generally say nothing about this structures, but I might look into it, the simulation was very simple and such ideas can be checked fast. However I think it will be difficult to see similarites between the cases. I node which sends two messages which are received by two sibyl nodes (not necessarily different one) could be pinpointed basically to the intersection of those two source sets. This can be near one or the other node, or far away. > > What exactly sucks about the tree? It would be helpful to understand this so > we can advise people not to abuse darknet by setting up such structures (e.g. > mirroring external authority structures). Clearly if you get a request from > below you in the tree, you can identify that the node is within that > subtree - is that the totality of the dangers of a tree topology? A tree structure does not suck, in fact it does prevent the correlation attacks to do any more harm than a single message. Beside the point you mentioned I dont see any problem for tree structures. (well beside the other obvious fact that it happens easily that a tree gets disconnected) > > Theorem 7: Cool! On the circle, if the source is just below the left attacker, > the left attacker will get requests which get very close to the exact > opposite of the request sender's own location. And you have the opposite with > the right hand attacker. The more blocks, the closer the requests will get to > the exact opposite point of the sender on the ring, and the more accurately > you can identify the source... > > Basically the operation is this (assuming pure greedy routing and no overload > etc): > We have a request for location T. Our location is L. We want to find S, the > location of the source node. We can reasonably conclude that S is between T' > (exact opposite of T) and L (our location), within the shortest arc linking > the two points. If there are enough blocks, enough different T's, the point > at which we stop getting requests tells us where the source is on the circle. > > However: > - Routing may not be purely greedy, because of e.g. backoff. yes you are right and all randomization effects are not accounted in this paper, the simulation did also a (perfect) shortest path routing and additionally the complete routing table was known by the attacker. well in the ring its obvious that you know the routing table, and this abstraction was used for general graphs aswell. > - The topology may not be close enough to a ring for this to work anyway. Of > course there's a question as to whether the network will work at all in that > case. I did it on general graphs, and Its worse for them since I cannot only pinpoint two directions. Lets say I have 10 edges I can at least divide the complete network in 10 parts (possibly equally in size) also the source of the message can send into 10 directions, and only a few of those starting directions will go through my sibyl nodes. For single messages this gives not much. I done a step by step analysis. The first message which was captured normally said nothing more than Its in a group of 150 nodes. After another node captured another message from this same file, often it reduced the setsize significantly since the routing for another message was so completly different that the intersection of those two sets gave a much smaller set normally in the region of 30 - 60 nodes. > - Routing may be deliberately obscured by e.g. a few random hops from time to > time. yes. > - Routing will not always stop when it reaches the closest node to the target. > Probe requests do, but real requests and inserts have an HTL limit, currently > continuing for 10 hops after the last improvement in the closeness to the > target. longer routes doesnt effect the mesurement at all, I throw the messages away (in my simulation) after the first sibyl node captured a message. If you do longer routes, it only affects it that potential sibyl node (which wasnt in range in the short route) can see the message, and if the routing is known it will give you still the source set, which will be bigger indeed. In reality since there are random effects the attack should work with a set of probabilities of those possible source nodes, probably throw away nodes with less than 1%, and in the end a case I referenced to a setsize of 1 will look in such a case like -> (88% 5% 4% 1% 1% 1% [likelyhood to be source] ) > - If the source is a long way away, we will only receive a small fraction of > the requests it sends. How much does this limit accuracy? If you're too far > away you may not get any requests, or not enough to connect them together. The likelihood is smaller, and if you only get one message it is a big set ( at least number of hops ) but if you get two differently routed message from a far node, it will be identified obviously easily, since the intersection of such sets are probably small. However this is only a feeling. And random effects do affect long routes more than small ones, so in reality your assertion is probably true. In a a perfect routing I would say the accurcy would be increased. > - Finding the target's rough location doesn't instantly find the target. An > attacker would probably try to narrow down the search by connecting to nodes > closer to the location in question. yes. And remember I have global knowledge. So I can pinpoint also nodes which I am not directly connceted. but If I could say: between Adresse 0.73435445 and 0.73437930 is a node which sent file A. Now what? This doesnt even give me the IP! So I need to go near this node in any case. > - It doesn't work on popular files which multiple requestors may be fetching > simultaneously short routes mean only nearby sibyl nodes get information. But it still works since the routing will be the same, but it will stop earlier, therefore less information. Well this are mostly my thoughs not based by much facts, but Its based on what I have seen and searched for. > > It would be fascinating to do a more accurate attack simulation! How much > information can be gained about a requestor, and to what degree is this > constrained by the above factors? (Introducing them one at a time, of > course...) > > Another attack you can do, if you are close to the target, is take into > account the exact local topology and use the proportion of requests coming to > your node to identify which node it is. > > "Beware of high degree" is an interesting message. I'm not quite sure how you > support it. Generally we've assumed that reasonably high degree is a good > thing - certainly it will *significantly* reduce the number of hops needed to > find data, and it's quite helpful in getting small-world-like networks... > (Too few connections and you probably won't have the occasional long range > one). Are you saying high average degree is a problem, or that individual > nodes with degree far above the average (aka ubernodes) are a problem? > well its based on some observations: one-connected-topologies dont give away more information even if multiple messages are dependant two-conncted-topologies give already away more information and the three-conncted graph I simulated was very intersting since correlation was very intersting (if a message was received) and if I think further, a full graph is practically a p2p network without anonymisation, since you request the most likely one directly. I think in a network with higher degree it will be easier to do correlation upon this facts. But there are no hard proves. To theoretically analyze a three connected graph would be so interesting, but so difficult, I dont even have an idea how to do it. There is a second effect: higher degrees means the routes get shorter. If you have not many sibyl nodes, short routes will mean you dont intercept many. But if you intercept one, the anonymity set is very small. (thats the old good routing versus anonymity problem) ubernodes are more attackable in this sense, since they can route to so many nodes, they will have shorter routes and the set sizes drop, and even if no sibyl node is near. two intercepted messags will reveal quite a lot about such big nodes.. Its not an average problem, well maybe too, but its also a problem for the individual. But these are the effects which I havent understood completly. But probably the most intersting ones. > As far as dependant vs independant messages go ... we have a few options: > - Bundle all dependant messages into huge files and transfer them all at once > in a single request. This isn't very practical: transferring a 1GB file with > no multi-sourcing is going to take roughly forever. Having variable block > sizes up to some large limit might help a bit, but would suck for various > network/practical/code reasons, as it did on 0.5, and it wouldn't solve the > problem, only make correlation attacks a bit harder. > - Bundle dependant requests together, and route them as a blob, keeping them > bundled for a number of random hops. This has performance issues, relies on > tunnels which may break if any node goes offline, doesn't work well with long > lived requests ... > - Premix routing. Establish an explicit anonymity set of most of the nodes > within some number of hops, and use the darknet to work out which nodes > are "real" versus which are bogus Sybil's, or try to plug in a network-wide > mixnet layer like I2P for opennet. > - Just live with it: you can identify your neighbours' traffic. This is > plainly unacceptable. The last one isnt as unaccaptle in the friends network, but its certainly a big problem for an opennet. And yes it is very difficult. What I learnt from the thesis is definitly that anonymity is very relative, and its more difficult than it often looks at first sight. It was interesting to hunt some of those questions down, especially there are not always such intuitive. Sorry for the lengthy answer, but I had to describe the answers in detail because some of those details are not mentioned at all in the paper, but there is a good reason for it, because its not well understood from me. I could've answered I dont know on some of them, but I think you are also interested in the ideas which are not fully understood, and might be elaborated in the future. Thanks for your questions, it was a very good input, and I hope it gives me the motivation to do really redo the paper. I'll meet my prof this month, so he can tell me what is that worse on the paper (not the ideas, but the writing) that I can improve it a lot. Greets Thomas From jUrner at arcor.de Thu Sep 27 11:16:16 2007 From: jUrner at arcor.de (Juergen Urner) Date: Thu, 27 Sep 2007 13:16:16 +0200 Subject: [Tech] ClientGet questions Message-ID: <46FB9100.5040909@arcor.de> Hello, I am playing a bit around with the client protocol (2.0) currently and a have few questions. One is regarding 'ClientGet'. To download a file to disk I have to pass the filepath and its size in the request. This is leaving me with one or the other headscratch. How am I supposed to know in advance what type of file I am requesting and what its size is? One of the (bit hackish) solutions I came up with sending a 'ClientGet' with a size of -1 or 0, hoping the request fails, to get the missing peaces from 'GetFailed' (code 21), but have no idea if this is intendet use or how reliable it is. Another headscratch is: how am I supposed to know in advance if a filepath I pass today can be written tommorow? In the meantime some other file with the same name may find its way into the folder, the folder might disappear... or whatever. So I wonder why the filepath (and TestDDA) is not requested somewhere near 'DataFound', when it is actually needed. Anyone any hints? Thanks, Juergen From toad at amphibian.dyndns.org Sat Sep 29 11:21:42 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Sat, 29 Sep 2007 12:21:42 +0100 Subject: [Tech] ClientGet questions In-Reply-To: <46FB9100.5040909@arcor.de> References: <46FB9100.5040909@arcor.de> Message-ID: <200709291221.48194.toad@amphibian.dyndns.org> On Thursday 27 September 2007 12:16, you wrote: > Hello, > > I am playing a bit around with the client protocol (2.0) > currently and a have few questions. > > One is regarding 'ClientGet'. To download a file to disk I have > to pass the filepath and its size in the request. This is leaving me > with one or the other headscratch. How am I supposed to know > in advance what type of file I am requesting and what its size > is? You only have to pass a maximum size. If the file is bigger than this then the node will fail the download. There is no reason to pass a maximum size if you want to download the file regardless of how big it is. You do need to pass the path to save it to, if you want to save it directly to disk. You can use ReturnType=direct to store it in temporary space inside the node and then stream it back over the FCP connection, but this is much less efficient on disk space for large downloads as we end up storing several copies at once. It is however simpler for small fetches. > > One of the (bit hackish) solutions I came up with sending a 'ClientGet' > with a size of -1 or 0, hoping the request fails, to get the missing > peaces from > 'GetFailed' (code 21), but have no idea if this is intendet use or how > reliable it is. 0 is too small. If you're just interested in the mime type, try something a bit bigger - say 32k (the basic units of size in freenet are 1k for SSKs and 32k for CHKs). > > Another headscratch is: how am I supposed to know in advance if a filepath > I pass today can be written tommorow? In the meantime some other file > with the same > name may find its way into the folder, the folder might disappear... or > whatever. > So I wonder why the filepath (and TestDDA) is not requested somewhere > near 'DataFound', > when it is actually needed. To save disk space. Downloading to temporary space is *really* expensive on disk space: we have to store a separate copy of the entire file, which may be huge, then stream it across, then the client writes its own copy, then we delete our copy. On top of all the temporary files we use assembling it. > > Anyone any hints? > > > > Thanks, Juergen -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070929/3dca35c9/attachment.pgp From hallc at lu.unisi.ch Sat Sep 29 13:07:12 2007 From: hallc at lu.unisi.ch (Cyrus Hall) Date: Sat, 29 Sep 2007 15:07:12 +0200 Subject: [Tech] Topologies of Freenet Message-ID: <1191071232.6058.15.camel@boulder> Ciao- I'm working on a study of node reachablity and mixing dynamics for random walks. To that end, I'm collecting both topologies from live systems and various project simulators which build topologies in a realistic manner. I'd like to include Freenet in the study, but I've ran into a bit of a snag. I've been looking around for a simulator of Freenet without success. There is the old www.neurogrid.net simulator, but it appears extremely out of date, particularly when considering the architectural changes to Freenet since 2003. Similarly, there are Hui Zhang's http://netweb.usc.edu/~huizhang/freenet.html (old, code doesn't inspire confidence), and EOS http://www.skoghs.se/eos/ (old, but the code is promising). In the dual thesis of Jonas Haeggstr?m and Hans-Emil Skogh, the same fellows who wrote EOS, they mention two simulators borne out of the Freenet project itself, Aurora and Serapis. Neither of them appear to be in the current SVN tree, and I can't find the old CVS tree except via Sourceforge's Krugle code search. And what I find there has a copyright of 2001... Is there something more up to date than all this? In particular, is there a simulator that accurately builds topologies that mimic the actual Freenet deployment? There also appears to have been recent discussion on adding more instrumentation to the running Freenet network in order to develop a deeper understanding of the live topology. Has anything come of that? Thanks, Cyrus Hall University of Lugano Department of Informatics From jUrner at arcor.de Sat Sep 29 13:37:10 2007 From: jUrner at arcor.de (Juergen Urner) Date: Sat, 29 Sep 2007 15:37:10 +0200 Subject: [Tech] ClientGet questions In-Reply-To: <200709291221.48194.toad@amphibian.dyndns.org> References: <46FB9100.5040909@arcor.de> <200709291221.48194.toad@amphibian.dyndns.org> Message-ID: <46FE5506.9040708@arcor.de> Matthew Toseland wrote: > On Thursday 27 September 2007 12:16, you wrote: > >> Hello, >> >> I am playing a bit around with the client protocol (2.0) >> currently and a have few questions. >> >> One is regarding 'ClientGet'. To download a file to disk I have >> to pass the filepath and its size in the request. This is leaving me >> with one or the other headscratch. How am I supposed to know >> in advance what type of file I am requesting and what its size >> is? >> > > You only have to pass a maximum size. If the file is bigger than this then the > node will fail the download. There is no reason to pass a maximum size if you > want to download the file regardless of how big it is. > ok > You do need to pass the path to save it to, if you want to save it directly to > disk. You can use ReturnType=direct to store it in temporary space inside the > node and then stream it back over the FCP connection, but this is much less > efficient on disk space for large downloads as we end up storing several > copies at once. It is however simpler for small fetches. > >> One of the (bit hackish) solutions I came up with sending a 'ClientGet' >> with a size of -1 or 0, hoping the request fails, to get the missing >> peaces from >> 'GetFailed' (code 21), but have no idea if this is intendet use or how >> reliable it is. >> > > 0 is too small. If you're just interested in the mime type, try something a > bit bigger - say 32k (the basic units of size in freenet are 1k for SSKs and > 32k for CHKs). > ok >> Another headscratch is: how am I supposed to know in advance if a filepath >> I pass today can be written tommorow? In the meantime some other file >> with the same >> name may find its way into the folder, the folder might disappear... or >> whatever. >> So I wonder why the filepath (and TestDDA) is not requested somewhere >> near 'DataFound', >> when it is actually needed. >> > > To save disk space. Downloading to temporary space is *really* expensive on > disk space: we have to store a separate copy of the entire file, which may be > huge, then stream it across, then the client writes its own copy, then we > delete our copy. On top of all the temporary files we use assembling it. > ok, I see. No other way to handle this. When I am at it... if you don't mind. A question regarding TestDDARequest. Somehow I am worried that tested directories might pile up in memory (could potentially be a huge number over time running) and was hoping a TestDDARequest with 'WantReadDirectory=false' and 'WantWriteDirectory=false' could cause the node to forget a directory and free resources. How is this handled? Thanks, Juergen From m.rogers at cs.ucl.ac.uk Sat Sep 29 16:08:22 2007 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Sat, 29 Sep 2007 17:08:22 +0100 Subject: [Tech] Topologies of Freenet In-Reply-To: <1191071232.6058.15.camel@boulder> References: <1191071232.6058.15.camel@boulder> Message-ID: <46FE7876.4070707@cs.ucl.ac.uk> Hi Cyrus, I worked on a Freenet simulator last summer, it's in the SVN repository: http://freenet.googlecode.com/svn/trunk/apps/load-balancing-sims/phase7/ Vive has been working on it this summer I believe. I can't speak for Vive's changes, but my version was developed before the deployment of opennet, so it assumes a darknet topology based on the social connections between users. I only simulated idealised social networks based on Kleinberg's small world model, so the simulator won't tell you much about the mixing properties of the real network. Simulating the current mixture of opennet (automatic topology construction) and darknet (manual topology construction) would be tricky to say the least... one of the factors behind the deployment of opennet was that the developers suspected the darknet topology was far from ideal, but they couldn't really measure it. Oskar Sandberg will be able to tell you about the mixing properties of the Kleinberg model - I assume it's fast mixing since swap requests only travel a few hops - and the Sybilguard paper has some references showing that real social networks are also fast mixing IIRC. But I don't know if that necessarily implies that an arbitrary mixture of the two is fast mixing! Cheers, Michael