From m.rogers at cs.ucl.ac.uk Thu Feb 1 21:28:37 2007 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Thu, 01 Feb 2007 21:28:37 +0000 Subject: [Tech] PageRank for premix routing? Message-ID: <45C25B85.2040201@cs.ucl.ac.uk> Matteo Dell'Amico (http://www.disi.unige.it/person/DellamicoM/) gave an interesting talk at UCL a few months ago, and I thought his work on neighbourhood maps might be relevant to Freenet. Neighbourhood maps start from the observation that social networks are fast mixing, meaning that if you take a random walk from any node, the probability of ending up at a given node quickly becomes independent of your starting point. That means a local view of the network is in some sense typical of the network as a whole. And that, in turn, means you can run algorithms like PageRank over a local view of the network, and get results that are similar to the results you'd get from a global view of the network. Matteo uses a local view of the network to compute subjective trust rankings: each node broadcasts its list of neighbours for a small number of hops, uses the received broadcasts to build a neighbourhood map, and runs PageRank over the map using itself as the source. Assuming that each link in the network represents a trust relationship, the resulting rankings give subjective trust values to the nodes. PageRank works by diffusing some property (trust in this case) through a network, starting from a source. In each round, the source produces one unit of trust and shares it across its outgoing edges. The trust arriving on each node's incoming edges is summed, reduced by a damping factor, and shared across its outgoing edges. A node's rank is the amount of trust it receives per round when the algorithm reaches a steady state. PageRank has two nice properties when used as a trust metric: you can't amplify your trust by creating lots of outgoing edges, and you can't 'trap' trust in a cluster of nodes that link to one another. Why is this relevant to Freenet? Premix routing is supposed to prevent statistical attacks where a node sees a series of related messages (eg requests for parts of the same file) and concludes that the sender of the messages must be nearby. In premix routing, the sender builds a tunnel to a random node and sends all the related messages through the tunnel. But how do you select a random node? Even in a darknet, I can create a hundred Sybil nodes and link them all to my 'real' node, without having to persuade anyone to trust them. A sender who chooses randomly from a list of nearby nodes will be likely to select one of my Sybil nodes, and a random walk will also be likely to end up on a Sybil node. But if the sender builds a neighbourhood map and uses PageRank to choose the probability of selecting each node, I gain no advantage by creating a large number of nodes. This wouldn't work on an opennet, however... Cheers, Michael From toad at amphibian.dyndns.org Mon Feb 5 12:12:26 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Mon, 5 Feb 2007 12:12:26 +0000 Subject: [Tech] Immediate priorities Message-ID: <20070205121226.GA31958@amphibian.dyndns.org> At the moment the priority is the CPU/memory performance issues which have been reported fairly widely. A big part of this comes down to object churn, and I have made significant progress on this on Saturday. But I seem to have broken something fairly major in the process. I will therefore fix this, and do some more work on this. Testers would be helpful; I know there are issues with the build I committed on Saturday (1ms ping time, backoff, ...), but even there I'd like to know more about how it fails. Once this is sorted I will put out a new build to the auto-update (there are reports of this not working, but I think that's just due to 1010 USK breakage). bback seems to have taken on the FCP changes that need to be done, so that leaves me free to get on with the remaining major issues: - Aftermath of 1010: Auto-update keys, ARK keys etc will need to be changed; if I have any peers still on 1009, I need to start up a node and insert the auto-update to them (although the network may be too fragmented for this to be of any use, it's worth trying). - Backoff / load limiting. Once the current issues have been sorted hopefully this will improve. - Location clustering / routing issues. May very well be caused by lack of opennet, but further investigation is required. - Opennet. PS Some people have complained about my absence during most of January. A big chunk of this was (unpaid) sick-leave; the rest was (also unpaid) holiday time. I'm back now! -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070205/65396cbd/attachment.pgp From toad at amphibian.dyndns.org Mon Feb 5 20:07:09 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Mon, 5 Feb 2007 20:07:09 +0000 Subject: [Tech] PageRank for premix routing? In-Reply-To: <45C25B85.2040201@cs.ucl.ac.uk> References: <45C25B85.2040201@cs.ucl.ac.uk> Message-ID: <20070205200709.GA11092@amphibian.dyndns.org> On Thu, Feb 01, 2007 at 09:28:37PM +0000, Michael Rogers wrote: > Matteo Dell'Amico (http://www.disi.unige.it/person/DellamicoM/) gave an > interesting talk at UCL a few months ago, and I thought his work on > neighbourhood maps might be relevant to Freenet. > > Neighbourhood maps start from the observation that social networks are > fast mixing, meaning that if you take a random walk from any node, the > probability of ending up at a given node quickly becomes independent of > your starting point. That means a local view of the network is in some > sense typical of the network as a whole. And that, in turn, means you > can run algorithms like PageRank over a local view of the network, and > get results that are similar to the results you'd get from a global view > of the network. PageRank is patented, and picking a fight with Google is a bad idea especially as they gave us 4 summer coders and $2000 last year. > > Matteo uses a local view of the network to compute subjective trust > rankings: each node broadcasts its list of neighbours for a small number > of hops, uses the received broadcasts to build a neighbourhood map, and > runs PageRank over the map using itself as the source. Assuming that > each link in the network represents a trust relationship, the resulting > rankings give subjective trust values to the nodes. I proposed a trust metric a month or two ago. Some sort of mechanism for using the topology to generate a trust metric for nearby nodes is going to be vital. Something which there is literature on would be great, but would almost certainly be patented. Having said that, something we make up is still likely to be covered by some patent somewhere - but at least you don't have the triple damages for using PageRank (FPI is based in California, remember?). Ian has said that we shouldn't pre-emptively not use published algorithms just because they might be patented - but using an algorithm which we know for certain to be patented, and which we did not create, is something else! > > PageRank works by diffusing some property (trust in this case) through a > network, starting from a source. In each round, the source produces one > unit of trust and shares it across its outgoing edges. The trust > arriving on each node's incoming edges is summed, reduced by a damping > factor, and shared across its outgoing edges. A node's rank is the > amount of trust it receives per round when the algorithm reaches > a steady state. Similar to my proposal. > > PageRank has two nice properties when used as a trust metric: you can't > amplify your trust by creating lots of outgoing edges, and you can't > 'trap' trust in a cluster of nodes that link to one another. Both of which we want. > > Why is this relevant to Freenet? Premix routing is supposed to prevent > statistical attacks where a node sees a series of related messages (eg > requests for parts of the same file) and concludes that the sender of > the messages must be nearby. In premix routing, the sender builds a > tunnel to a random node and sends all the related messages through the > tunnel. But how do you select a random node? Even in a darknet, I can > create a hundred Sybil nodes and link them all to my 'real' node, > without having to persuade anyone to trust them. A sender who chooses > randomly from a list of nearby nodes will be likely to select one of my > Sybil nodes, and a random walk will also be likely to end up on a Sybil > node. But if the sender builds a neighbourhood map and uses PageRank to > choose the probability of selecting each node, I gain no advantage by > creating a large number of nodes. Right. > > This wouldn't work on an opennet, however... Opennet would require a totally different premix routing algorithm, probably something like I2P where you choose random nodes from the entire network and try to disguise the tunnel creation. I don't know whether it is worth trying to implement such a thing, as opennet is not viable in hostile environments and such premix could be compromized through Sybil attacks, and it would be significant extra work. However I do worry that once we have opennet nobody will use darknet. :| > > Cheers, > Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070205/f745b974/attachment.pgp From m.rogers at cs.ucl.ac.uk Tue Feb 6 09:26:35 2007 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Tue, 06 Feb 2007 09:26:35 +0000 Subject: [Tech] PageRank for premix routing? In-Reply-To: <20070205200709.GA11092@amphibian.dyndns.org> References: <45C25B85.2040201@cs.ucl.ac.uk> <20070205200709.GA11092@amphibian.dyndns.org> Message-ID: <45C849CB.3080400@cs.ucl.ac.uk> Matthew Toseland wrote: > PageRank is patented, and picking a fight with Google is a bad idea > especially as they gave us 4 summer coders and $2000 last year. Sorry, did I say PageRank? I meant eigenvector centrality. :-) See the related work section of this paper by Jon Kleinberg (who else?) - it turns out people have been calculating eigenvector centrality in social networks since at least the 70s and possibly the 50s: http://www.cs.cornell.edu/home/kleinber/auth.pdf I agree that it would be stupid (and ungrateful) to antagonise Google, but there seems to be plenty of prior art for using eigenvector centrality as a trust metric. I'll try to chase up the Katz, Hubbell and Pinski-Narin refs tonight. > However I do worry that once we have opennet nobody will use darknet. :| Agreed. Cheers, Michael From m.rogers at cs.ucl.ac.uk Wed Feb 7 13:30:59 2007 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Wed, 07 Feb 2007 13:30:59 +0000 Subject: [Tech] PageRank for premix routing? In-Reply-To: <45C849CB.3080400@cs.ucl.ac.uk> References: <45C25B85.2040201@cs.ucl.ac.uk> <20070205200709.GA11092@amphibian.dyndns.org> <45C849CB.3080400@cs.ucl.ac.uk> Message-ID: <45C9D493.3050406@cs.ucl.ac.uk> So I followed up a few of the refs from the Kleinberg paper... Katz came up with the idea of propagating trust/status/etc across a graph in 1953. In 1976, Pinski and Narin suggested normalising each node's output by dividing the output on each outgoing edge by the node's outdegree. Geller pointed out in 1978 that their model was equivalent to a Markov chain: the scores assigned to the nodes followed the Markov chain's stationary distribution. In other words, propagating trust/status/etc with normalisation at each node is equivalent to taking random walks from random starting points and counting how many times you end up at each node. (So either my assertion that a random walk would be likely to end at a Sybil node was wrong, or trust propagation isn't Sybil-resistant.) The only difference between Geller's model and PageRank is the damping factor: in PageRank you continue your random walk with probability d or jump to a random node with probability 1-d. (Incidentally, when the algorithm's described this way rather than in terms of a transition matrix, it's easy to see how you could implement it on a web spider.) Conclusion: we should be safe from the PageRank patent as long as we don't use a damping factor - the rest of the algorithm was published (and applied to social networks) 20 years before the patent. Cheers, Michael From toad at amphibian.dyndns.org Wed Feb 7 23:31:42 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Wed, 7 Feb 2007 23:31:42 +0000 Subject: [Tech] PageRank for premix routing? In-Reply-To: <45C849CB.3080400@cs.ucl.ac.uk> References: <45C25B85.2040201@cs.ucl.ac.uk> <20070205200709.GA11092@amphibian.dyndns.org> <45C849CB.3080400@cs.ucl.ac.uk> Message-ID: <20070207233142.GA9284@amphibian.dyndns.org> On Tue, Feb 06, 2007 at 09:26:35AM +0000, Michael Rogers wrote: > Matthew Toseland wrote: > > PageRank is patented, and picking a fight with Google is a bad idea > > especially as they gave us 4 summer coders and $2000 last year. > > Sorry, did I say PageRank? I meant eigenvector centrality. :-) See the > related work section of this paper by Jon Kleinberg (who else?) - it > turns out people have been calculating eigenvector centrality in social > networks since at least the 70s and possibly the 50s: > > http://www.cs.cornell.edu/home/kleinber/auth.pdf > > I agree that it would be stupid (and ungrateful) to antagonise Google, > but there seems to be plenty of prior art for using eigenvector > centrality as a trust metric. I'll try to chase up the Katz, Hubbell and > Pinski-Narin refs tonight. Cool! > > > However I do worry that once we have opennet nobody will use darknet. :| > > Agreed. Which may be a reason for only implementing premix routing on darknet. If people want Tor they can use Tor. > > Cheers, > Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070207/1dd97f50/attachment.pgp From toad at amphibian.dyndns.org Wed Feb 7 23:32:47 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Wed, 7 Feb 2007 23:32:47 +0000 Subject: [Tech] PageRank for premix routing? In-Reply-To: <45C9D493.3050406@cs.ucl.ac.uk> References: <45C25B85.2040201@cs.ucl.ac.uk> <20070205200709.GA11092@amphibian.dyndns.org> <45C849CB.3080400@cs.ucl.ac.uk> <45C9D493.3050406@cs.ucl.ac.uk> Message-ID: <20070207233247.GB9284@amphibian.dyndns.org> Hmmm but does it work? Does it prevent Sybil? On Wed, Feb 07, 2007 at 01:30:59PM +0000, Michael Rogers wrote: > So I followed up a few of the refs from the Kleinberg paper... > > Katz came up with the idea of propagating trust/status/etc across a > graph in 1953. In 1976, Pinski and Narin suggested normalising each > node's output by dividing the output on each outgoing edge by the node's > outdegree. Geller pointed out in 1978 that their model was equivalent to > a Markov chain: the scores assigned to the nodes followed the Markov > chain's stationary distribution. In other words, propagating > trust/status/etc with normalisation at each node is equivalent to taking > random walks from random starting points and counting how many times you > end up at each node. (So either my assertion that a random walk would be > likely to end at a Sybil node was wrong, or trust propagation isn't > Sybil-resistant.) > > The only difference between Geller's model and PageRank is the damping > factor: in PageRank you continue your random walk with probability d or > jump to a random node with probability 1-d. (Incidentally, when the > algorithm's described this way rather than in terms of a transition > matrix, it's easy to see how you could implement it on a web spider.) > > Conclusion: we should be safe from the PageRank patent as long as we > don't use a damping factor - the rest of the algorithm was published > (and applied to social networks) 20 years before the patent. > > Cheers, > Michael > _______________________________________________ > Tech mailing list > Tech at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070207/23700eb6/attachment.pgp From m.rogers at cs.ucl.ac.uk Thu Feb 8 13:02:43 2007 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Thu, 08 Feb 2007 13:02:43 +0000 Subject: [Tech] PageRank for premix routing? In-Reply-To: <20070207233247.GB9284@amphibian.dyndns.org> References: <45C25B85.2040201@cs.ucl.ac.uk> <20070205200709.GA11092@amphibian.dyndns.org> <45C849CB.3080400@cs.ucl.ac.uk> <45C9D493.3050406@cs.ucl.ac.uk> <20070207233247.GB9284@amphibian.dyndns.org> Message-ID: <45CB1F73.3030403@cs.ucl.ac.uk> Matthew Toseland wrote: > Hmmm but does it work? Does it prevent Sybil? Good question... Matteo (who wrote the neighbourhood maps paper) pointed me towards two papers that use network flow algorithms to prevent Sybil attacks: http://p2pecon.berkeley.edu/pub/p243-EC04.pdf http://people.cornell.edu/pages/ejf27/pfiles/sybils.pdf Sounds like it might be similar to the approach used by Advogato: http://www.advogato.org/trust-metric.html Cheers, Michael From bbackde at googlemail.com Sat Feb 10 18:09:58 2007 From: bbackde at googlemail.com (bbackde at googlemail.com) Date: Sat, 10 Feb 2007 19:09:58 +0100 Subject: [Tech] SimpleProgress does not show filesize In-Reply-To: <20070104143652.GD4525@freenetproject.org> References: <20070104143652.GD4525@freenetproject.org> Message-ID: While I'm on the FCP2 code I will add a new parameter to the SimpleProgress for a GET that returns the computed filesize when a GET is finalized. Ok? On 1/4/07, Florent Daigni?re (NextGen$) wrote: > * bbackde at googlemail.com [2006-12-29 22:09:04]: > > > If a SimpleProgress message is sent with Finalized=true, does'nt the > > node know the final filesize at this time? I assume yes, and I want > > that the size is sent in the SimpleProgress message if it is known. > > And I *want* you to fill in a ticket on the bug tracking system using the > category "wishlist or feature" :p > > More seriously speaking ... they are more urgent matters to be addressed > ; will do but not now. > > NextGen$ > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.6 (GNU/Linux) > > iD8DBQFFnREEU/Z/dHFfxtcRApGMAKDywR4DSCwDj+/CWmTna/7JdRWSYwCghIIN > pCJIVZdUAzQeR2FZ+hrAHCM= > =ZQCM > -----END PGP SIGNATURE----- > > > _______________________________________________ > Tech mailing list > Tech at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > > From juiceman69 at gmail.com Sat Feb 10 21:41:46 2007 From: juiceman69 at gmail.com (Juiceman) Date: Sat, 10 Feb 2007 16:41:46 -0500 Subject: [Tech] BDB uses far more memory than it's supposed to; ideas? In-Reply-To: <20070130165441.GA24058@amphibian.dyndns.org> References: <20070130165441.GA24058@amphibian.dyndns.org> Message-ID: <8b525dee0702101341s53921ab9u195549bf9f1c19c8@mail.gmail.com> Maybe I am completely off base here, but according to http://www.oracle.com/technology/documentation/berkeley-db/je/relnotes.html You must set the memory using a few different configuration properties. I searched the Freenet source and was unable to find: EnvironmentConfig.setCacheSize je.maxMemory je.maxMemoryPercent Are you using a command or function other than these? On 1/30/07, Matthew Toseland wrote: > After various reports of OOMs, and after high CPU usage which might have > been related to excessive GCing, I did some profiling. > > Having analysed the first 70% of memory on the dump, with a 100MB > overall limit, the Berkeley DB Java Edition that we use for the database > uses at least 54.5MB of RAM. databaseMaxMemory is not overridden in the > config, and therefore should be at the default value of 20MB. > > The documentation says: > "Note that the cache does not include transient objects created by the JE > library, such as cursors, locks and transactions." > > It is possible that the cache only refers to the first stack trace, > which is 17MB. However there are many other traces which appear to be > dealing with the database log (BDB is a log-structured database), many > of them rather large - for example the next 3 traces of 10MB, 5MB and > 5MB respectively. > > Any ideas? We can: > - Just ignore it. We may have to increase the default memory limit. But > anecdotally OOMs happen when you have lots of queued requests. > - Use another database. BDB has been fairly unreliable, hence the code > in the store to reconstruct the database (store index) from the store > file by deleting the database and parsing each key. > - Given the second item, if we had a reliable database we could store > queued requests in it, thus limiting the overall memory usage > regardless of the size of the request queue. But that would be a > significant amount of work even with a database. > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.6 (GNU/Linux) > > iD8DBQFFv3hRA9rUluQ9pFARAlC/AJ9u0JzMj746VIK2hSYA76NNbjxs6ACeMS8E > TTux6RxGK0Eo4PGlKQzCldM= > =c7LM > -----END PGP SIGNATURE----- > > _______________________________________________ > Tech mailing list > Tech at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > -- I may disagree with what you have to say, but I shall defend, to the death, your right to say it. - Voltaire Those who would give up Liberty, to purchase temporary Safety, deserve neither Liberty nor Safety. - Ben Franklin From juiceman69 at gmail.com Sat Feb 10 22:28:10 2007 From: juiceman69 at gmail.com (Juiceman) Date: Sat, 10 Feb 2007 17:28:10 -0500 Subject: [Tech] BDB uses far more memory than it's supposed to; ideas? In-Reply-To: <8b525dee0702101341s53921ab9u195549bf9f1c19c8@mail.gmail.com> References: <20070130165441.GA24058@amphibian.dyndns.org> <8b525dee0702101341s53921ab9u195549bf9f1c19c8@mail.gmail.com> Message-ID: <8b525dee0702101428l1884662ewa320067bf35a1f1e@mail.gmail.com> Another thing I noticed is from the API http://www.stylusstudio.com/api/sleepycat/berkeleydb/index-all.htm setCacheSize(long, int) - Method in class com.sleepycat.db.DbEnv Set the size of the shared memory buffer pool -- that is, the cache. setCacheSize(long, int) - Method in class com.sleepycat.db.Db Set the size of the shared memory buffer pool -- that is, the cache. These seem to be calling the class class com.sleepycat.db.Db or class com.sleepycat.db.Db, yet in our Node.java you are only importing import com.sleepycat.je.DatabaseException; import com.sleepycat.je.Environment; import com.sleepycat.je.EnvironmentConfig; import com.sleepycat.je.EnvironmentMutableConfig; import com.sleepycat.je.StatsConfig; Is the sleepycat.je vs sleepycat.db relevant? Sorry if I am asking a stupid questions. I am a code hacker (in the traditional sense) at best. > > -- > I may disagree with what you have to say, but I shall defend, to the > death, your right to say it. - Voltaire > Those who would give up Liberty, to purchase temporary Safety, deserve > neither Liberty nor Safety. - Ben Franklin > -- I may disagree with what you have to say, but I shall defend, to the death, your right to say it. - Voltaire Those who would give up Liberty, to purchase temporary Safety, deserve neither Liberty nor Safety. - Ben Franklin From toad at amphibian.dyndns.org Sat Feb 10 23:38:08 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Sat, 10 Feb 2007 23:38:08 +0000 Subject: [Tech] SimpleProgress does not show filesize In-Reply-To: References: <20070104143652.GD4525@freenetproject.org> Message-ID: <20070210233808.GA8237@amphibian.dyndns.org> On Sat, Feb 10, 2007 at 07:09:58PM +0100, bbackde at googlemail.com wrote: > While I'm on the FCP2 code I will add a new parameter to the > SimpleProgress for a GET that returns the computed filesize when a GET > is finalized. Sounds good to me. > > Ok? > > On 1/4/07, Florent Daigni?re (NextGen$) wrote: > > * bbackde at googlemail.com [2006-12-29 22:09:04]: > > > > > If a SimpleProgress message is sent with Finalized=true, does'nt the > > > node know the final filesize at this time? I assume yes, and I want > > > that the size is sent in the SimpleProgress message if it is known. > > > > And I *want* you to fill in a ticket on the bug tracking system using the > > category "wishlist or feature" :p > > > > More seriously speaking ... they are more urgent matters to be addressed > > ; will do but not now. > > > > NextGen$ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070210/fc162e6f/attachment.pgp From toad at amphibian.dyndns.org Sat Feb 10 23:39:13 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Sat, 10 Feb 2007 23:39:13 +0000 Subject: [Tech] BDB uses far more memory than it's supposed to; ideas? In-Reply-To: <8b525dee0702101428l1884662ewa320067bf35a1f1e@mail.gmail.com> References: <20070130165441.GA24058@amphibian.dyndns.org> <8b525dee0702101341s53921ab9u195549bf9f1c19c8@mail.gmail.com> <8b525dee0702101428l1884662ewa320067bf35a1f1e@mail.gmail.com> Message-ID: <20070210233913.GB8237@amphibian.dyndns.org> We do set the maximum cache size, in absolute terms, in Node, via EnvironmentConfig. On Sat, Feb 10, 2007 at 05:28:10PM -0500, Juiceman wrote: > Another thing I noticed is from the API > http://www.stylusstudio.com/api/sleepycat/berkeleydb/index-all.htm > > setCacheSize(long, int) - Method in class com.sleepycat.db.DbEnv > Set the size of the shared memory buffer pool -- that is, the cache. > setCacheSize(long, int) - Method in class com.sleepycat.db.Db > Set the size of the shared memory buffer pool -- that is, the cache. > > These seem to be calling the class class com.sleepycat.db.Db or class > com.sleepycat.db.Db, yet in our Node.java you are only importing > import com.sleepycat.je.DatabaseException; > import com.sleepycat.je.Environment; > import com.sleepycat.je.EnvironmentConfig; > import com.sleepycat.je.EnvironmentMutableConfig; > import com.sleepycat.je.StatsConfig; > > Is the sleepycat.je vs sleepycat.db relevant? > > Sorry if I am asking a stupid questions. I am a code hacker (in the > traditional sense) at best. > > > > > > -- > > I may disagree with what you have to say, but I shall defend, to the > > death, your right to say it. - Voltaire > > Those who would give up Liberty, to purchase temporary Safety, deserve > > neither Liberty nor Safety. - Ben Franklin > > > > > -- > I may disagree with what you have to say, but I shall defend, to the > death, your right to say it. - Voltaire > Those who would give up Liberty, to purchase temporary Safety, deserve > neither Liberty nor Safety. - Ben Franklin > _______________________________________________ > Tech mailing list > Tech at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070210/d3710aa2/attachment.pgp From juiceman69 at gmail.com Sun Feb 11 19:48:06 2007 From: juiceman69 at gmail.com (Juiceman) Date: Sun, 11 Feb 2007 14:48:06 -0500 Subject: [Tech] BDB uses far more memory than it's supposed to; ideas? In-Reply-To: <20070210233913.GB8237@amphibian.dyndns.org> References: <20070130165441.GA24058@amphibian.dyndns.org> <8b525dee0702101341s53921ab9u195549bf9f1c19c8@mail.gmail.com> <8b525dee0702101428l1884662ewa320067bf35a1f1e@mail.gmail.com> <20070210233913.GB8237@amphibian.dyndns.org> Message-ID: <8b525dee0702111148v4df9ac7m2de6ec89e1961ec9@mail.gmail.com> Hmm, I'm out of ideas. Here is the official forum if you would like to ask the BDB developers. http://forums.oracle.com/forums/forum.jspa?forumID=273&start=0 On 2/10/07, Matthew Toseland wrote: > We do set the maximum cache size, in absolute terms, in Node, via > EnvironmentConfig. > > On Sat, Feb 10, 2007 at 05:28:10PM -0500, Juiceman wrote: > > Another thing I noticed is from the API > > http://www.stylusstudio.com/api/sleepycat/berkeleydb/index-all.htm > > > > setCacheSize(long, int) - Method in class com.sleepycat.db.DbEnv > > Set the size of the shared memory buffer pool -- that is, the cache. > > setCacheSize(long, int) - Method in class com.sleepycat.db.Db > > Set the size of the shared memory buffer pool -- that is, the cache. > > > > These seem to be calling the class class com.sleepycat.db.Db or class > > com.sleepycat.db.Db, yet in our Node.java you are only importing > > import com.sleepycat.je.DatabaseException; > > import com.sleepycat.je.Environment; > > import com.sleepycat.je.EnvironmentConfig; > > import com.sleepycat.je.EnvironmentMutableConfig; > > import com.sleepycat.je.StatsConfig; > > > > Is the sleepycat.je vs sleepycat.db relevant? > > > > Sorry if I am asking a stupid questions. I am a code hacker (in the > > traditional sense) at best. > > > > > > > > > > -- > > > I may disagree with what you have to say, but I shall defend, to the > > > death, your right to say it. - Voltaire > > > Those who would give up Liberty, to purchase temporary Safety, deserve > > > neither Liberty nor Safety. - Ben Franklin > > > > > > > > > -- > > I may disagree with what you have to say, but I shall defend, to the > > death, your right to say it. - Voltaire > > Those who would give up Liberty, to purchase temporary Safety, deserve > > neither Liberty nor Safety. - Ben Franklin > > _______________________________________________ > > Tech mailing list > > Tech at freenetproject.org > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.6 (GNU/Linux) > > iD8DBQFFzlehA9rUluQ9pFARAtdvAKCF4SSz973/5I6Ij1mBC95Mnjr1FACfcg32 > uyBZ2BIC9IRCN0arojSIwn4= > =gnpJ > -----END PGP SIGNATURE----- > > _______________________________________________ > Tech mailing list > Tech at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > -- I may disagree with what you have to say, but I shall defend, to the death, your right to say it. - Voltaire Those who would give up Liberty, to purchase temporary Safety, deserve neither Liberty nor Safety. - Ben Franklin From nextgens at freenetproject.org Sun Feb 11 21:16:06 2007 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re_=28NextGen$=29?=) Date: Sun, 11 Feb 2007 22:16:06 +0100 Subject: [Tech] 0.5 seednode file Message-ID: <20070211211606.GA4008@freenetproject.org> Hi, Some people have been complaining that the 0.5 seedfile we bundle on our website is outdated : Indeed it is. Emu is still harvesting data and assembling/publishing it. The problem is that most -if not all- of our 0.5 sources are down or unreachable. Are some people still running .5 ? Would they contribute to the seedfile generation letting our server grab some data from their fproxy ? NextGen$ From nextgens at freenetproject.org Sat Feb 10 18:17:18 2007 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re_=28NextGen$=29?=) Date: Sat, 10 Feb 2007 19:17:18 +0100 Subject: [Tech] SimpleProgress does not show filesize In-Reply-To: References: <20070104143652.GD4525@freenetproject.org> Message-ID: <20070210181717.GA7594@freenetproject.org> * bbackde at googlemail.com [2007-02-10 19:09:58]: > While I'm on the FCP2 code I will add a new parameter to the > SimpleProgress for a GET that returns the computed filesize when a GET > is finalized. > > Ok? Well, if you do it, it's fine by me :) > > On 1/4/07, Florent Daigni?re (NextGen$) wrote: > >* bbackde at googlemail.com [2006-12-29 22:09:04]: > > > >> If a SimpleProgress message is sent with Finalized=true, does'nt the > >> node know the final filesize at this time? I assume yes, and I want > >> that the size is sent in the SimpleProgress message if it is known. > > > >And I *want* you to fill in a ticket on the bug tracking system using the > >category "wishlist or feature" :p > > > >More seriously speaking ... they are more urgent matters to be addressed > >; will do but not now. > > > >NextGen$ From bbackde at googlemail.com Mon Feb 12 19:19:00 2007 From: bbackde at googlemail.com (bbackde at googlemail.com) Date: Mon, 12 Feb 2007 20:19:00 +0100 Subject: [Tech] SimpleProgress does not show filesize In-Reply-To: <20070210181717.GA7594@freenetproject.org> References: <20070104143652.GD4525@freenetproject.org> <20070210181717.GA7594@freenetproject.org> Message-ID: A question on this: do I have to compute the file size from the block count * 32kb, or can I use the true file size? where is the true file size stored? The node have to know it to finally truncate the resulting file to the correct length, true? On 2/10/07, Florent Daigni?re (NextGen$) wrote: > * bbackde at googlemail.com [2007-02-10 19:09:58]: > > > While I'm on the FCP2 code I will add a new parameter to the > > SimpleProgress for a GET that returns the computed filesize when a GET > > is finalized. > > > > Ok? > > Well, if you do it, it's fine by me :) > > > > > On 1/4/07, Florent Daigni?re (NextGen$) wrote: > > >* bbackde at googlemail.com [2006-12-29 22:09:04]: > > > > > >> If a SimpleProgress message is sent with Finalized=true, does'nt the > > >> node know the final filesize at this time? I assume yes, and I want > > >> that the size is sent in the SimpleProgress message if it is known. > > > > > >And I *want* you to fill in a ticket on the bug tracking system using the > > >category "wishlist or feature" :p > > > > > >More seriously speaking ... they are more urgent matters to be addressed > > >; will do but not now. > > > > > >NextGen$ > From bbackde at googlemail.com Tue Feb 13 20:10:24 2007 From: bbackde at googlemail.com (bbackde at googlemail.com) Date: Tue, 13 Feb 2007 21:10:24 +0100 Subject: [Tech] From Frost: missing USKFetcher removal? Message-ID: Did you see this message from Frost? Is he right? ----- Jano at vKDFgv2GdJjiIWSG0rZpzM55_Ps ----- 2007.02.13 - 14:47:52GMT ----- I don't have bookmarks. But even then, I don't understand why the USKChecker counts don't go down. See this new graph from the latest hours. I've launched fetchs for ~300 different USKs some hours ago. I'd expect the USKChecker count to reach some plateau and start dropping after a while... ----- Jano at vKDFgv2GdJjiIWSG0rZpzM55_Ps ----- 2007.02.13 - 18:36:47GMT ----- I've inspected the USKManager a bit more and I've found that this method: synchronized void finished(USKFetcher f) { USK u = f.getOriginalUSK(); fetchersByUSK.remove(u); } is the only place where a fetcher is removed from fetchersByUSK. However, according to eclipse, there's no references to it. I.e., nobody is calling that method. After placing some logs in USKManager, I indeed see that after retrieving some USK using the /-1/ edition form, it remains in the fetchersByUSK collection. From toad at amphibian.dyndns.org Wed Feb 14 01:46:16 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Wed, 14 Feb 2007 01:46:16 +0000 Subject: [Tech] SimpleProgress does not show filesize In-Reply-To: References: <20070104143652.GD4525@freenetproject.org> <20070210181717.GA7594@freenetproject.org> Message-ID: <20070214014616.GA23897@amphibian.dyndns.org> On Mon, Feb 12, 2007 at 08:19:00PM +0100, bbackde at googlemail.com wrote: > A question on this: do I have to compute the file size from the block > count * 32kb, or can I use the true file size? where is the true file > size stored? The node have to know it to finally truncate the > resulting file to the correct length, true? The final file size is known... but only if we are fetching the final layer of the pyramid. FetchException has fields for the final size and MIME type and whether they are final. Find the constructor which sets these and have a look at the callers, and you should be able to find what you need. > > On 2/10/07, Florent Daigni?re (NextGen$) wrote: > > * bbackde at googlemail.com [2007-02-10 19:09:58]: > > > > > While I'm on the FCP2 code I will add a new parameter to the > > > SimpleProgress for a GET that returns the computed filesize when a GET > > > is finalized. > > > > > > Ok? > > > > Well, if you do it, it's fine by me :) > > > > > > > > On 1/4/07, Florent Daigni?re (NextGen$) wrote: > > > >* bbackde at googlemail.com [2006-12-29 22:09:04]: > > > > > > > >> If a SimpleProgress message is sent with Finalized=true, does'nt the > > > >> node know the final filesize at this time? I assume yes, and I want > > > >> that the size is sent in the SimpleProgress message if it is known. > > > > > > > >And I *want* you to fill in a ticket on the bug tracking system using the > > > >category "wishlist or feature" :p > > > > > > > >More seriously speaking ... they are more urgent matters to be addressed > > > >; will do but not now. > > > > > > > >NextGen$ > > > _______________________________________________ > Tech mailing list > Tech at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070214/2d52df20/attachment.pgp From dreidland at houston.rr.com Wed Feb 14 03:59:28 2007 From: dreidland at houston.rr.com (David Reidland) Date: Tue, 13 Feb 2007 21:59:28 -0600 Subject: [Tech] download resuming is broken In-Reply-To: <20070214014616.GA23897@amphibian.dyndns.org> Message-ID: <200702140359.l1E3xSCa013024@ms-smtp-04.texas.rr.com> I have done a lot of testing and download resuming is broken. Insert resuming works fine. This is on Windows XP with Freenet 0.7 Build #1015 r11780. I am not the only person having this problem. -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.441 / Virus Database: 268.17.37/682 - Release Date: 2/12/2007 1:23 PM From m.rogers at cs.ucl.ac.uk Wed Feb 14 09:59:16 2007 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Wed, 14 Feb 2007 09:59:16 +0000 Subject: [Tech] Let's use TCP and TLS Message-ID: <45D2DD74.2000000@cs.ucl.ac.uk> Let's use TCP and TLS. Disadvantages: * Most users will have to forward a port. However, the popularity of BitTorrent shows that most users can manage this (especially with UPnP and NAT-PMP support, which we can copy from Azureus or Limewire). * TCP delivers messages in order, which we probably don't need. However, it would be nice not to have to worry about malicious reordering in the higher layers of the protocol (two words: model checking). Advantages: * We don't have to maintain our own UDP transport layer. * We don't have to maintain as much crypto code (which is notoriously hard to debug). * We can use standard, peer-reviewed cryptographic protocols instead of rolling our own (again, hard to debug). * Freenet traffic becomes much harder to identify (lots of other software uses TLS over TCP, including HTTPS, POP, IMAP, SMTP...) Some more thoughts about disguising Freenet traffic: * Use an ephemeral port for each outgoing TCP connection (this would defeat the filter mentioned in the SecurityFocus article... by the way, check out who the author used to work for) * Add a simple stego wrapper to the TLS socket: the wrapper keeps outgoing data in a queue and passes random-sized chunks of data to the socket at random intervals (the intervals get shorter when the queue gets longer). By the time the data's been packed into records by TLS and coalesced into sensible-sized packets by TCP, the original size and timing of the messages should be pretty well disguised. It would be easy to extend this to CBR padding at a later date (keep the mean interval the same regardless of queue length, send padding when the queue is empty). I know this suggestion is unlikely to be accepted, but I'm not trolling - we could save ourselves a lot of work by using what's already out there, and BitTorrent shows that it wouldn't hurt performance or adoption. Cheers, Michael From toad at amphibian.dyndns.org Wed Feb 14 17:11:03 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Wed, 14 Feb 2007 17:11:03 +0000 Subject: [Tech] Let's use TCP and TLS In-Reply-To: <45D2DD74.2000000@cs.ucl.ac.uk> References: <45D2DD74.2000000@cs.ucl.ac.uk> Message-ID: <20070214171103.GA20090@amphibian.dyndns.org> On Wed, Feb 14, 2007 at 09:59:16AM +0000, Michael Rogers wrote: > Let's use TCP and TLS. As a transport plugin, sure. As the default transport plugin and something to implement in the near future, no. > > Disadvantages: > > * Most users will have to forward a port. However, the popularity of > BitTorrent shows that most users can manage this (especially with UPnP > and NAT-PMP support, which we can copy from Azureus or Limewire). UPnP has serious problems. It works about 50% of the time, and if you happen to be behind an ISP NAT or especially on an untrusted LAN it's a major vulnerability. UDP will often work even behind relatively hostile firewalls e.g. it is possible to get it past many nasty corporate PATs with the relevant flags, and it will in most cases when combined with STUN work almost out of the box. > > * TCP delivers messages in order, which we probably don't need. However, > it would be nice not to have to worry about malicious reordering in the > higher layers of the protocol (two words: model checking). It is rare that message re-ordering is a problem. However it does use some resources to track recent unmatched messages, and the level of resources required will increase with the bandwidth used. Also in-order delivery increases latency. Per hop latency dominates request times. > > Advantages: > > * We don't have to maintain our own UDP transport layer. > > * We don't have to maintain as much crypto code (which is notoriously > hard to debug). We have to maintain buggy platform dependant UP&P code instead? > > * We can use standard, peer-reviewed cryptographic protocols instead of > rolling our own (again, hard to debug). Which don't necessarily meet our needs. > > * Freenet traffic becomes much harder to identify (lots of other > software uses TLS over TCP, including HTTPS, POP, IMAP, SMTP...) This is a fair point. > > Some more thoughts about disguising Freenet traffic: > > * Use an ephemeral port for each outgoing TCP connection (this would > defeat the filter mentioned in the SecurityFocus article... by the way, > check out who the author used to work for) This has been suggested on Frost. With UDP it is impossible because of NAT-related issues (port rewriting etc). With TCP it is standard practice, but it doesn't buy us any real protection from traffic flow analysis attacks, which look at the number of connections from an IP address, their typical uptime, the type of the destination IPs etc. > > * Add a simple stego wrapper to the TLS socket: the wrapper keeps > outgoing data in a queue and passes random-sized chunks of data to the > socket at random intervals (the intervals get shorter when the queue > gets longer). By the time the data's been packed into records by TLS and > coalesced into sensible-sized packets by TCP, the original size and > timing of the messages should be pretty well disguised. It would be easy > to extend this to CBR padding at a later date (keep the mean interval > the same regardless of queue length, send padding when the queue is empty). Very high latency, otherwise a good idea if we were using TLS, but not enough to swing me in favour of TLS. > > I know this suggestion is unlikely to be accepted, but I'm not trolling > - we could save ourselves a lot of work by using what's already out > there, and BitTorrent shows that it wouldn't hurt performance or adoption. Don't modern BitTorrent's use UDP to avoid firewall issues? BitTorrent was originally adopted by geeks. The more hoops you have to get users to jump through (e.g. port forwarding), the fewer users will use your product. We can't assume that everyone who uses freenet will be familiar with port forwarding already. > > Cheers, > Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070214/8abc77b0/attachment.pgp From ian at locut.us Wed Feb 14 23:30:39 2007 From: ian at locut.us (Ian Clarke) Date: Wed, 14 Feb 2007 16:30:39 -0700 Subject: [Tech] Let's use TCP and TLS In-Reply-To: <45D2DD74.2000000@cs.ucl.ac.uk> References: <45D2DD74.2000000@cs.ucl.ac.uk> Message-ID: <823242bd0702141530w5143972m46e582a6d95fc495@mail.gmail.com> On 2/14/07, Michael Rogers wrote: > Let's use TCP and TLS. > > Disadvantages: > > * Most users will have to forward a port. However, the popularity of > BitTorrent shows that most users can manage this (especially with UPnP > and NAT-PMP support, which we can copy from Azureus or Limewire). I don't think BitTorrent's popularity proves this at all, I don't have the statistics but I suspect that the vast majority of BitTorrent users do not set up port-forwarding. We used TCP prior to Freenet 0.7, and learned the hard way that TCP simply isn't the right tool for the job, it does a bunch of things we don't need (such as guaranteed in-order delivery), and can't do some of the things we do need (such as NAT traversal). I do agree that we need to work on disguising Freenet traffic, but there are no shortage of UDP-based protocols that we can use for cover (eg. Skype). Ian. -- Founder and CEO, Thoof Inc Email: ian at thoof.com Phone: 310 295 0164 Cell: 310 593 3724 AIM: ian.clarke at mac.com Skype: sanity From m.rogers at cs.ucl.ac.uk Thu Feb 15 14:41:02 2007 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Thu, 15 Feb 2007 14:41:02 +0000 Subject: [Tech] Let's use TCP and TLS In-Reply-To: <20070214171103.GA20090@amphibian.dyndns.org> References: <45D2DD74.2000000@cs.ucl.ac.uk> <20070214171103.GA20090@amphibian.dyndns.org> Message-ID: <45D470FE.4020502@cs.ucl.ac.uk> > Don't modern BitTorrent's use UDP to avoid firewall issues? Not for data transfer - some use UDP to communicate with the tracker, but that's not because of firewall issues, it's because UDP is a good match for the tracker protocol. And TCP is a good match for data transfer. ;-) > BitTorrent was originally adopted by geeks. So were iPods - that doesn't mean other people are incapable of using them, it just means geeks like new technology. But anyway, if you don't want to use TCP then fair enough, I just thought I'd put the idea on the table. :-) Cheers, Michael From toad at amphibian.dyndns.org Thu Feb 15 15:47:26 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Thu, 15 Feb 2007 15:47:26 +0000 Subject: [Tech] Let's use TCP and TLS In-Reply-To: <45D470FE.4020502@cs.ucl.ac.uk> References: <45D2DD74.2000000@cs.ucl.ac.uk> <20070214171103.GA20090@amphibian.dyndns.org> <45D470FE.4020502@cs.ucl.ac.uk> Message-ID: <20070215154726.GA18670@amphibian.dyndns.org> On Thu, Feb 15, 2007 at 02:41:02PM +0000, Michael Rogers wrote: > > Don't modern BitTorrent's use UDP to avoid firewall issues? > > Not for data transfer - some use UDP to communicate with the tracker, > but that's not because of firewall issues, it's because UDP is a good > match for the tracker protocol. And TCP is a good match for data > transfer. ;-) Well, we need both mechanisms. We have lots of control messages, which fit best with UDP (presumably bittorrent provides its own retransmission? can communication with the tracker tolerate packet loss?). And we would need to multiplex our TCP connections if we used them for data transfer. > > > BitTorrent was originally adopted by geeks. > > So were iPods - that doesn't mean other people are incapable of using > them, it just means geeks like new technology. > > But anyway, if you don't want to use TCP then fair enough, I just > thought I'd put the idea on the table. :-) :) > > Cheers, > Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070215/f6092d97/attachment.pgp From mrflibble at urbantakeover.freeserve.co.uk Tue Feb 27 23:06:22 2007 From: mrflibble at urbantakeover.freeserve.co.uk (Mr. Flibble) Date: Tue, 27 Feb 2007 23:06:22 -0000 Subject: [Tech] 0.5 seednode file In-Reply-To: <20070211211606.GA4008@freenetproject.org> Message-ID: <02a601c75ac3$e67ddfc0$0a7b0001@deepspace3> > Are some people still running .5 ? Would they contribute to the > seedfile generation letting our server grab some data from their > fproxy ? I'm still running mine 24/7ish, what do you need from me? From bbackde at googlemail.com Wed Feb 28 07:22:36 2007 From: bbackde at googlemail.com (bbackde at googlemail.com) Date: Wed, 28 Feb 2007 08:22:36 +0100 Subject: [Tech] FYI: Anonymous networks vulnerable to attacks with fake routing data Message-ID: I don't know if this is old news. See http://www.heise-security.co.uk/news/85965 (german: http://www.heise.de/newsticker/meldung/85944 ) From nextgens at freenetproject.org Wed Feb 28 12:36:06 2007 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re_=28NextGen$=29?=) Date: Wed, 28 Feb 2007 13:36:06 +0100 Subject: [Tech] 0.5 seednode file In-Reply-To: <02a601c75ac3$e67ddfc0$0a7b0001@deepspace3> References: <20070211211606.GA4008@freenetproject.org> <02a601c75ac3$e67ddfc0$0a7b0001@deepspace3> Message-ID: <20070228123606.GA4128@freenetproject.org> * Mr. Flibble [2007-02-27 23:06:22]: > > Are some people still running .5 ? Would they contribute to the > > seedfile generation letting our server grab some data from their > > fproxy ? > > I'm still running mine 24/7ish, what do you need from me? Possibly that you allow TCP connections to your node's Fproxy port from emu.freenetproject.org(80.68.80.201) and a FQDN or ip address where you can be reached. NextGen$