From spapkt at yahoo.com Wed May 2 14:03:58 2007 From: spapkt at yahoo.com (SPKT) Date: Wed, 2 May 2007 07:03:58 -0700 (PDT) Subject: [Tech] two long-term proposals... Message-ID: <70217.51920.qm@web63106.mail.re1.yahoo.com> Dear Freenet Developers, I could say "I have a brilliant idea" but I realize it sounds quite trivial :) However, I really think I have two specific solutions for two Freenet problems. The first one is about archiving and accessing archived messages in Freenet groups. The second one proposes how to make the Freenet Project more popular. I'm presenting these two ideas in detailed way below. If you find them useful (or at least partially useful) I would be more than happy as I'm a great Freenet fan. But if my ideas don't appeal to you and you find them worthless then just ignore my e-mail. I believe that sharing ideas (even if they're somehow obvious and banal) may improve the project. Maybe you will use some of themin future when creating the 1.0 version of Freenet :) Best regards, Karol * * * * * * * * * * * * * * Ideas: * * * * * * * * * * * * * * 1. The retention problem in Freenet message groups (boards) Short description: Storing and accessing archived messages using groups' readers - software like Frost or FMB. Longer description: Everyone using Frost or FMB may be disappointed by the "retention problem". It means that messages posted to Freenet groups are accessible in the network for a very small period of time. That is because of the Freenet's specific architecture. There are no archives of particular Freenet groups because there are no servers which could keep older messages. This problem makes Freenet groups quite inconvenient comparing them with Usenet groups or web based discussion boards (forums) where archived messages are easy accessible for users. A new user who joins a group expects to read its archived messages and eventually to answer some of them. Solution: Well, when a group is created all messages can be automatically archived by users who want to do this. Using a special feature in a newsreader (for example, called "auto-store messages") they can be saved and stored off-line (sic!) on users' hard drives. A new user joining a group may request to download its archived messages by placing such request in a special control message sending to the group (for example, called "download group archive"). Once message is sent it may be automatically proceeded by these users who have stored (off-line!) archived messages and who agree to send them to the requester (this newsreader option can be called, for example, "auto-send archives on request"). Then it is sending in the typical Freenet's way - the compressed package is anonymously uploaded to Freenet and the user downloads it automatically using his group reader (newsreader). That's all. Additional information: Of course some archives of a group may differ as not all users are still on-line and have all messages. But a newsreader should be able to complete archives using different sources (downloading archives from other users, comparing them and creating the one complete archive of a group) - this option can be called, for example, "reconstruct group archive". More users join and read a group (and post to it), more completed archives the group may have. It is efficient because it promotes well-organized groups with a lot of active users while ephemeral groups disappear as nobody reads them and archives their messages (content). Storing groups' archives off-line and uploading them on request cause that the whole Freenet network works more efficient due to the fact that this solution saves traffic and space. In general, this solution is somehow based on the Usenet structure with the advantage of Freenet anonymity. Conclusions: The retention problem described above is inevitably connected with newsreaders - software for accessing groups (reading and posting to them). It should be designed in the way to allow users to operate very easily. Clear but in the same time advanced interface is the key. For example, the tree-thread list structure (collapse / expand threads) or the web-boards answering style (an answer for a particular thread shifts this thread to the top of the threads list). Such solutions can be copied from some good Usenet readers. It is really worth introducing because it may encourage more people to use Freenet groups (and so Freenet itself) which may even become the real alternative to Usenet in some future. Off course I realize that perhaps this proposal is rather for Frost or other newsreaders software developers than for Freenet Developers. But I think Freenet Developers should also be noticed about this idea. However, this kind of scheme (storing information packages off-line and sending them on request by uploading them to Freenet) may be used even for creating Freenet web-based forums in some future. 2. How to popularize Freenet among internet users? Translating the main web page of the Freenet Project into other languages and creating an easy as well as users-friendly web-based forum. This idea is obvious and easy to be done. Translating the main web page of the Freenet Project (freenetproject.org) into other languages can be very beneficial. I'm sure there are a lot of Freenet friends who would like to support the project this way. 7-10 language versions of the Freenet web page could bring a serious number of new users to the network. These language versions of the main web page don't have to be the exact and full translation of the original English version. It may be something like a simplified version of the main web page, including step-by-step instructions (how to download the software, install it, connect to the network and use it - illustrated with proper screenshots), basic information about Freenet and its creators, philosophy, FAQ and the most important news. The web page content is rather "static" (it is not changed often and it is updated quite rarely) so it would be easy to maintain other language versions. Maybe it is a good idea to announce that the Freenet Project is looking for voluntary translators of its web page? Think about it. Another feature improving Freenet's popularity would be the web based forum where people interested in Freenet could ask questions and gain more information about the project. It would be much more convenient than existing mailing lists and their archives. The forum could have also language sections. This idea is also easy to introduce and may be very valuable in popularizing the Freenet Project. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From rah at bash.sh Thu May 3 15:21:48 2007 From: rah at bash.sh (Bob Ham) Date: Thu, 03 May 2007 16:21:48 +0100 Subject: [Tech] Cache and Store Message-ID: <1178205708.10595.14.camel@localhost> Hi there, What's the difference between the key Cache and the key Store, as described on the statistics page? What are the conditions for the keys going into and out of the two stores? Also, I understand that the configured store size limit is split exactly 50:50 between the two. This is a problem. Only a very small percentage of the Store is in use, but the Cache is nearing full. I've allocated as much space through the configuration page as I'm prepared to give. A very large portion of this space (about 40%) is wasted; it's not being used by the Store while it could be used by the Cache. I want to assign different limits to each set of keys. Is there a reason this can't happen? Regards, Bob -- Bob Ham From m.rogers at cs.ucl.ac.uk Thu May 3 16:02:32 2007 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Thu, 03 May 2007 17:02:32 +0100 Subject: [Tech] Cache and Store In-Reply-To: <1178205708.10595.14.camel@localhost> References: <1178205708.10595.14.camel@localhost> Message-ID: <463A0798.5090506@cs.ucl.ac.uk> Bob Ham wrote: > What's the difference between the key Cache and the key Store, as > described on the statistics page? What are the conditions for the keys > going into and out of the two stores? The cache contains the keys most recently inserted or requested via your node (either locally or by your peers). The store contains the keys most recently inserted (and requested? not sure) via your node, for which your node's location is closer to the key than any of your peers' locations are. So the cache tends to contain data that's popular at the moment, and the store tends to contain data that's close to your node's location. > Also, I understand that the configured store size limit is split exactly > 50:50 between the two. This is a problem. Only a very small percentage > of the Store is in use, but the Cache is nearing full. The store takes a lot longer to fill up, but once it's full it should provide better long-term storage (less redundancy between nodes). Cheers, Michael From rah at bash.sh Thu May 3 16:43:46 2007 From: rah at bash.sh (Bob Ham) Date: Thu, 03 May 2007 17:43:46 +0100 Subject: [Tech] Cache and Store In-Reply-To: <463A0798.5090506@cs.ucl.ac.uk> References: <1178205708.10595.14.camel@localhost> <463A0798.5090506@cs.ucl.ac.uk> Message-ID: <1178210627.10595.24.camel@localhost> On Thu, 2007-05-03 at 17:02 +0100, Michael Rogers wrote: > Bob Ham wrote: > ... So the cache tends to contain data that's popular at the > moment, and the store tends to contain data that's close to your node's > location. What is a node's "location"? > > Also, I understand that the configured store size limit is split exactly > > 50:50 between the two. This is a problem. Only a very small percentage > > of the Store is in use, but the Cache is nearing full. > > The store takes a lot longer to fill up, but once it's full it should > provide better long-term storage (less redundancy between nodes). Sure. The issue is the loss of potential storage space in the time it takes to fill up. What I propose is two things: firstly, the 50:50 split seems pretty arbitrary; it should be configurable. Secondly, the node should fill up each store regardless of the configured split point until it reaches the maximum overall store size. At that time, it should reduce the size of whichever store is over its allocation as and when the other store needs it. Bob -- Bob Ham From juiceman69 at gmail.com Thu May 3 22:12:28 2007 From: juiceman69 at gmail.com (Juiceman) Date: Thu, 3 May 2007 18:12:28 -0400 Subject: [Tech] Cache and Store In-Reply-To: <1178210627.10595.24.camel@localhost> References: <1178205708.10595.14.camel@localhost> <463A0798.5090506@cs.ucl.ac.uk> <1178210627.10595.24.camel@localhost> Message-ID: <8b525dee0705031512i4a72e35fqb126fe6c5fe0ff17@mail.gmail.com> On 5/3/07, Bob Ham wrote: > On Thu, 2007-05-03 at 17:02 +0100, Michael Rogers wrote: > > Bob Ham wrote: > > > ... So the cache tends to contain data that's popular at the > > moment, and the store tends to contain data that's close to your node's > > location. > > What is a node's "location"? > > > > > Also, I understand that the configured store size limit is split exactly > > > 50:50 between the two. This is a problem. Only a very small percentage > > > of the Store is in use, but the Cache is nearing full. > > > > The store takes a lot longer to fill up, but once it's full it should > > provide better long-term storage (less redundancy between nodes). > > Sure. The issue is the loss of potential storage space in the time it > takes to fill up. What I propose is two things: firstly, the 50:50 > split seems pretty arbitrary; it should be configurable. > > Secondly, the node should fill up each store regardless of the > configured split point until it reaches the maximum overall store size. > At that time, it should reduce the size of whichever store is over its > allocation as and when the other store needs it. > > Bob > I have always wished the store/cache allocation could be configurable. -- I may disagree with what you have to say, but I shall defend, to the death, your right to say it. - Voltaire Those who would give up Liberty, to purchase temporary Safety, deserve neither Liberty nor Safety. - Ben Franklin From toad at amphibian.dyndns.org Thu May 3 22:29:56 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Thu, 3 May 2007 23:29:56 +0100 Subject: [Tech] Cache and Store In-Reply-To: <1178210627.10595.24.camel@localhost> References: <1178205708.10595.14.camel@localhost> <463A0798.5090506@cs.ucl.ac.uk> <1178210627.10595.24.camel@localhost> Message-ID: <20070503222956.GK24370@amphibian.dyndns.org> On Thu, May 03, 2007 at 05:43:46PM +0100, Bob Ham wrote: > On Thu, 2007-05-03 at 17:02 +0100, Michael Rogers wrote: > > Bob Ham wrote: > > > ... So the cache tends to contain data that's popular at the > > moment, and the store tends to contain data that's close to your node's > > location. > > What is a node's "location"? > > > > > Also, I understand that the configured store size limit is split exactly > > > 50:50 between the two. This is a problem. Only a very small percentage > > > of the Store is in use, but the Cache is nearing full. > > > > The store takes a lot longer to fill up, but once it's full it should > > provide better long-term storage (less redundancy between nodes). > > Sure. The issue is the loss of potential storage space in the time it > takes to fill up. What I propose is two things: firstly, the 50:50 > split seems pretty arbitrary; it should be configurable. > > Secondly, the node should fill up each store regardless of the > configured split point until it reaches the maximum overall store size. > At that time, it should reduce the size of whichever store is over its > allocation as and when the other store needs it. It's not easy to do online shrinking without ending up losing the most valuable rather than the least valuable data. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070503/d91341e0/attachment.pgp From rah at bash.sh Fri May 4 07:43:43 2007 From: rah at bash.sh (Bob Ham) Date: Fri, 04 May 2007 08:43:43 +0100 Subject: [Tech] Cache and Store In-Reply-To: <20070503222956.GK24370@amphibian.dyndns.org> References: <1178205708.10595.14.camel@localhost> <463A0798.5090506@cs.ucl.ac.uk> <1178210627.10595.24.camel@localhost> <20070503222956.GK24370@amphibian.dyndns.org> Message-ID: <1178264623.3889.0.camel@localhost> On Thu, 2007-05-03 at 23:29 +0100, Matthew Toseland wrote: > On Thu, May 03, 2007 at 05:43:46PM +0100, Bob Ham wrote: > > Secondly, the node should fill up each store regardless of the > > configured split point until it reaches the maximum overall store size. > > At that time, it should reduce the size of whichever store is over its > > allocation as and when the other store needs it. > > It's not easy to do online shrinking without ending up losing the most > valuable rather than the least valuable data. What do you mean by "easy", exactly? Bob -- Bob Ham From toad at amphibian.dyndns.org Fri May 4 12:21:48 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Fri, 4 May 2007 13:21:48 +0100 Subject: [Tech] Cache and Store In-Reply-To: <1178264623.3889.0.camel@localhost> References: <1178205708.10595.14.camel@localhost> <463A0798.5090506@cs.ucl.ac.uk> <1178210627.10595.24.camel@localhost> <20070503222956.GK24370@amphibian.dyndns.org> <1178264623.3889.0.camel@localhost> Message-ID: <20070504122148.GA3728@amphibian.dyndns.org> On Fri, May 04, 2007 at 08:43:43AM +0100, Bob Ham wrote: > On Thu, 2007-05-03 at 23:29 +0100, Matthew Toseland wrote: > > On Thu, May 03, 2007 at 05:43:46PM +0100, Bob Ham wrote: > > > Secondly, the node should fill up each store regardless of the > > > configured split point until it reaches the maximum overall store size. > > > At that time, it should reduce the size of whichever store is over its > > > allocation as and when the other store needs it. > > > > It's not easy to do online shrinking without ending up losing the most > > valuable rather than the least valuable data. > > What do you mean by "easy", exactly? Well... it's possible, but it's probably best to wait until we get around to doing the drastic datastore changes that we have queued for a while, here's the wishlist: - Include the key for SSKs so that we can reconstruct the SSK store as well as the other two. - Histogram of keys stored/cached by key/location. - An extra layer of encryption: You look up a key by H(H(key+salt)) and decrypt it using H(key+salt). (To make it harder to analyse confiscated datastores). Optional as incompatible with the previous item. - The "client cache" (ephemerally encrypted cache of only stuff requested by the local user). - Temporary files allocated from chains of blocks in the encrypted store. - No free blocks: just move them to the bottom of the LRU. - One big file for all stores for a given keysize (e.g. CHKs plus tempfiles, store and cache). Or alternatively keep the data in the database (would be simpler but less reliable based on past experience as no way to recover). Most of these have bugs filed. At the moment none of them is a priority, but if you want to work on them that'd be great. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070504/9b827ca3/attachment.pgp From rah at bash.sh Fri May 4 12:44:41 2007 From: rah at bash.sh (Bob Ham) Date: Fri, 04 May 2007 13:44:41 +0100 Subject: [Tech] Cache and Store In-Reply-To: <20070504122148.GA3728@amphibian.dyndns.org> References: <1178205708.10595.14.camel@localhost> <463A0798.5090506@cs.ucl.ac.uk> <1178210627.10595.24.camel@localhost> <20070503222956.GK24370@amphibian.dyndns.org> <1178264623.3889.0.camel@localhost> <20070504122148.GA3728@amphibian.dyndns.org> Message-ID: <1178282681.3889.3.camel@localhost> On Fri, 2007-05-04 at 13:21 +0100, Matthew Toseland wrote: > On Fri, May 04, 2007 at 08:43:43AM +0100, Bob Ham wrote: > > On Thu, 2007-05-03 at 23:29 +0100, Matthew Toseland wrote: > > > On Thu, May 03, 2007 at 05:43:46PM +0100, Bob Ham wrote: > > > > Secondly, the node should fill up each store regardless of the > > > > configured split point until it reaches the maximum overall store size. > > > > At that time, it should reduce the size of whichever store is over its > > > > allocation as and when the other store needs it. > > > > > > It's not easy to do online shrinking without ending up losing the most > > > valuable rather than the least valuable data. > > > > What do you mean by "easy", exactly? > > Well... it's possible, but it's probably best to wait until we get > around to doing the drastic datastore changes that we have queued for a > while, here's the wishlist: > ... > Most of these have bugs filed. At the moment none of them is a priority, > but if you want to work on them that'd be great. Woah there cowboy! We can't discuss working on bugs or implementation of wish list items here! I'm afraid you're going to have to resend your email to the -devl list so that I can read it there instead of here. From m.rogers at cs.ucl.ac.uk Fri May 4 13:00:34 2007 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Fri, 04 May 2007 14:00:34 +0100 Subject: [Tech] Cache and Store In-Reply-To: <1178210627.10595.24.camel@localhost> References: <1178205708.10595.14.camel@localhost> <463A0798.5090506@cs.ucl.ac.uk> <1178210627.10595.24.camel@localhost> Message-ID: <463B2E72.7000708@cs.ucl.ac.uk> Bob Ham wrote: > What is a node's "location"? Every node has a location between 0 and 1 - think of it as a position on a circle. Keys and node locations can be mapped onto the same circle, which makes it possible to say which of a given set of nodes is closest to a given key. That's the basis for Freenet's greedy routing algorithm. > the 50:50 > split seems pretty arbitrary; it should be configurable. Why, so each user can choose a different arbitrary value instead of everyone using the same arbitrary value? Cheers, Michael From rah at bash.sh Fri May 4 13:24:26 2007 From: rah at bash.sh (Bob Ham) Date: Fri, 04 May 2007 14:24:26 +0100 Subject: [Tech] Cache and Store In-Reply-To: <463B2E72.7000708@cs.ucl.ac.uk> References: <1178205708.10595.14.camel@localhost> <463A0798.5090506@cs.ucl.ac.uk> <1178210627.10595.24.camel@localhost> <463B2E72.7000708@cs.ucl.ac.uk> Message-ID: <1178285066.3889.12.camel@localhost> On Fri, 2007-05-04 at 14:00 +0100, Michael Rogers wrote: > Bob Ham wrote: > > What is a node's "location"? > > Every node has a location between 0 and 1 Does the location have any meaning? > > the 50:50 > > split seems pretty arbitrary; it should be configurable. > > Why, so each user can choose a different arbitrary value instead of > everyone using the same arbitrary value? Yes, that is what I said. Bob From m.rogers at cs.ucl.ac.uk Fri May 4 17:46:06 2007 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Fri, 04 May 2007 18:46:06 +0100 Subject: [Tech] Cache and Store In-Reply-To: <1178285066.3889.12.camel@localhost> References: <1178205708.10595.14.camel@localhost> <463A0798.5090506@cs.ucl.ac.uk> <1178210627.10595.24.camel@localhost> <463B2E72.7000708@cs.ucl.ac.uk> <1178285066.3889.12.camel@localhost> Message-ID: <463B715E.3020101@cs.ucl.ac.uk> Bob Ham wrote: > Does the location have any meaning? The initial location is chosen at random, then nodes swap locations to produce a layout that will be efficient for greedy routing. For more information see the Defcon slides and Oskar Sandberg's paper: http://www.math.chalmers.se/~ossa/defcon13/ http://www.math.chalmers.se/~ossa/swroute.pdf Cheers, Michael From juiceman69 at gmail.com Fri May 4 23:57:05 2007 From: juiceman69 at gmail.com (Juiceman) Date: Fri, 4 May 2007 19:57:05 -0400 Subject: [Tech] Cache and Store In-Reply-To: <463B2E72.7000708@cs.ucl.ac.uk> References: <1178205708.10595.14.camel@localhost> <463A0798.5090506@cs.ucl.ac.uk> <1178210627.10595.24.camel@localhost> <463B2E72.7000708@cs.ucl.ac.uk> Message-ID: <8b525dee0705041657u90e16c7j2caca44a1a46483f@mail.gmail.com> On 5/4/07, Michael Rogers wrote: > Bob Ham wrote: > > What is a node's "location"? > > Every node has a location between 0 and 1 - think of it as a position on > a circle. Keys and node locations can be mapped onto the same circle, > which makes it possible to say which of a given set of nodes is closest > to a given key. That's the basis for Freenet's greedy routing algorithm. > > > the 50:50 > > split seems pretty arbitrary; it should be configurable. > > Why, so each user can choose a different arbitrary value instead of > everyone using the same arbitrary value? > Well, I for one, would like to be a "deep cache" for long term storage, others like Bob would like to cache as much as possible immediately. I think Freenet could benefit from this. -- I may disagree with what you have to say, but I shall defend, to the death, your right to say it. - Voltaire Those who would give up Liberty, to purchase temporary Safety, deserve neither Liberty nor Safety. - Ben Franklin From rah at bash.sh Sat May 5 20:41:10 2007 From: rah at bash.sh (Bob Ham) Date: Sat, 05 May 2007 21:41:10 +0100 Subject: [Tech] Cache and Store In-Reply-To: <463B715E.3020101@cs.ucl.ac.uk> References: <1178205708.10595.14.camel@localhost> <463A0798.5090506@cs.ucl.ac.uk> <1178210627.10595.24.camel@localhost> <463B2E72.7000708@cs.ucl.ac.uk> <1178285066.3889.12.camel@localhost> <463B715E.3020101@cs.ucl.ac.uk> Message-ID: <1178397670.9397.0.camel@orchid.arb.net> On Fri, 2007-05-04 at 18:46 +0100, Michael Rogers wrote: > Bob Ham wrote: > > Does the location have any meaning? > > The initial location is chosen at random, then nodes swap locations to > produce a layout that will be efficient for greedy routing. For more > information see the Defcon slides and Oskar Sandberg's paper: > > http://www.math.chalmers.se/~ossa/defcon13/ > http://www.math.chalmers.se/~ossa/swroute.pdf Thanks Bob -- Bob Ham -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070505/59f61a72/attachment.pgp From m.rogers at cs.ucl.ac.uk Wed May 9 11:03:04 2007 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Wed, 09 May 2007 12:03:04 +0100 Subject: [Tech] Cache and Store In-Reply-To: <8b525dee0705041657u90e16c7j2caca44a1a46483f@mail.gmail.com> References: <1178205708.10595.14.camel@localhost> <463A0798.5090506@cs.ucl.ac.uk> <1178210627.10595.24.camel@localhost> <463B2E72.7000708@cs.ucl.ac.uk> <8b525dee0705041657u90e16c7j2caca44a1a46483f@mail.gmail.com> Message-ID: <4641AA68.3030205@cs.ucl.ac.uk> Juiceman wrote: >>> the 50:50 >>> split seems pretty arbitrary; it should be configurable. >> Why, so each user can choose a different arbitrary value instead of >> everyone using the same arbitrary value? >> > > Well, I for one, would like to be a "deep cache" for long term > storage, others like Bob would like to cache as much as possible > immediately. I think Freenet could benefit from this. Would it be possible to dynamically balance the amount of space allocated to the cache and store? Here's what I have in mind: when the cache is below its quota and a key is cached, remove the least recently used key from the store instead of the cache. And vice versa: when the store is below its quota and a key is stored, remove the least recently used key from the cache. That way we don't waste any space while the store is filling up (the cache uses the free space), and if we later change the quotas it will gradually move to the new quotas instead of having to resize the whole thing at once. On a related note, is there any evidence that LRU performs better than, say, random replacement for Freenet's purposes? IIRC there's some evidence that LRU in a distributed cache produces too many copies of popular data: http://www.cs.princeton.edu/~qlv/download/searchp2p_ics02.pdf http://gnunet.org/papers/p2pmulti.pdf Cheers, Michael From toad at amphibian.dyndns.org Wed May 9 15:27:31 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Wed, 9 May 2007 16:27:31 +0100 Subject: [Tech] Cache and Store In-Reply-To: <4641AA68.3030205@cs.ucl.ac.uk> References: <1178205708.10595.14.camel@localhost> <8b525dee0705041657u90e16c7j2caca44a1a46483f@mail.gmail.com> <4641AA68.3030205@cs.ucl.ac.uk> Message-ID: <200705091627.32055.toad@amphibian.dyndns.org> On Wednesday 09 May 2007 12:03, Michael Rogers wrote: > Juiceman wrote: > >>> the 50:50 > >>> split seems pretty arbitrary; it should be configurable. > >> > >> Why, so each user can choose a different arbitrary value instead of > >> everyone using the same arbitrary value? > > > > Well, I for one, would like to be a "deep cache" for long term > > storage, others like Bob would like to cache as much as possible > > immediately. I think Freenet could benefit from this. > > Would it be possible to dynamically balance the amount of space > allocated to the cache and store? Here's what I have in mind: when the > cache is below its quota and a key is cached, remove the least recently > used key from the store instead of the cache. And vice versa: when the > store is below its quota and a key is stored, remove the least recently > used key from the cache. That way we don't waste any space while the > store is filling up (the cache uses the free space), and if we later > change the quotas it will gradually move to the new quotas instead of > having to resize the whole thing at once. Yes, it's possible. File a bug for it; there are definitely related bugs already filed. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070509/f90e2c40/attachment.pgp From toad at amphibian.dyndns.org Wed May 9 15:26:56 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Wed, 9 May 2007 16:26:56 +0100 Subject: [Tech] Cache and Store In-Reply-To: <4641AA68.3030205@cs.ucl.ac.uk> References: <1178205708.10595.14.camel@localhost> <8b525dee0705041657u90e16c7j2caca44a1a46483f@mail.gmail.com> <4641AA68.3030205@cs.ucl.ac.uk> Message-ID: <200705091626.56460.toad@amphibian.dyndns.org> On Wednesday 09 May 2007 12:03, Michael Rogers wrote: > On a related note, is there any evidence that LRU performs better than, > say, random replacement for Freenet's purposes? IIRC there's some > evidence that LRU in a distributed cache produces too many copies of > popular data: > > http://www.cs.princeton.edu/~qlv/download/searchp2p_ics02.pdf > http://gnunet.org/papers/p2pmulti.pdf No evidence that I know of but feel free to investigate. > > Cheers, > Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070509/fb2928f7/attachment.pgp From rah at bash.sh Wed May 9 19:28:20 2007 From: rah at bash.sh (Bob Ham) Date: Wed, 09 May 2007 20:28:20 +0100 Subject: [Tech] Cache and Store In-Reply-To: <4641AA68.3030205@cs.ucl.ac.uk> References: <1178205708.10595.14.camel@localhost> <463A0798.5090506@cs.ucl.ac.uk> <1178210627.10595.24.camel@localhost> <463B2E72.7000708@cs.ucl.ac.uk> <8b525dee0705041657u90e16c7j2caca44a1a46483f@mail.gmail.com> <4641AA68.3030205@cs.ucl.ac.uk> Message-ID: <1178738901.9397.10.camel@orchid.arb.net> On Wed, 2007-05-09 at 12:03 +0100, Michael Rogers wrote: > Would it be possible to dynamically balance the amount of space > allocated to the cache and store? Here's what I have in mind: when the > cache is below its quota and a key is cached, remove the least recently > used key from the store instead of the cache. And vice versa: when the > store is below its quota and a key is stored, remove the least recently > used key from the cache. That was what I proposed near the start of this thread. I would note, as well, that the store-shrinking code should already exist for cases when the user reduces the configured size of the store. Bob -- Bob Ham -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070509/db78a924/attachment.pgp From toad at amphibian.dyndns.org Wed May 9 19:38:27 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Wed, 9 May 2007 20:38:27 +0100 Subject: [Tech] Captured topology data; ossa2007 In-Reply-To: <823242bd0705091113o5589b379lbfd602f66b77547c@mail.gmail.com> References: <200705091257.43613.toad@amphibian.dyndns.org> <823242bd0705091113o5589b379lbfd602f66b77547c@mail.gmail.com> Message-ID: <200705092038.27348.toad@amphibian.dyndns.org> On Wednesday 09 May 2007 19:13, you wrote: > Well, what does the data say? Does it support or refute any of these > theories? Well, there is a spike at 0.2 on my peers and my intercepted swap locations. This has been true for a while; before the network reset it was at 0.0. Having said that it's not that big a spike any more. But the intercepted swap data only goes out to 6 hops. That leaves probe requests. Which continue to be perverse, but I think we have some chance of investigating with the new recorded data: INFO | jvm 1 | 2007/05/08 20:21:43 | LOCATION 0: 1.3539134879558823E-5 - estimated nodes: 73859.96290721533 (62 hops) INFO | jvm 1 | 2007/05/08 20:27:31 | LOCATION 1: 2.361622498392446E-5 - estimated nodes: 84687.54008574181 (73 hops) INFO | jvm 1 | 2007/05/08 20:32:56 | LOCATION 2: 0.004470353498917001 - estimated nodes: 671.0878682696541 (40 hops) ... INFO | jvm 1 | 2007/05/09 20:36:29 | LOCATION 247: 0.9916986858272543 - estimated nodes: 250.07595910356943 (42 hops) Clearly something is wrong here. It might very well be routing that is wrong. I'll have a look at the detailed probe logs. We might have to introduce even more topology monitoring mechanisms. :( > > Ian. > > On 5/9/07, Matthew Toseland wrote: > > 1. We have added code to accurately capture the long-range network > > topology by monitoring swap requests (and adding a per node uid for each > > location mentioned), and through probe requests (a request routed to a > > location, which returns info such as how close it got to the location, > > how many hops it took etc; we have now added per-hop info on the peers of > > the nodes it passed through). Obviously neither mechanism is perfect: > > swapping only gives us 6 hops, and probe requests rely on working > > routing. > > > > What should we do with this data? We have a few theories: > > - There may be a well-connected core of developer nodes etc, and then > > most of the rest of the network is newbies connected exclusively to > > newbies, no long links. Probably because of the current means for newbies > > to obtain noderefs. - Selective pressure for node locations to cluster > > together: Core nodes get very close together locations (around 0.25 at > > the moment), and newbies get the rest. When a newbie node arrives, if its > > location is close to the core it moves to the core, possibly displacing > > not-so-close locations. If it's not close to the core it stays in newbie > > space. And newbie locations are dumped when the newbies leave. Result: > > Newbies occupy most of the location space, but are fairly sparsely > > distributed; established nodes occupy a very small area, much denser than > > the newbies' area. Before the last network reset this was very obvious in > > the locations from the swap data, it's not so obvious any more, so maybe > > it isn't a real effect.. > > > > The basic question here is does #freenet-refs cause serious problems in > > the network topology, and are there other major problems such as location > > clustering as described above. > > > > How do we use the collected data to verify these theories, or get other > > data on the network? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070509/92db0214/attachment.pgp From toad at amphibian.dyndns.org Wed May 9 23:36:39 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Thu, 10 May 2007 00:36:39 +0100 Subject: [Tech] Cache and Store In-Reply-To: <1178738901.9397.10.camel@orchid.arb.net> References: <1178205708.10595.14.camel@localhost> <4641AA68.3030205@cs.ucl.ac.uk> <1178738901.9397.10.camel@orchid.arb.net> Message-ID: <200705100036.40005.toad@amphibian.dyndns.org> On Wednesday 09 May 2007 20:28, Bob Ham wrote: > On Wed, 2007-05-09 at 12:03 +0100, Michael Rogers wrote: > > Would it be possible to dynamically balance the amount of space > > allocated to the cache and store? Here's what I have in mind: when the > > cache is below its quota and a key is cached, remove the least recently > > used key from the store instead of the cache. And vice versa: when the > > store is below its quota and a key is stored, remove the least recently > > used key from the cache. > > That was what I proposed near the start of this thread. I would note, > as well, that the store-shrinking code should already exist for cases > when the user reduces the configured size of the store. > > Bob It does, but as I have already stated at least once, it is difficult to efficiently do an online shrink while preserving the most recently used data. It is of course possible. One way to do it is to swap the key you'd be deleting with the least recently used key just before truncating. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070510/32d35e2a/attachment.pgp From juiceman69 at gmail.com Thu May 10 01:36:49 2007 From: juiceman69 at gmail.com (Juiceman) Date: Wed, 9 May 2007 21:36:49 -0400 Subject: [Tech] Cache and Store In-Reply-To: <200705100036.40005.toad@amphibian.dyndns.org> References: <1178205708.10595.14.camel@localhost> <4641AA68.3030205@cs.ucl.ac.uk> <1178738901.9397.10.camel@orchid.arb.net> <200705100036.40005.toad@amphibian.dyndns.org> Message-ID: <8b525dee0705091836l1cfe3102m5df648f99b963e84@mail.gmail.com> On 5/9/07, Matthew Toseland wrote: > On Wednesday 09 May 2007 20:28, Bob Ham wrote: > > On Wed, 2007-05-09 at 12:03 +0100, Michael Rogers wrote: > > > Would it be possible to dynamically balance the amount of space > > > allocated to the cache and store? Here's what I have in mind: when the > > > cache is below its quota and a key is cached, remove the least recently > > > used key from the store instead of the cache. And vice versa: when the > > > store is below its quota and a key is stored, remove the least recently > > > used key from the cache. > > > > That was what I proposed near the start of this thread. I would note, > > as well, that the store-shrinking code should already exist for cases > > when the user reduces the configured size of the store. > > > > Bob > > It does, but as I have already stated at least once, it is difficult to > efficiently do an online shrink while preserving the most recently used data. > > It is of course possible. One way to do it is to swap the key you'd be > deleting with the least recently used key just before truncating. > > _______________________________________________ > Tech mailing list > Tech at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech > > This is not exactly what you are looking for but I am 90% finished coding a config option to have the store and the cache be non-equal and user configurable in size. This could be changed manually as often as you wished as your store filled up. I was going to run it by the list after I had completed it. This was something that I wanted to see if I could figure out how to do (whether or not it gets merged into the codebase.) -- I may disagree with what you have to say, but I shall defend, to the death, your right to say it. - Voltaire Those who would give up Liberty, to purchase temporary Safety, deserve neither Liberty nor Safety. - Ben Franklin From toad at amphibian.dyndns.org Thu May 10 10:31:22 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Thu, 10 May 2007 11:31:22 +0100 Subject: [Tech] Cache and Store In-Reply-To: <8b525dee0705091836l1cfe3102m5df648f99b963e84@mail.gmail.com> References: <1178205708.10595.14.camel@localhost> <200705100036.40005.toad@amphibian.dyndns.org> <8b525dee0705091836l1cfe3102m5df648f99b963e84@mail.gmail.com> Message-ID: <200705101131.28453.toad@amphibian.dyndns.org> On Thursday 10 May 2007 02:36, Juiceman wrote: > On 5/9/07, Matthew Toseland wrote: > > This is not exactly what you are looking for but I am 90% finished > coding a config option to have the store and the cache be non-equal > and user configurable in size. This could be changed manually as > often as you wished as your store filled up. > > I was going to run it by the list after I had completed it. > > This was something that I wanted to see if I could figure out how to > do (whether or not it gets merged into the codebase.) As long as it's an expert option, I don't see any reason why it shouldn't be accepted. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070510/66c6417d/attachment.pgp From nextgens at freenetproject.org Thu May 10 10:52:16 2007 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Thu, 10 May 2007 12:52:16 +0200 Subject: [Tech] Cache and Store In-Reply-To: <200705101131.28453.toad@amphibian.dyndns.org> References: <1178205708.10595.14.camel@localhost> <200705100036.40005.toad@amphibian.dyndns.org> <8b525dee0705091836l1cfe3102m5df648f99b963e84@mail.gmail.com> <200705101131.28453.toad@amphibian.dyndns.org> Message-ID: <20070510105216.GE5480@freenetproject.org> * Matthew Toseland [2007-05-10 11:31:22]: > On Thursday 10 May 2007 02:36, Juiceman wrote: > > On 5/9/07, Matthew Toseland wrote: > > > > This is not exactly what you are looking for but I am 90% finished > > coding a config option to have the store and the cache be non-equal > > and user configurable in size. This could be changed manually as > > often as you wished as your store filled up. > > > > I was going to run it by the list after I had completed it. > > > > This was something that I wanted to see if I could figure out how to > > do (whether or not it gets merged into the codebase.) > > As long as it's an expert option, I don't see any reason why it shouldn't be > accepted. Preventing users from their stupidity ? We don't want everyone to have a cache but no specialized data on the basis that "with a bigger cache downloads are 'resuming' ; I don't care about others nor the network so I only cache" NextGen$ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070510/65c66683/attachment.pgp From toad at amphibian.dyndns.org Thu May 10 11:53:31 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Thu, 10 May 2007 12:53:31 +0100 Subject: [Tech] Cache and Store In-Reply-To: <20070510105216.GE5480@freenetproject.org> References: <1178205708.10595.14.camel@localhost> <200705101131.28453.toad@amphibian.dyndns.org> <20070510105216.GE5480@freenetproject.org> Message-ID: <200705101253.39094.toad@amphibian.dyndns.org> On Thursday 10 May 2007 11:52, Florent Daigni?re wrote: > * Matthew Toseland [2007-05-10 11:31:22]: > > > > As long as it's an expert option, I don't see any reason why it shouldn't > > be accepted. > > Preventing users from their stupidity ? > > We don't want everyone to have a cache but no specialized data on the > basis that "with a bigger cache downloads are 'resuming' ; I don't care > about others nor the network so I only cache" Fair point. If they want to break their node that much they can maintain a fork. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070510/a4bc90fe/attachment.pgp From rah at bash.sh Thu May 10 20:38:58 2007 From: rah at bash.sh (Bob Ham) Date: Thu, 10 May 2007 21:38:58 +0100 Subject: [Tech] Cache and Store In-Reply-To: <200705100036.40005.toad@amphibian.dyndns.org> References: <1178205708.10595.14.camel@localhost> <4641AA68.3030205@cs.ucl.ac.uk> <1178738901.9397.10.camel@orchid.arb.net> <200705100036.40005.toad@amphibian.dyndns.org> Message-ID: <1178829538.9397.11.camel@orchid.arb.net> On Thu, 2007-05-10 at 00:36 +0100, Matthew Toseland wrote: > On Wednesday 09 May 2007 20:28, Bob Ham wrote: > > That was what I proposed near the start of this thread. I would note, > > as well, that the store-shrinking code should already exist for cases > > when the user reduces the configured size of the store. > It does, but as I have already stated at least once, it is difficult to > efficiently do an online shrink while preserving the most recently used data. > > It is of course possible. One way to do it is to swap the key you'd be > deleting with the least recently used key just before truncating. What's your point? -- Bob Ham -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070510/bb74fe1d/attachment.pgp From rah at bash.sh Thu May 10 20:45:59 2007 From: rah at bash.sh (Bob Ham) Date: Thu, 10 May 2007 21:45:59 +0100 Subject: [Tech] Cache and Store In-Reply-To: <200705101253.39094.toad@amphibian.dyndns.org> References: <1178205708.10595.14.camel@localhost> <200705101131.28453.toad@amphibian.dyndns.org> <20070510105216.GE5480@freenetproject.org> <200705101253.39094.toad@amphibian.dyndns.org> Message-ID: <1178829959.9397.21.camel@orchid.arb.net> On Thu, 2007-05-10 at 12:53 +0100, Matthew Toseland wrote: > On Thursday 10 May 2007 11:52, Florent Daigni?re wrote: > > * Matthew Toseland [2007-05-10 11:31:22]: > > > > > > As long as it's an expert option, I don't see any reason why it shouldn't > > > be accepted. > > > > Preventing users from their stupidity ? > > > > We don't want everyone to have a cache but no specialized data on the > > basis that "with a bigger cache downloads are 'resuming' ; I don't care > > about others nor the network so I only cache" > > Fair point. If they want to break their node that much they can maintain a > fork. This is a real cognitive problem showing up right there. It isn't your responsibility to second-guess the user. There are valid reasons for the node to have this functionality. The only reason for it not to is to inhibit users. That's what Microsoft do. Bob -- Bob Ham -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070510/62cac8e2/attachment.pgp From nextgens at freenetproject.org Thu May 10 20:47:39 2007 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Thu, 10 May 2007 22:47:39 +0200 Subject: [Tech] Cache and Store In-Reply-To: <1178829959.9397.21.camel@orchid.arb.net> References: <1178205708.10595.14.camel@localhost> <200705101131.28453.toad@amphibian.dyndns.org> <20070510105216.GE5480@freenetproject.org> <200705101253.39094.toad@amphibian.dyndns.org> <1178829959.9397.21.camel@orchid.arb.net> Message-ID: <20070510204737.GG5480@freenetproject.org> * Bob Ham [2007-05-10 21:45:59]: > On Thu, 2007-05-10 at 12:53 +0100, Matthew Toseland wrote: > > On Thursday 10 May 2007 11:52, Florent Daigni?re wrote: > > > * Matthew Toseland [2007-05-10 11:31:22]: > > > > > > > > As long as it's an expert option, I don't see any reason why it shouldn't > > > > be accepted. > > > > > > Preventing users from their stupidity ? > > > > > > We don't want everyone to have a cache but no specialized data on the > > > basis that "with a bigger cache downloads are 'resuming' ; I don't care > > > about others nor the network so I only cache" > > > > Fair point. If they want to break their node that much they can maintain a > > fork. > > This is a real cognitive problem showing up right there. It isn't your > responsibility to second-guess the user. There are valid reasons for > the node to have this functionality. The only reason for it not to is > to inhibit users. That's what Microsoft do. Indeed... and experience has shown that it works. NextGen$ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070510/9ece8195/attachment.pgp From toad at amphibian.dyndns.org Thu May 10 21:20:52 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Thu, 10 May 2007 22:20:52 +0100 Subject: [Tech] Cache and Store In-Reply-To: <1178829538.9397.11.camel@orchid.arb.net> References: <1178205708.10595.14.camel@localhost> <200705100036.40005.toad@amphibian.dyndns.org> <1178829538.9397.11.camel@orchid.arb.net> Message-ID: <200705102221.00628.toad@amphibian.dyndns.org> On Thursday 10 May 2007 21:38, Bob Ham wrote: > On Thu, 2007-05-10 at 00:36 +0100, Matthew Toseland wrote: > > On Wednesday 09 May 2007 20:28, Bob Ham wrote: > > > That was what I proposed near the start of this thread. I would note, > > > as well, that the store-shrinking code should already exist for cases > > > when the user reduces the configured size of the store. > > > > It does, but as I have already stated at least once, it is difficult to > > efficiently do an online shrink while preserving the most recently used > > data. > > > > It is of course possible. One way to do it is to swap the key you'd be > > deleting with the least recently used key just before truncating. > > What's your point? The point is that as presently implemented we have two ways to shrink a store: - Online, fast, doesn't preserve most valuable data. - Offline, slow, does preserve most valuable data. Obviously the latter is preferred but we can only do it on startup. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070510/70c0b8c0/attachment.pgp From juiceman69 at gmail.com Fri May 11 00:15:50 2007 From: juiceman69 at gmail.com (Juiceman) Date: Thu, 10 May 2007 20:15:50 -0400 Subject: [Tech] Cache and Store In-Reply-To: <200705101253.39094.toad@amphibian.dyndns.org> References: <1178205708.10595.14.camel@localhost> <200705101131.28453.toad@amphibian.dyndns.org> <20070510105216.GE5480@freenetproject.org> <200705101253.39094.toad@amphibian.dyndns.org> Message-ID: <8b525dee0705101715w55434d04k2c29eda8d59ba5a6@mail.gmail.com> On 5/10/07, Matthew Toseland wrote: > On Thursday 10 May 2007 11:52, Florent Daigni?re wrote: > > * Matthew Toseland [2007-05-10 11:31:22]: > > > > > > As long as it's an expert option, I don't see any reason why it shouldn't > > > be accepted. > > > > Preventing users from their stupidity ? > > > > We don't want everyone to have a cache but no specialized data on the > > basis that "with a bigger cache downloads are 'resuming' ; I don't care > > about others nor the network so I only cache" > > Fair point. If they want to break their node that much they can maintain a > fork. > I would make it an expert option and put in sanity checks to make sure the ratio never goes outside 10 to 1 and each store is at least 32MB. The default would still be 50% My reasoning is, I run a 18GB datastore and there is no way my node could serve up 9GB of cached data without overloading my bandwidth and probably starving requests that could hit the specialized store. What do you think? -- I may disagree with what you have to say, but I shall defend, to the death, your right to say it. - Voltaire Those who would give up Liberty, to purchase temporary Safety, deserve neither Liberty nor Safety. - Ben Franklin From rah at bash.sh Fri May 11 07:19:49 2007 From: rah at bash.sh (Bob Ham) Date: Fri, 11 May 2007 08:19:49 +0100 Subject: [Tech] Cache and Store In-Reply-To: <20070510204737.GG5480@freenetproject.org> References: <1178205708.10595.14.camel@localhost> <200705101131.28453.toad@amphibian.dyndns.org> <20070510105216.GE5480@freenetproject.org> <200705101253.39094.toad@amphibian.dyndns.org> <1178829959.9397.21.camel@orchid.arb.net> <20070510204737.GG5480@freenetproject.org> Message-ID: <1178867989.9397.23.camel@orchid.arb.net> On Thu, 2007-05-10 at 22:47 +0200, Florent Daigni?re wrote: > * Bob Ham [2007-05-10 21:45:59]: > > > On Thu, 2007-05-10 at 12:53 +0100, Matthew Toseland wrote: > > > On Thursday 10 May 2007 11:52, Florent Daigni?re wrote: > > > > * Matthew Toseland [2007-05-10 11:31:22]: > > > > > > > > > > As long as it's an expert option, I don't see any reason why it shouldn't > > > > > be accepted. > > > > > > > > Preventing users from their stupidity ? > > > > > > > > We don't want everyone to have a cache but no specialized data on the > > > > basis that "with a bigger cache downloads are 'resuming' ; I don't care > > > > about others nor the network so I only cache" > > > > > > Fair point. If they want to break their node that much they can maintain a > > > fork. > > > > This is a real cognitive problem showing up right there. It isn't your > > responsibility to second-guess the user. There are valid reasons for > > the node to have this functionality. The only reason for it not to is > > to inhibit users. That's what Microsoft do. > > Indeed... and experience has shown that it works. What do you mean "it works"? What does it work to do? Bob -- Bob Ham -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070511/3a6faeb5/attachment.pgp From rah at bash.sh Fri May 11 07:24:54 2007 From: rah at bash.sh (Bob Ham) Date: Fri, 11 May 2007 08:24:54 +0100 Subject: [Tech] Cache and Store In-Reply-To: <200705102221.00628.toad@amphibian.dyndns.org> References: <1178205708.10595.14.camel@localhost> <200705100036.40005.toad@amphibian.dyndns.org> <1178829538.9397.11.camel@orchid.arb.net> <200705102221.00628.toad@amphibian.dyndns.org> Message-ID: <1178868294.9397.27.camel@orchid.arb.net> On Thu, 2007-05-10 at 22:20 +0100, Matthew Toseland wrote: > On Thursday 10 May 2007 21:38, Bob Ham wrote: > > On Thu, 2007-05-10 at 00:36 +0100, Matthew Toseland wrote: > > > On Wednesday 09 May 2007 20:28, Bob Ham wrote: > > > > That was what I proposed near the start of this thread. I would note, > > > > as well, that the store-shrinking code should already exist for cases > > > > when the user reduces the configured size of the store. > > > > > > It does, but as I have already stated at least once, it is difficult to > > > efficiently do an online shrink while preserving the most recently used > > > data. > > > > > > It is of course possible. One way to do it is to swap the key you'd be > > > deleting with the least recently used key just before truncating. > > > > What's your point? > Obviously the latter is preferred but we can only do it on startup. I'm still unsure as to why you're telling us this. Is your point that work still needs to be done on store shrinking? Bob -- Bob Ham -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070511/2f6045c3/attachment.pgp From nextgens at freenetproject.org Fri May 11 10:42:19 2007 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Fri, 11 May 2007 12:42:19 +0200 Subject: [Tech] Cache and Store In-Reply-To: <1178867989.9397.23.camel@orchid.arb.net> References: <1178205708.10595.14.camel@localhost> <200705101131.28453.toad@amphibian.dyndns.org> <20070510105216.GE5480@freenetproject.org> <200705101253.39094.toad@amphibian.dyndns.org> <1178829959.9397.21.camel@orchid.arb.net> <20070510204737.GG5480@freenetproject.org> <1178867989.9397.23.camel@orchid.arb.net> Message-ID: <20070511104218.GA5613@freenetproject.org> * Bob Ham [2007-05-11 08:19:49]: > On Thu, 2007-05-10 at 22:47 +0200, Florent Daigni?re wrote: > > * Bob Ham [2007-05-10 21:45:59]: > > > > > On Thu, 2007-05-10 at 12:53 +0100, Matthew Toseland wrote: > > > > On Thursday 10 May 2007 11:52, Florent Daigni?re wrote: > > > > > * Matthew Toseland [2007-05-10 11:31:22]: > > > > > > > > > > > > As long as it's an expert option, I don't see any reason why it shouldn't > > > > > > be accepted. > > > > > > > > > > Preventing users from their stupidity ? > > > > > > > > > > We don't want everyone to have a cache but no specialized data on the > > > > > basis that "with a bigger cache downloads are 'resuming' ; I don't care > > > > > about others nor the network so I only cache" > > > > > > > > Fair point. If they want to break their node that much they can maintain a > > > > fork. > > > > > > This is a real cognitive problem showing up right there. It isn't your > > > responsibility to second-guess the user. There are valid reasons for > > > the node to have this functionality. The only reason for it not to is > > > to inhibit users. That's what Microsoft do. > > > > Indeed... and experience has shown that it works. > > What do you mean "it works"? What does it work to do? > Shall I remind you that most of our users do use Microsoft products and are happy with them ? I'm against the introduction of new useless (no need to debate the meaning of useless this time; show me some stats/simulations if you want to be heard) settings, especially when they can harm the network... So far we don't have content reachability problems and if we did, I would suggest to fix the datastore code or to prevent users from nuking it on a regular basis. The 50%-50% limit is arbitrary, so are many "unknown" and "hidden" settings in the node; unless you have strong evidence (with a theorical basis) explaining why it's bad/wrong you will be directed to the bug tracking system and the "wishlist" category. I will ask you to explain the "This is a problem." in your first mail [1] as well (so that we could at least agree on the fact that we don't have the same definition of what a problem is either ;) ). Moreover some related work has to be done on the shrinking code, mroggers has suggested that the LRU policy might not be the best solution to choose which data will be pruned; he came with some references and is likely to be heard... You didn't, did you ? Anyway, consequently his suggestion is likely to be implemented before any other "related" work is done in that area of the code. IMO we shouldn't give users incentives to use a "maybe broken and harmful" schrinker, hence I'm against letting him play with the ratio of the cache/store. NextGen$ [1] http://archives.freenetproject.org/message/20070503.152148.95c943dd.en.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070511/d058e73c/attachment.pgp From rah at bash.sh Fri May 11 13:08:32 2007 From: rah at bash.sh (Bob Ham) Date: Fri, 11 May 2007 14:08:32 +0100 Subject: [Tech] Cache and Store In-Reply-To: <20070511104218.GA5613@freenetproject.org> References: <1178205708.10595.14.camel@localhost> <200705101131.28453.toad@amphibian.dyndns.org> <20070510105216.GE5480@freenetproject.org> <200705101253.39094.toad@amphibian.dyndns.org> <1178829959.9397.21.camel@orchid.arb.net> <20070510204737.GG5480@freenetproject.org> <1178867989.9397.23.camel@orchid.arb.net> <20070511104218.GA5613@freenetproject.org> Message-ID: <1178888912.3916.4.camel@localhost> On Fri, 2007-05-11 at 12:42 +0200, Florent Daigni?re wrote: > * Bob Ham [2007-05-11 08:19:49]: > > > On Thu, 2007-05-10 at 22:47 +0200, Florent Daigni?re wrote: > > > * Bob Ham [2007-05-10 21:45:59]: > > > > > > > This is a real cognitive problem showing up right there. It isn't your > > > > responsibility to second-guess the user. There are valid reasons for > > > > the node to have this functionality. The only reason for it not to is > > > > to inhibit users. That's what Microsoft do. > > > > > > Indeed... and experience has shown that it works. > > > > What do you mean "it works"? What does it work to do? > > > > Shall I remind you that > ... > ... > ... > ... > ... > ... > ... > hence > I'm against letting him play with the ratio of the cache/store. You didn't answer the question. What do you mean "it works"? Bob From nextgens at freenetproject.org Fri May 11 13:25:25 2007 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Fri, 11 May 2007 15:25:25 +0200 Subject: [Tech] Cache and Store In-Reply-To: <1178888912.3916.4.camel@localhost> References: <1178205708.10595.14.camel@localhost> <200705101131.28453.toad@amphibian.dyndns.org> <20070510105216.GE5480@freenetproject.org> <200705101253.39094.toad@amphibian.dyndns.org> <1178829959.9397.21.camel@orchid.arb.net> <20070510204737.GG5480@freenetproject.org> <1178867989.9397.23.camel@orchid.arb.net> <20070511104218.GA5613@freenetproject.org> <1178888912.3916.4.camel@localhost> Message-ID: <20070511132525.GC5645@freenetproject.org> * Bob Ham [2007-05-11 14:08:32]: > On Fri, 2007-05-11 at 12:42 +0200, Florent Daigni?re wrote: > > * Bob Ham [2007-05-11 08:19:49]: > > > > > On Thu, 2007-05-10 at 22:47 +0200, Florent Daigni?re wrote: > > > > * Bob Ham [2007-05-10 21:45:59]: > > > > > > > > > This is a real cognitive problem showing up right there. It isn't your > > > > > responsibility to second-guess the user. There are valid reasons for > > > > > the node to have this functionality. The only reason for it not to is > > > > > to inhibit users. That's what Microsoft do. > > > > > > > > Indeed... and experience has shown that it works. > > > > > > What do you mean "it works"? What does it work to do? > > > > > > > Shall I remind you that > > ... > > ... > > ... > > ... > > ... > > ... > > ... > > hence > > I'm against letting him play with the ratio of the cache/store. > > You didn't answer the question. What do you mean "it works"? I did in the part you've stripped. Inhibiting users seems to be something that most of them like. Feel free to explain to me why the lack of that feature is a problem now. NextGen$ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070511/6c65fb80/attachment.pgp From rah at bash.sh Fri May 11 15:40:44 2007 From: rah at bash.sh (Bob Ham) Date: Fri, 11 May 2007 16:40:44 +0100 Subject: [Tech] Cache and Store In-Reply-To: <20070511132525.GC5645@freenetproject.org> References: <1178205708.10595.14.camel@localhost> <200705101131.28453.toad@amphibian.dyndns.org> <20070510105216.GE5480@freenetproject.org> <200705101253.39094.toad@amphibian.dyndns.org> <1178829959.9397.21.camel@orchid.arb.net> <20070510204737.GG5480@freenetproject.org> <1178867989.9397.23.camel@orchid.arb.net> <20070511104218.GA5613@freenetproject.org> <1178888912.3916.4.camel@localhost> <20070511132525.GC5645@freenetproject.org> Message-ID: <1178898044.17619.5.camel@orchid.arb.net> On Fri, 2007-05-11 at 15:25 +0200, Florent Daigni?re wrote: > * Bob Ham [2007-05-11 14:08:32]: > > > On Fri, 2007-05-11 at 12:42 +0200, Florent Daigni?re wrote: > > > * Bob Ham [2007-05-11 08:19:49]: > > > > > > > On Thu, 2007-05-10 at 22:47 +0200, Florent Daigni?re wrote: > > > > > * Bob Ham [2007-05-10 21:45:59]: > > > > > > > > > > > This is a real cognitive problem showing up right there. It isn't your > > > > > > responsibility to second-guess the user. There are valid reasons for > > > > > > the node to have this functionality. The only reason for it not to is > > > > > > to inhibit users. That's what Microsoft do. > > > > > > > > > > Indeed... and experience has shown that it works. > > > > > > > > What do you mean "it works"? What does it work to do? > > > > > > > ... > > > > You didn't answer the question. What do you mean "it works"? > > I did in the part you've stripped. Inhibiting users seems to be > something that most of them like. I couldn't see anything relating to inhibiting users in the part that I stripped. Also, I'm still not sure what the answer to my question is. Perhaps I'm not making myself clear. There's a goal which you're saying is achieved by inhibiting users (at least that's what I understand by the phrase "it works.") What is the goal? What does inhibiting users achieve? Bob -- Bob Ham -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070511/5010649e/attachment.pgp From nextgens at freenetproject.org Fri May 11 16:10:17 2007 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Fri, 11 May 2007 18:10:17 +0200 Subject: [Tech] Cache and Store In-Reply-To: <1178898044.17619.5.camel@orchid.arb.net> References: <200705101131.28453.toad@amphibian.dyndns.org> <20070510105216.GE5480@freenetproject.org> <200705101253.39094.toad@amphibian.dyndns.org> <1178829959.9397.21.camel@orchid.arb.net> <20070510204737.GG5480@freenetproject.org> <1178867989.9397.23.camel@orchid.arb.net> <20070511104218.GA5613@freenetproject.org> <1178888912.3916.4.camel@localhost> <20070511132525.GC5645@freenetproject.org> <1178898044.17619.5.camel@orchid.arb.net> Message-ID: <20070511161016.GG5645@freenetproject.org> * Bob Ham [2007-05-11 16:40:44]: > On Fri, 2007-05-11 at 15:25 +0200, Florent Daigni?re wrote: > > * Bob Ham [2007-05-11 14:08:32]: > > > > > On Fri, 2007-05-11 at 12:42 +0200, Florent Daigni?re wrote: > > > > * Bob Ham [2007-05-11 08:19:49]: > > > > > > > > > On Thu, 2007-05-10 at 22:47 +0200, Florent Daigni?re wrote: > > > > > > * Bob Ham [2007-05-10 21:45:59]: > > > > > > > > > > > > > This is a real cognitive problem showing up right there. It isn't your > > > > > > > responsibility to second-guess the user. There are valid reasons for > > > > > > > the node to have this functionality. The only reason for it not to is > > > > > > > to inhibit users. That's what Microsoft do. > > > > > > > > > > > > Indeed... and experience has shown that it works. > > > > > > > > > > What do you mean "it works"? What does it work to do? > > > > > > > > > ... > > > > > > You didn't answer the question. What do you mean "it works"? > > > > I did in the part you've stripped. Inhibiting users seems to be > > something that most of them like. > > I couldn't see anything relating to inhibiting users in the part that I > stripped. Also, I'm still not sure what the answer to my question is. > > Perhaps I'm not making myself clear. There's a goal which you're saying > is achieved by inhibiting users (at least that's what I understand by > the phrase "it works.") What is the goal? What does inhibiting users > achieve? Improving the usability, providing them an interface like they are used to. Not inhibiting them would force us to document what side effects changing that setting might have; It's not something we want to dedicate ressources on doing. NextGen$ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070511/fcd3e269/attachment.pgp From rah at bash.sh Fri May 11 19:57:06 2007 From: rah at bash.sh (Bob Ham) Date: Fri, 11 May 2007 20:57:06 +0100 Subject: [Tech] Cache and Store In-Reply-To: <20070511161016.GG5645@freenetproject.org> References: <200705101131.28453.toad@amphibian.dyndns.org> <20070510105216.GE5480@freenetproject.org> <200705101253.39094.toad@amphibian.dyndns.org> <1178829959.9397.21.camel@orchid.arb.net> <20070510204737.GG5480@freenetproject.org> <1178867989.9397.23.camel@orchid.arb.net> <20070511104218.GA5613@freenetproject.org> <1178888912.3916.4.camel@localhost> <20070511132525.GC5645@freenetproject.org> <1178898044.17619.5.camel@orchid.arb.net> <20070511161016.GG5645@freenetproject.org> Message-ID: <1178913427.17619.22.camel@orchid.arb.net> On Fri, 2007-05-11 at 18:10 +0200, Florent Daigni?re wrote: > * Bob Ham [2007-05-11 16:40:44]: > > > On Fri, 2007-05-11 at 15:25 +0200, Florent Daigni?re wrote: > > > * Bob Ham [2007-05-11 14:08:32]: > > > > > > > On Fri, 2007-05-11 at 12:42 +0200, Florent Daigni?re wrote: > > > > > * Bob Ham [2007-05-11 08:19:49]: > > > > > > > > > > > On Thu, 2007-05-10 at 22:47 +0200, Florent Daigni?re wrote: > > > > > > > * Bob Ham [2007-05-10 21:45:59]: > > > > > > > > > > > > > > > This is a real cognitive problem showing up right there. It isn't your > > > > > > > > responsibility to second-guess the user. There are valid reasons for > > > > > > > > the node to have this functionality. The only reason for it not to is > > > > > > > > to inhibit users. That's what Microsoft do. > > > > > > > > > > > > > > Indeed... and experience has shown that it works. > > > > > > > > > > > > What do you mean "it works"? What does it work to do? > > > > > > > > > > > ... > > > > > > > > You didn't answer the question. What do you mean "it works"? > > > > > > I did in the part you've stripped. Inhibiting users seems to be > > > something that most of them like. > > > > I couldn't see anything relating to inhibiting users in the part that I > > stripped. Also, I'm still not sure what the answer to my question is. > > > > Perhaps I'm not making myself clear. There's a goal which you're saying > > is achieved by inhibiting users (at least that's what I understand by > > the phrase "it works.") What is the goal? What does inhibiting users > > achieve? > > Improving the usability, providing them an interface like they are used > to. Not inhibiting them would force us to document what side effects > changing that setting might have; It's not something we want to dedicate > ressources on doing. Just so you understand, I'm talking generally here, not specifically about the store split issue (and, reading back, it might not have been clear but that's what I have been talking about.) It seems you don't want to improve the functionality of the node because improving the functionality involves communicating to users how to use the node properly. Either that, or changing the entire node's UI to be simpler; to something existing users aren't used to. I can see now what the issue is: the project is more concerned with forwarding political goals than it is with writing good software. That is not something I want to be involved with. I will discontinue my node's operation and leave you in peace. Bob -- Bob Ham -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070511/84935452/attachment.pgp From nextgens at freenetproject.org Fri May 11 21:35:14 2007 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Fri, 11 May 2007 23:35:14 +0200 Subject: [Tech] Cache and Store In-Reply-To: <1178913427.17619.22.camel@orchid.arb.net> References: <200705101253.39094.toad@amphibian.dyndns.org> <1178829959.9397.21.camel@orchid.arb.net> <20070510204737.GG5480@freenetproject.org> <1178867989.9397.23.camel@orchid.arb.net> <20070511104218.GA5613@freenetproject.org> <1178888912.3916.4.camel@localhost> <20070511132525.GC5645@freenetproject.org> <1178898044.17619.5.camel@orchid.arb.net> <20070511161016.GG5645@freenetproject.org> <1178913427.17619.22.camel@orchid.arb.net> Message-ID: <20070511213513.GF5489@freenetproject.org> * Bob Ham [2007-05-11 20:57:06]: > On Fri, 2007-05-11 at 18:10 +0200, Florent Daigni?re wrote: > > * Bob Ham [2007-05-11 16:40:44]: > > > > > On Fri, 2007-05-11 at 15:25 +0200, Florent Daigni?re wrote: > > > > * Bob Ham [2007-05-11 14:08:32]: > > > > > > > > > On Fri, 2007-05-11 at 12:42 +0200, Florent Daigni?re wrote: > > > > > > * Bob Ham [2007-05-11 08:19:49]: > > > > > > > > > > > > > On Thu, 2007-05-10 at 22:47 +0200, Florent Daigni?re wrote: > > > > > > > > * Bob Ham [2007-05-10 21:45:59]: > > > > > > > > > > > > > > > > > This is a real cognitive problem showing up right there. It isn't your > > > > > > > > > responsibility to second-guess the user. There are valid reasons for > > > > > > > > > the node to have this functionality. The only reason for it not to is > > > > > > > > > to inhibit users. That's what Microsoft do. > > > > > > > > > > > > > > > > Indeed... and experience has shown that it works. > > > > > > > > > > > > > > What do you mean "it works"? What does it work to do? > > > > > > > > > > > > > ... > > > > > > > > > > You didn't answer the question. What do you mean "it works"? > > > > > > > > I did in the part you've stripped. Inhibiting users seems to be > > > > something that most of them like. > > > > > > I couldn't see anything relating to inhibiting users in the part that I > > > stripped. Also, I'm still not sure what the answer to my question is. > > > > > > Perhaps I'm not making myself clear. There's a goal which you're saying > > > is achieved by inhibiting users (at least that's what I understand by > > > the phrase "it works.") What is the goal? What does inhibiting users > > > achieve? > > > > Improving the usability, providing them an interface like they are used > > to. Not inhibiting them would force us to document what side effects > > changing that setting might have; It's not something we want to dedicate > > ressources on doing. > > Just so you understand, I'm talking generally here, not specifically > about the store split issue (and, reading back, it might not have been > clear but that's what I have been talking about.) > Indeed it wasn't clear to me... I guess that starting an other thread would have been more appropriated. > It seems you don't want to improve the functionality of the node because > improving the functionality involves communicating to users how to use > the node properly. You don't get the point here : that's only part of the problem; It's not about explaining how to "use" the node but also explaining them how the node/network works. We can't expect the average user to understand it and even if we could we shouldn't assume that he has to get it in order to be able to use it. > Either that, or changing the entire node's UI to be > simpler; to something existing users aren't used to. Users are reluctant to changes, yes. > I can see now what the issue is: the project is more concerned with > forwarding political goals than it is with writing good software. I don't see why nor how having more settings and functionalities is related to writing "good" software. To me, a "good" software is something doing its job and not requiring me to understand how it does it. > That is not something I want to be involved with. I will discontinue my > node's operation and leave you in peace. Your call :) NextGen$ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070511/fe74951c/attachment.pgp From rah at bash.sh Fri May 11 22:18:30 2007 From: rah at bash.sh (Bob Ham) Date: Fri, 11 May 2007 23:18:30 +0100 Subject: [Tech] Cache and Store In-Reply-To: <20070511213513.GF5489@freenetproject.org> References: <200705101253.39094.toad@amphibian.dyndns.org> <1178829959.9397.21.camel@orchid.arb.net> <20070510204737.GG5480@freenetproject.org> <1178867989.9397.23.camel@orchid.arb.net> <20070511104218.GA5613@freenetproject.org> <1178888912.3916.4.camel@localhost> <20070511132525.GC5645@freenetproject.org> <1178898044.17619.5.camel@orchid.arb.net> <20070511161016.GG5645@freenetproject.org> <1178913427.17619.22.camel@orchid.arb.net> <20070511213513.GF5489@freenetproject.org> Message-ID: <1178921910.9348.0.camel@orchid.arb.net> On Fri, 2007-05-11 at 23:35 +0200, Florent Daigni?re wrote: > We can't expect the average user to understand it Why not? -- Bob Ham -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070511/845ad563/attachment.pgp From juiceman69 at gmail.com Fri May 11 23:32:05 2007 From: juiceman69 at gmail.com (Juiceman) Date: Fri, 11 May 2007 19:32:05 -0400 Subject: [Tech] Cache and Store In-Reply-To: <20070511104218.GA5613@freenetproject.org> References: <1178205708.10595.14.camel@localhost> <200705101131.28453.toad@amphibian.dyndns.org> <20070510105216.GE5480@freenetproject.org> <200705101253.39094.toad@amphibian.dyndns.org> <1178829959.9397.21.camel@orchid.arb.net> <20070510204737.GG5480@freenetproject.org> <1178867989.9397.23.camel@orchid.arb.net> <20070511104218.GA5613@freenetproject.org> Message-ID: <8b525dee0705111632o33a496cco3784b05c0918b5dc@mail.gmail.com> On 5/11/07, Florent Daigni?re wrote: > * Bob Ham [2007-05-11 08:19:49]: > > > On Thu, 2007-05-10 at 22:47 +0200, Florent Daigni?re wrote: > > > * Bob Ham [2007-05-10 21:45:59]: > > > > > > > On Thu, 2007-05-10 at 12:53 +0100, Matthew Toseland wrote: > > > > > On Thursday 10 May 2007 11:52, Florent Daigni?re wrote: > > > > > > * Matthew Toseland [2007-05-10 11:31:22]: > > > > > > > > > > > > > > As long as it's an expert option, I don't see any reason why it shouldn't > > > > > > > be accepted. > > > > > > > > > > > > Preventing users from their stupidity ? > > > > > > > > > > > > We don't want everyone to have a cache but no specialized data on the > > > > > > basis that "with a bigger cache downloads are 'resuming' ; I don't care > > > > > > about others nor the network so I only cache" > > > > > > > > > > Fair point. If they want to break their node that much they can maintain a > > > > > fork. > > > > > > > > This is a real cognitive problem showing up right there. It isn't your > > > > responsibility to second-guess the user. There are valid reasons for > > > > the node to have this functionality. The only reason for it not to is > > > > to inhibit users. That's what Microsoft do. > > > > > > Indeed... and experience has shown that it works. > > > > What do you mean "it works"? What does it work to do? > > > > Shall I remind you that most of our users do use Microsoft products and > are happy with them ? > > I'm against the introduction of new useless (no need to debate the > meaning of useless this time; show me some stats/simulations if you want > to be heard) settings, especially when they can harm the network... So > far we don't have content reachability problems and if we did, I would > suggest to fix the datastore code or to prevent users from nuking it on > a regular basis. > > The 50%-50% limit is arbitrary, so are many "unknown" and "hidden" > settings in the node; unless you have strong evidence (with a theorical > basis) explaining why it's bad/wrong you will be directed to the bug > tracking system and the "wishlist" category. I will ask you to explain > the "This is a problem." in your first mail [1] as well (so that we > could at least agree on the fact that we don't have the same definition > of what a problem is either ;) ). > > Moreover some related work has to be done on the shrinking code, > mroggers has suggested that the LRU policy might not be the best > solution to choose which data will be pruned; he came with some > references and is likely to be heard... You didn't, did you ? Anyway, > consequently his suggestion is likely to be implemented before any other > "related" work is done in that area of the code. IMO we shouldn't give > users incentives to use a "maybe broken and harmful" schrinker, hence > I'm against letting him play with the ratio of the cache/store. > No, I don't have data and/or references. As you said yourself "the 50%-50% limit is arbitrary". Why did you decide on 50%, where is your statical results from a simulation? What I do know is my gut tells me I probably have one the largest caches in Freenet. Any request that passes thru my node has a good chance of finding the data in my node. Hence, my outbound bandwidth is always pegged at 25KB/sec (which I don't have a problem with) I just have a feeling the "deep store of data" that I want my node to concentrate on is not being served because its answering "random requests." Is there a histogram of inbound and outbound transfers somewhere I can check? BTW, I recently "nuked" my datastore so I would stop seeing error messages about my store and corruption (and I knew it would fill back up quick enough with random requests). Maybe we should hide error messages from users so they think everything is "peachy"? -- I may disagree with what you have to say, but I shall defend, to the death, your right to say it. - Voltaire Those who would give up Liberty, to purchase temporary Safety, deserve neither Liberty nor Safety. - Ben Franklin From nextgens at freenetproject.org Fri May 11 23:50:00 2007 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Sat, 12 May 2007 01:50:00 +0200 Subject: [Tech] Cache and Store In-Reply-To: <1178921910.9348.0.camel@orchid.arb.net> References: <20070510204737.GG5480@freenetproject.org> <1178867989.9397.23.camel@orchid.arb.net> <20070511104218.GA5613@freenetproject.org> <1178888912.3916.4.camel@localhost> <20070511132525.GC5645@freenetproject.org> <1178898044.17619.5.camel@orchid.arb.net> <20070511161016.GG5645@freenetproject.org> <1178913427.17619.22.camel@orchid.arb.net> <20070511213513.GF5489@freenetproject.org> <1178921910.9348.0.camel@orchid.arb.net> Message-ID: <20070511235000.GB9562@freenetproject.org> * Bob Ham [2007-05-11 23:18:30]: > On Fri, 2007-05-11 at 23:35 +0200, Florent Daigni?re wrote: > > We can't expect the average user to understand it > > Why not? > Because it is too complex and its understanding relies on some knowledge which would have to introduced as well ? See the definition of "good software" I gave in the part you have stipped of my previous mail. I'm fed up with replying to you, you keep on stripping my replies, asking more and more questions without ever replying to mine. This is probably my last reply to you. NextGen$ PS: suggested reading : http://www.catb.org/~esr/faqs/smart-questions.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070512/af09e116/attachment.pgp From juiceman69 at gmail.com Fri May 11 23:59:13 2007 From: juiceman69 at gmail.com (Juiceman) Date: Fri, 11 May 2007 19:59:13 -0400 Subject: [Tech] Cache and Store In-Reply-To: <8b525dee0705111632o33a496cco3784b05c0918b5dc@mail.gmail.com> References: <1178205708.10595.14.camel@localhost> <200705101131.28453.toad@amphibian.dyndns.org> <20070510105216.GE5480@freenetproject.org> <200705101253.39094.toad@amphibian.dyndns.org> <1178829959.9397.21.camel@orchid.arb.net> <20070510204737.GG5480@freenetproject.org> <1178867989.9397.23.camel@orchid.arb.net> <20070511104218.GA5613@freenetproject.org> <8b525dee0705111632o33a496cco3784b05c0918b5dc@mail.gmail.com> Message-ID: <8b525dee0705111659s12f9aea8l232893067e2c5551@mail.gmail.com> > > No, I don't have data and/or references. As you said yourself "the > 50%-50% limit is arbitrary". Why did you decide on 50%, where is > your statical results from a simulation? What I do know is my gut > tells me I probably have one the largest caches in Freenet. Any > request that passes thru my node has a good chance of finding the data > in my node. Hence, my outbound bandwidth is always pegged at 25KB/sec > (which I don't have a problem with) I just have a feeling the "deep > store of data" that I want my node to concentrate on is not being > served because its answering "random requests." Is there a histogram > of inbound and outbound transfers somewhere I can check? > > BTW, I recently "nuked" my datastore so I would stop seeing error > messages about my store and corruption (and I knew it would fill back > up quick enough with random requests). Maybe we should hide error > messages from users so they think everything is "peachy"? > What if I make a check that the store can never be smaller than the cache? Would that alleviate your concerns? It would reduce functionality while still giving me the option. Oh, I also forgot to say when you decide you want to experiment with the ratio of the cache to store in the future, because you will someday you will just have to create the code anyways. So what's the harm in letting those of us who use the "expert settings" try this? Maybe we need a third level of settings "Developer" for all the settings that have the comment "Leave this alone". Why do we even include them? Developers can "maintain their own fork" (in your words) Here is a list of options that even "experts" shouldn't touch, by your reasoning: (My comments in quotes TMCI section "This has no relevence to anyone other than a *nix geek or a developer." # Override the CSS with a custom one (WARNING!)That setting allows you to override the node's CSS with a custom file. WARNING: CSSes can be dangerous and won't be filtered! use at your own risks. (consider mailing devl at freenetproject to get it included in the main distribution ;) ) Disable probabilistic HTL (don't touch this unless you know what you are doing) Maximum HTL (FOR DEVELOPER USE ONLY!) Frequency of dropping packets. Testing option used by devs to simulate packet loss. 0 means never artificially drop a packet. Don't touch this! Interval between swap attempting to send swap requests in milliseconds. Leave this alone! Disable all hang checkers/watchdog functions. Set this if you are profiling Fred. Disable all hang checkers/watchdog functions. Set this if you are profiling Fred. Enables the user to tweak the time in between GC and forced finalization. SHOULD NOT BE CHANGED unless you know what you're doing! -1 means: disable forced call to System.gc() and System.runFinalization() node.scheduler section Whether to enable testnet mode (DANGEROUS!). Testnet mode eliminates your anonymity in exchange for greatly assisting the developers in debugging the node. Now, why is my proposed change any worse than these? From nextgens at freenetproject.org Sat May 12 00:01:22 2007 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Sat, 12 May 2007 02:01:22 +0200 Subject: [Tech] Cache and Store In-Reply-To: <8b525dee0705111632o33a496cco3784b05c0918b5dc@mail.gmail.com> References: <1178205708.10595.14.camel@localhost> <200705101131.28453.toad@amphibian.dyndns.org> <20070510105216.GE5480@freenetproject.org> <200705101253.39094.toad@amphibian.dyndns.org> <1178829959.9397.21.camel@orchid.arb.net> <20070510204737.GG5480@freenetproject.org> <1178867989.9397.23.camel@orchid.arb.net> <20070511104218.GA5613@freenetproject.org> <8b525dee0705111632o33a496cco3784b05c0918b5dc@mail.gmail.com> Message-ID: <20070512000122.GC9562@freenetproject.org> * Juiceman [2007-05-11 19:32:05]: > On 5/11/07, Florent Daigni?re wrote: > >* Bob Ham [2007-05-11 08:19:49]: > > > >> On Thu, 2007-05-10 at 22:47 +0200, Florent Daigni?re wrote: > >> > * Bob Ham [2007-05-10 21:45:59]: > >> > > >> > > On Thu, 2007-05-10 at 12:53 +0100, Matthew Toseland wrote: > >> > > > On Thursday 10 May 2007 11:52, Florent Daigni?re wrote: > >> > > > > * Matthew Toseland [2007-05-10 > >11:31:22]: > >> > > > > > > >> > > > > > As long as it's an expert option, I don't see any reason why > >it shouldn't > >> > > > > > be accepted. > >> > > > > > >> > > > > Preventing users from their stupidity ? > >> > > > > > >> > > > > We don't want everyone to have a cache but no specialized data > >on the > >> > > > > basis that "with a bigger cache downloads are 'resuming' ; I > >don't care > >> > > > > about others nor the network so I only cache" > >> > > > > >> > > > Fair point. If they want to break their node that much they can > >maintain a > >> > > > fork. > >> > > > >> > > This is a real cognitive problem showing up right there. It isn't > >your > >> > > responsibility to second-guess the user. There are valid reasons for > >> > > the node to have this functionality. The only reason for it not to > >is > >> > > to inhibit users. That's what Microsoft do. > >> > > >> > Indeed... and experience has shown that it works. > >> > >> What do you mean "it works"? What does it work to do? > >> > > > >Shall I remind you that most of our users do use Microsoft products and > >are happy with them ? > > > >I'm against the introduction of new useless (no need to debate the > >meaning of useless this time; show me some stats/simulations if you want > >to be heard) settings, especially when they can harm the network... So > >far we don't have content reachability problems and if we did, I would > >suggest to fix the datastore code or to prevent users from nuking it on > >a regular basis. > > > >The 50%-50% limit is arbitrary, so are many "unknown" and "hidden" > >settings in the node; unless you have strong evidence (with a theorical > >basis) explaining why it's bad/wrong you will be directed to the bug > >tracking system and the "wishlist" category. I will ask you to explain > >the "This is a problem." in your first mail [1] as well (so that we > >could at least agree on the fact that we don't have the same definition > >of what a problem is either ;) ). > > > >Moreover some related work has to be done on the shrinking code, > >mroggers has suggested that the LRU policy might not be the best > >solution to choose which data will be pruned; he came with some > >references and is likely to be heard... You didn't, did you ? Anyway, > >consequently his suggestion is likely to be implemented before any other > >"related" work is done in that area of the code. IMO we shouldn't give > >users incentives to use a "maybe broken and harmful" schrinker, hence > >I'm against letting him play with the ratio of the cache/store. > > > > No, I don't have data and/or references. As you said yourself "the > 50%-50% limit is arbitrary". Why did you decide on 50%, where is > your statical results from a simulation? I am not suggesting to change it, that's the point. Why would your arbitrary choosen value be any better than the current default ? > What I do know is my gut > tells me I probably have one the largest caches in Freenet. Any > request that passes thru my node has a good chance of finding the data > in my node. Hence, my outbound bandwidth is always pegged at 25KB/sec > (which I don't have a problem with) I just have a feeling the "deep > store of data" that I want my node to concentrate on is not being > served because its answering "random requests." Replying to requests using the cache doesn't prevent your store from specializing. > Is there a histogram of inbound and outbound transfers somewhere I can check? Not that I'm aware of. > > BTW, I recently "nuked" my datastore so I would stop seeing error > messages about my store and corruption (and I knew it would fill back > up quick enough with random requests). Maybe we should hide error > messages from users so they think everything is "peachy"? Thanks for prooving me right... The urgency is to fix the DB code, not to implemenet a better schrinking mechanism. NextGen$ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070512/199674ae/attachment.pgp From nextgens at freenetproject.org Sat May 12 00:06:49 2007 From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re?=) Date: Sat, 12 May 2007 02:06:49 +0200 Subject: [Tech] Cache and Store In-Reply-To: <8b525dee0705111659s12f9aea8l232893067e2c5551@mail.gmail.com> References: <1178205708.10595.14.camel@localhost> <200705101131.28453.toad@amphibian.dyndns.org> <20070510105216.GE5480@freenetproject.org> <200705101253.39094.toad@amphibian.dyndns.org> <1178829959.9397.21.camel@orchid.arb.net> <20070510204737.GG5480@freenetproject.org> <1178867989.9397.23.camel@orchid.arb.net> <20070511104218.GA5613@freenetproject.org> <8b525dee0705111632o33a496cco3784b05c0918b5dc@mail.gmail.com> <8b525dee0705111659s12f9aea8l232893067e2c5551@mail.gmail.com> Message-ID: <20070512000648.GD9562@freenetproject.org> * Juiceman [2007-05-11 19:59:13]: > > > >No, I don't have data and/or references. As you said yourself "the > >50%-50% limit is arbitrary". Why did you decide on 50%, where is > >your statical results from a simulation? What I do know is my gut > >tells me I probably have one the largest caches in Freenet. Any > >request that passes thru my node has a good chance of finding the data > >in my node. Hence, my outbound bandwidth is always pegged at 25KB/sec > >(which I don't have a problem with) I just have a feeling the "deep > >store of data" that I want my node to concentrate on is not being > >served because its answering "random requests." Is there a histogram > >of inbound and outbound transfers somewhere I can check? > > > >BTW, I recently "nuked" my datastore so I would stop seeing error > >messages about my store and corruption (and I knew it would fill back > >up quick enough with random requests). Maybe we should hide error > >messages from users so they think everything is "peachy"? > > > > What if I make a check that the store can never be smaller than the > cache? Would that alleviate your concerns? It would reduce > functionality while still giving me the option. Fine but if I understood correctly what you want to do, it defeats the purpose. > > Oh, I also forgot to say when you decide you want to experiment with > the ratio of the cache to store in the future, because you will > someday you will just have to create the code anyways. So what's the > harm in letting those of us who use the "expert settings" try this? > > Maybe we need a third level of settings "Developer" for all the > settings that have the comment "Leave this alone". Why do we even > include them? Developers can "maintain their own fork" (in your > words) Convenience. > > Here is a list of options that even "experts" shouldn't touch, by your > reasoning: (My comments in quotes > > TMCI section "This has no relevence to anyone other than a *nix geek > or a developer." > > # Override the CSS with a custom one (WARNING!)That setting allows you > to override the node's CSS with a custom file. WARNING: CSSes can be > dangerous and won't be filtered! use at your own risks. (consider > mailing devl at freenetproject to get it included in the main > distribution ;) ) > > Disable probabilistic HTL (don't touch this unless you know what you are > doing) > > Maximum HTL (FOR DEVELOPER USE ONLY!) > > Frequency of dropping packets. Testing option used by devs to simulate > packet loss. 0 means never artificially drop a packet. Don't touch > this! > > Interval between swap attempting to send swap requests in > milliseconds. Leave this alone! > > Disable all hang checkers/watchdog functions. Set this if you are > profiling Fred. > > Disable all hang checkers/watchdog functions. Set this if you are > profiling Fred. > > Enables the user to tweak the time in between GC and forced > finalization. SHOULD NOT BE CHANGED unless you know what you're doing! > -1 means: disable forced call to System.gc() and > System.runFinalization() > > node.scheduler section > > Whether to enable testnet mode (DANGEROUS!). Testnet mode eliminates > your anonymity in exchange for greatly assisting the developers in > debugging the node. > > > > Now, why is my proposed change any worse than these? Those have local effects... what you propose to introduce can have network-wide effects. I'm not sure it's worst, but in any case it's yetAnotherOne. NextGen$ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070512/fd7b2d0f/attachment.pgp From rah at bash.sh Sat May 12 00:19:35 2007 From: rah at bash.sh (Bob Ham) Date: Sat, 12 May 2007 01:19:35 +0100 Subject: [Tech] Cache and Store In-Reply-To: <20070511235000.GB9562@freenetproject.org> References: <20070510204737.GG5480@freenetproject.org> <1178867989.9397.23.camel@orchid.arb.net> <20070511104218.GA5613@freenetproject.org> <1178888912.3916.4.camel@localhost> <20070511132525.GC5645@freenetproject.org> <1178898044.17619.5.camel@orchid.arb.net> <20070511161016.GG5645@freenetproject.org> <1178913427.17619.22.camel@orchid.arb.net> <20070511213513.GF5489@freenetproject.org> <1178921910.9348.0.camel@orchid.arb.net> <20070511235000.GB9562@freenetproject.org> Message-ID: <1178929175.9348.3.camel@orchid.arb.net> On Sat, 2007-05-12 at 01:50 +0200, Florent Daigni?re wrote: > * Bob Ham [2007-05-11 23:18:30]: > > > On Fri, 2007-05-11 at 23:35 +0200, Florent Daigni?re wrote: > > > We can't expect the average user to understand it > > > > Why not? > > > > Because it is too complex There is the "problem" I spoke of. Bob -- Bob Ham -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070512/ef1c3d45/attachment.pgp From rah at bash.sh Sat May 12 11:52:02 2007 From: rah at bash.sh (Bob Ham) Date: Sat, 12 May 2007 12:52:02 +0100 Subject: [Tech] Cache and Store In-Reply-To: <1178929175.9348.3.camel@orchid.arb.net> References: <20070510204737.GG5480@freenetproject.org> <1178867989.9397.23.camel@orchid.arb.net> <20070511104218.GA5613@freenetproject.org> <1178888912.3916.4.camel@localhost> <20070511132525.GC5645@freenetproject.org> <1178898044.17619.5.camel@orchid.arb.net> <20070511161016.GG5645@freenetproject.org> <1178913427.17619.22.camel@orchid.arb.net> <20070511213513.GF5489@freenetproject.org> <1178921910.9348.0.camel@orchid.arb.net> <20070511235000.GB9562@freenetproject.org> <1178929175.9348.3.camel@orchid.arb.net> Message-ID: <1178970722.9348.4.camel@orchid.arb.net> On Sat, 2007-05-12 at 01:19 +0100, Bob Ham wrote: > On Sat, 2007-05-12 at 01:50 +0200, Florent Daigni?re wrote: > > * Bob Ham [2007-05-11 23:18:30]: > > > > > On Fri, 2007-05-11 at 23:35 +0200, Florent Daigni?re wrote: > > > > We can't expect the average user to understand it > > > > > > Why not? > > > > > > > Because it is too complex > > There is the "problem" I spoke of. I should be more specific: it isn't too complex for users to understand. That you think it is, is the "problem." Bob -- Bob Ham -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070512/0c22a957/attachment.pgp From batosai at batosai.net Sun May 13 18:35:40 2007 From: batosai at batosai.net (batosai) Date: Sun, 13 May 2007 20:35:40 +0200 Subject: [Tech] [freenet-support] Freenet 0.7 build 1032 and call for translators In-Reply-To: <464743CD.4090205@batosai.net> References: <200705102228.44271.toad@amphibian.dyndns.org> <464743CD.4090205@batosai.net> Message-ID: <46475A7C.3040303@batosai.net> batosai a ?crit : > Matthew Toseland a ?crit : > >> If you want to translate Freenet into your native >> language, please contact us, and use the built-in functionality of your node >> to do it: set your language on the config page (set it to unlisted if your >> language isn't there already), and use the Translation page to translate >> strings. > > French translation is almost done. I noticed a problem with one > translation key : TranslationToadlet.translationUpdateTitle > > Even translated, english version is still displayed. Weird isn't it ? May I had a few translation keys are still missing ? Homepage : - The "Key :" before the textbox. Configuration page : - The titles of each section. Useless for FCP or FProxy, usefull for NODE, UPDATER, etc. Friends page : - "Peer statistics" title (the key exists but isn't used) - Friends connection state are translated in "Peer statistics", but not in the friends list. Queue page : - the "Compress" checkbox - "Insert file" and "Browse..." buttons Statistics page : - Title of the page : "Statistics for" - "Current Activity" an "Statistic gathering" titles That's all for normal mode, maybe I'll find more in advanced mode... Anyway, thanks for the work. I'll now ask NextGen$ for a review ;) Regards From toad at amphibian.dyndns.org Tue May 15 18:32:28 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Tue, 15 May 2007 19:32:28 +0100 Subject: [Tech] Cache and Store In-Reply-To: <1178913427.17619.22.camel@orchid.arb.net> References: <200705101131.28453.toad@amphibian.dyndns.org> <20070511161016.GG5645@freenetproject.org> <1178913427.17619.22.camel@orchid.arb.net> Message-ID: <200705151932.33868.toad@amphibian.dyndns.org> The user has complete freedom to hack his node in any bizarre way he wants to. On the other hand, we are entitled to only provide options which we deem useful. For example, we don't provide a ReportMyIPAddressToTheNSA option. As far as the specific issue goes, there may be a real solution (adaptive sizing) at some point, there have been suggestions. The point is, it will take time, and it isn't a priority right now. Therefore the place for it is the bugtracker. On Friday 11 May 2007 20:57, Bob Ham wrote: > > > Perhaps I'm not making myself clear. There's a goal which you're > > > saying is achieved by inhibiting users (at least that's what I > > > understand by the phrase "it works.") What is the goal? What does > > > inhibiting users achieve? > > > > Improving the usability, providing them an interface like they are used > > to. Not inhibiting them would force us to document what side effects > > changing that setting might have; It's not something we want to dedicate > > ressources on doing. > > Just so you understand, I'm talking generally here, not specifically > about the store split issue (and, reading back, it might not have been > clear but that's what I have been talking about.) > > It seems you don't want to improve the functionality of the node because > improving the functionality involves communicating to users how to use > the node properly. Either that, or changing the entire node's UI to be > simpler; to something existing users aren't used to. > > I can see now what the issue is: the project is more concerned with > forwarding political goals than it is with writing good software. That > is not something I want to be involved with. I will discontinue my > node's operation and leave you in peace. > > Bob -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070515/d5c72056/attachment.pgp From toad at amphibian.dyndns.org Tue May 15 18:37:57 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Tue, 15 May 2007 19:37:57 +0100 Subject: [Tech] Cache and Store In-Reply-To: <8b525dee0705101715w55434d04k2c29eda8d59ba5a6@mail.gmail.com> References: <1178205708.10595.14.camel@localhost> <200705101253.39094.toad@amphibian.dyndns.org> <8b525dee0705101715w55434d04k2c29eda8d59ba5a6@mail.gmail.com> Message-ID: <200705151937.58014.toad@amphibian.dyndns.org> On Friday 11 May 2007 01:15, Juiceman wrote: > I would make it an expert option and put in sanity checks to make sure > the ratio never goes outside 10 to 1 and each store is at least 32MB. > The default would still be 50% > > My reasoning is, I run a 18GB datastore and there is no way my node > could serve up 9GB of cached data without overloading my bandwidth and > probably starving requests that could hit the specialized store. > > What do you think? What exactly are you trying to do here? Increase the size of the store at the expense of the cache? Your node should be able to deal with a 250TB store. If it doesn't there is something wrong with the system. Many of these parameters have network-wide effects: The rules are not the same as when you are writing a word processor, because you are building a self-organising network, not a standalone device. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070515/6f261b91/attachment.pgp From toad at amphibian.dyndns.org Tue May 15 18:38:44 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Tue, 15 May 2007 19:38:44 +0100 Subject: [Tech] Cache and Store In-Reply-To: <1178868294.9397.27.camel@orchid.arb.net> References: <1178205708.10595.14.camel@localhost> <200705102221.00628.toad@amphibian.dyndns.org> <1178868294.9397.27.camel@orchid.arb.net> Message-ID: <200705151938.45272.toad@amphibian.dyndns.org> On Friday 11 May 2007 08:24, Bob Ham wrote: > On Thu, 2007-05-10 at 22:20 +0100, Matthew Toseland wrote: > > On Thursday 10 May 2007 21:38, Bob Ham wrote: > > > On Thu, 2007-05-10 at 00:36 +0100, Matthew Toseland wrote: > > > > On Wednesday 09 May 2007 20:28, Bob Ham wrote: > > > > > That was what I proposed near the start of this thread. I would > > > > > note, as well, that the store-shrinking code should already exist > > > > > for cases when the user reduces the configured size of the store. > > > > > > > > It does, but as I have already stated at least once, it is difficult > > > > to efficiently do an online shrink while preserving the most recently > > > > used data. > > > > > > > > It is of course possible. One way to do it is to swap the key you'd > > > > be deleting with the least recently used key just before truncating. > > > > > > What's your point? > > > > Obviously the latter is preferred but we can only do it on startup. > > I'm still unsure as to why you're telling us this. Is your point that > work still needs to be done on store shrinking? No, it's that dynamically shrinking the cache as the store grows isn't feasible without additional work on shrinking. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070515/822b8d74/attachment.pgp From rah at bash.sh Thu May 17 08:45:08 2007 From: rah at bash.sh (Bob Ham) Date: Thu, 17 May 2007 09:45:08 +0100 Subject: [Tech] Cache and Store In-Reply-To: <200705151938.45272.toad@amphibian.dyndns.org> References: <1178205708.10595.14.camel@localhost> <200705102221.00628.toad@amphibian.dyndns.org> <1178868294.9397.27.camel@orchid.arb.net> <200705151938.45272.toad@amphibian.dyndns.org> Message-ID: <1179391508.8292.3.camel@localhost> On Tue, 2007-05-15 at 19:38 +0100, Matthew Toseland wrote: > On Friday 11 May 2007 08:24, Bob Ham wrote: > > On Thu, 2007-05-10 at 22:20 +0100, Matthew Toseland wrote: > > > On Thursday 10 May 2007 21:38, Bob Ham wrote: > > > > On Thu, 2007-05-10 at 00:36 +0100, Matthew Toseland wrote: > > > > > On Wednesday 09 May 2007 20:28, Bob Ham wrote: > > > > > > That was what I proposed near the start of this thread. I would > > > > > > note, as well, that the store-shrinking code should already exist > > > > > > for cases when the user reduces the configured size of the store. > > > > > > > > > > It does, but as I have already stated at least once, it is difficult > > > > > to efficiently do an online shrink while preserving the most recently > > > > > used data. > > > > > > > > > > It is of course possible. One way to do it is to swap the key you'd > > > > > be deleting with the least recently used key just before truncating. > > > > > > > > What's your point? > > > > > > Obviously the latter is preferred but we can only do it on startup. > > > > I'm still unsure as to why you're telling us this. Is your point that > > work still needs to be done on store shrinking? > > No, it's that dynamically shrinking the cache as the store grows isn't > feasible without additional work on shrinking. Erm.. it isn't feasible without additional work on itself, either. Your point is completely redundant. No code is feasible without its dependencies. Bob -- Bob Ham From toad at amphibian.dyndns.org Fri May 18 15:56:39 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Fri, 18 May 2007 16:56:39 +0100 Subject: [Tech] Cache and Store In-Reply-To: <1179391508.8292.3.camel@localhost> References: <1178205708.10595.14.camel@localhost> <200705151938.45272.toad@amphibian.dyndns.org> <1179391508.8292.3.camel@localhost> Message-ID: <200705181656.40067.toad@amphibian.dyndns.org> On Thursday 17 May 2007 09:45, Bob Ham wrote: > > > > > > It is of course possible. One way to do it is to swap the key > > > > > > you'd be deleting with the least recently used key just before > > > > > > truncating. > > > > > > > > > > What's your point? > > > > > > > > Obviously the latter is preferred but we can only do it on startup. > > > > > > I'm still unsure as to why you're telling us this. Is your point that > > > work still needs to be done on store shrinking? > > > > No, it's that dynamically shrinking the cache as the store grows isn't > > feasible without additional work on shrinking. > > Erm.. it isn't feasible without additional work on itself, either. Your > point is completely redundant. No code is feasible without its > dependencies. The point is, it's significant work and not a priority right now. > > Bob -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070518/6cd2a029/attachment.pgp From toad at amphibian.dyndns.org Fri May 18 17:34:34 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Fri, 18 May 2007 18:34:34 +0100 Subject: [Tech] ASL2 definitely not compatible with GPL3, new draft Message-ID: <200705181834.34635.toad@amphibian.dyndns.org> http://www.fsfeurope.org/projects/gplv3/brussels-rms-transcript.en.html RMS on GPL3 draft 3 (significant changes from draft 2). "So we are compatible with that clause in the Apache licence but a few months ago we noticed that there was another clause in the Apache licence requiring indemnity in certain cases, and there's no way we can be compatible with that. So we're not going to achieve that goal of making GPL version three compatible with the existing Apache licence. I regret that." Which means we can't comply 100% with the law while including Apache Commons code in Freenet. :| Except maybe by adding an exception to the license, which requires permission of all copyright holders, and that's a *MAJOR* PITA. More on GPL3 draft 3: http://www.fsfeurope.org/projects/gplv3/gplv3.en.html Rationale for changes from draft 2: http://gplv3.fsf.org/rationale From m.rogers at cs.ucl.ac.uk Fri May 18 19:49:18 2007 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Fri, 18 May 2007 20:49:18 +0100 Subject: [Tech] ASL2 definitely not compatible with GPL3, new draft In-Reply-To: <200705181834.34635.toad@amphibian.dyndns.org> References: <200705181834.34635.toad@amphibian.dyndns.org> Message-ID: <464E033E.3060702@cs.ucl.ac.uk> Matthew Toseland wrote: > So we're not going to achieve that goal of making GPL version three > compatible with the existing Apache licence. I regret that." I haven't really followed the GPLv3 debate so sorry if this is a stupid question, but can't we just stick with GPLv2 if we need to? Cheers, Michaek From toad at amphibian.dyndns.org Fri May 18 22:40:49 2007 From: toad at amphibian.dyndns.org (Matthew Toseland) Date: Fri, 18 May 2007 23:40:49 +0100 Subject: [Tech] ASL2 definitely not compatible with GPL3, new draft In-Reply-To: <464E033E.3060702@cs.ucl.ac.uk> References: <200705181834.34635.toad@amphibian.dyndns.org> <464E033E.3060702@cs.ucl.ac.uk> Message-ID: <200705182340.54857.toad@amphibian.dyndns.org> On Friday 18 May 2007 20:49, Michael Rogers wrote: > Matthew Toseland wrote: > > So we're not going to achieve that goal of making GPL version three > > compatible with the existing Apache licence. I regret that." > > I haven't really followed the GPLv3 debate so sorry if this is a stupid > question, but can't we just stick with GPLv2 if we need to? No. ASL2 is less compatible with GPL2 than it is with GPL3. We can stick with GPL2 if we don't use any ASL2 code. Problem is, Apache Commons is ASL2. :( Having said that, the bzip2 implementation is GPL2+ so we can use that. It's only tar support that's a problem then. (Tar support would be really helpful imho, it would produce significantly smaller containers). We don't use any of this code yet, but we'd like to. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070518/32b2d765/attachment.pgp From juiceman69 at gmail.com Sat May 19 04:11:45 2007 From: juiceman69 at gmail.com (Juiceman) Date: Sat, 19 May 2007 00:11:45 -0400 Subject: [Tech] Cache and Store In-Reply-To: <200705151937.58014.toad@amphibian.dyndns.org> References: <1178205708.10595.14.camel@localhost> <200705101253.39094.toad@amphibian.dyndns.org> <8b525dee0705101715w55434d04k2c29eda8d59ba5a6@mail.gmail.com> <200705151937.58014.toad@amphibian.dyndns.org> Message-ID: <8b525dee0705182111v5ec5bae1lad166fc8cfe75b4e@mail.gmail.com> On 5/15/07, Matthew Toseland wrote: > On Friday 11 May 2007 01:15, Juiceman wrote: > > I would make it an expert option and put in sanity checks to make sure > > the ratio never goes outside 10 to 1 and each store is at least 32MB. > > The default would still be 50% > > > > My reasoning is, I run a 18GB datastore and there is no way my node > > could serve up 9GB of cached data without overloading my bandwidth and > > probably starving requests that could hit the specialized store. > > > > What do you think? > > What exactly are you trying to do here? Increase the size of the store at the > expense of the cache? > That is it exactly. I was coding an option to do this.