Timestamps are in GMT/BST.
[0:15] * Tarantuka (n=eddie207@) Quit ("Quitte")
[0:20] * luke771_afk is now known as Luke771
[0:33] <Luke771> ?!?
[0:34] <Luke771> I checked on my main node and found that it wad down, checked the other node's friends page and found out that it had been down for ~3hrs.
[0:35] <Luke771> re-launched the node using the script the process seems to be running but fproxy can't be accessed
[0:36] <Luke771> the last log entry is timestamped some 3 hours ago (matches down time), but it's weird, it only reads '#'
[0:37] <Luke771> no logs entries for now (latest restart)
[0:40] <Luke771> now this: I killed the process but the startup script insists thet freenete is running...
[0:40] <Luke771> brb
[0:40] * Luke771 (n=luke@) Quit ("Leaving")
[0:48] * Luke771 (n=luke@) has joined #freenet
[0:49] <Luke771> it's up now. I used the old "windows cure" (rebooting)
[0:49] <Luke771> (it works on *nix too, sometimes)
[0:49] <Luke771> but I still have no idea what happened
[0:50] * infinity0 (n=infinity@) Quit ("Leaving.")
[1:19] * sleon (i=sleon@) Quit (Read error: 110 (Connection timed out))
[1:23] * NEOatNHNG1 (n=michael@) has joined #freenet
[1:24] * sleon (i=sleon@) has joined #freenet
[1:24] * ChanServ sets mode +o sleon
[1:38] * zerwas (i=zerwas@) Quit (Remote closed the connection)
[1:39] * NEOatNHNG (n=michael@) Quit (Read error: 110 (Connection timed out))
[1:40] * zerwas (i=zerwas@) has joined #freenet
[1:56] * toad_ (n=toad@) Quit (Remote closed the connection)
[2:03] * zerwas_ (i=zerwas@) has joined #freenet
[2:04] * zerwas (i=zerwas@) Quit (Nick collision from services.)
[2:04] * zerwas_ is now known as zerwas
[2:15] * caytchen (n=caytchen@) Quit (Read error: 104 (Connection reset by peer))
[2:16] * gasi__ (n=gasi@) has joined #freenet
[2:32] * gasi_ (n=gasi@) Quit (Read error: 110 (Connection timed out))
[2:39] * Luke771 (n=luke@) Quit ("Leaving")
[3:00] * FrinkC (n=FrinkC@) Quit ("Bye")
[3:15] * archangelpetro (n=hellequi@) has joined #freenet
[3:15] * zerwas (i=zerwas@) Quit (Remote closed the connection)
[3:22] * zerwas (i=zerwas@) has joined #freenet
[4:05] * doow_ (n=doow@) has joined #freenet
[4:17] * doow (n=doow@) Quit (Read error: 110 (Connection timed out))
[4:29] * Pichu0102 (n=pichu010@) Quit (Client Quit)
[4:29] * Pichu0102 (n=pichu010@) has joined #freenet
[4:30] * natalia (i=shapki@) has joined #freenet
[4:30] <natalia> Hello
[4:30] <natalia> I need to get version 1153 but I cannot even connect in order to get the update. What should I do?
[4:32] * natalia sighs
[4:38] <zerwas> natalia, hm, wait for the server to work again?
[4:40] <natalia> yeah i think it just came up
[4:40] <natalia> it was acting funny at first
[4:45] <zerwas> :>
[5:09] * banyan (n=banyan@) has joined #freenet
[5:10] <banyan> hello! Stupid question: how do I find out what my noderef is?
[5:11] <banyan> so another user can add my node.
[5:20] * natalia (i=shapki@) Quit (Read error: 110 (Connection timed out))
[5:29] <mrdmx> its in your friends page
[5:35] * Nande (n=KVIrc@) Quit ("byes, tengan la bondad de ser felices. ;)")
[5:35] <banyan> so it is! thanks.
[5:36] * banyan (n=banyan@) Quit ("Help! I'm being renditioned!")
[5:54] * Pichu0102 (n=pichu010@) Quit (Remote closed the connection)
[5:56] * Pichu0102 (n=pichu010@) has joined #freenet
[6:08] * obinou1 (n=jaube@) has joined #freenet
[6:39] * christl_1 is now known as christl_s
[7:21] * tsp (n=tyler@) Quit (Read error: 104 (Connection reset by peer))
[7:24] * tsp (n=tyler@) has joined #freenet
[7:25] * Luke771 (n=luke@) has joined #freenet
[7:33] * zerwas (i=zerwas@) Quit ("Keep on flowing ...")
[7:39] * dbkr- (n=nndbkr@) has joined #freenet
[7:46] * obinou1 (n=jaube@) has left #freenet
[8:00] * dbkr (n=ndbkr@) Quit (Read error: 110 (Connection timed out))
[8:00] * dbkr- is now known as dbkr
[8:14] * DaffyDuck_ (n=jan@) Quit ("Leaving")
[8:17] * zerwas (i=zerwas@) has joined #freenet
[8:21] * zerwas (i=zerwas@) Quit (Client Quit)
[8:34] <nextgens> hi
[8:35] <nextgens> Luke771> kork> are you trying to win an idiocy contest or something? symlinking /dev/urandom to /dev/random deserves the top prize!
[8:36] <nextgens> if your system is in lack of real entropy generate some IO: something like "find /" in a terminal will do it
[8:38] <Luke771> nextgens: lol
[8:38] <Luke771> in fact it didnt work
[8:38] <Luke771> I reverted after a while
[8:39] <Luke771> the node is running fine now
[8:40] <Luke771> oh besides, remember that node that you asked me to test with the new openjdk for ubuntu?
[8:40] <Luke771> when that VM was up with the node tunning, the whole host box became so slow it could take several minutes to move the mouse cursor
[8:41] <Luke771> I don't know if it was the node or ubuntu, or the openjdk, or the combination. I have two more VM's running nodes, one debian and one windows, and the guest is perfectly usabl4e
[8:43] <Luke771> only when I truned on that ubuntu VM with that openjdk node, everything slwoed down under the threshiold of usability (and well behyong the threshold of poss-off'ing)
[8:43] <Luke771> turned*
[8:45] <nextgens> well, if it's only a performance problem it can be addressed
[8:46] <Luke771> anyhow, it didn't crash, that's the main point
[8:50] <Luke771> by the way, I'm having some problems with the L10n file
[8:50] <nextgens> hmm?
[8:50] <nextgens> what kind of pb ?
[8:51] <Luke771> the idea is to update the trasnlation on the fly as I fnotice things that should be changed and only download the l120n file once a month and send it
[8:51] <Luke771> but lately
[8:51] <Luke771> every time I look at the trasnlation, the strings that I edited are back to normal
[8:51] <Luke771> where nortmal is the way they were before I changed them
[8:52] <Luke771> ok, I know the answer: file a bug
[8:52] <Luke771> but the idea was to have someone else to check if they get the same problem, it may be something local after all
[8:53] <Luke771> the edited l10n file is still there but it looks like it isn't loaded after a restart
[8:53] <nextgens> I don't understand what you mean
[8:53] <nextgens> some strings won't be updated until you reboot
[8:53] <nextgens> that's "normal"
[8:54] <nextgens> but that's not your problem, is it ?
[8:54] <Luke771> no
[8:54] <nextgens> okay
[8:54] <Luke771> that's not what I mean
[8:54] <nextgens> so maybe it's a merge problem
[8:54] <Luke771> <== stupid
[8:54] <Luke771> I have 4 nodes running, I can check myself
[8:54] <nextgens> you submit the files and someone screws the merge up
[8:55] <nextgens> is that it ?
[8:55] <Luke771> no
[8:55] <Luke771> this is before I submit them
[8:55] <nextgens> hrrrm
[8:55] <Luke771> locally, when fproxy is supposed to loadmy edited version that overwrite the one that comes with freenet
[8:56] <Luke771> usually, I could edit the file, leave the work half done, restart the node, and see it as I left it
[8:56] <Luke771> and continue
[8:56] <Luke771> now I have to start from scratch
[8:56] <nextgens> okay
[8:56] <Luke771> the edited parts with a different background in fproxy arent there any more
[8:56] <Luke771> that's why I say it looks like the l10n override file isn't loaded
[8:57] <nextgens> iirc toad has introduced a config switch
[8:57] <nextgens> let me check
[8:57] * FrinkC (n=FrinkC@) has joined #freenet
[8:58] <Luke771> if it is so, would that mean that I have to set it to use the l10n override file?
[8:59] <nextgens> no, the l10n override is going to be used in any case
[8:59] <nextgens> the config switch was about displaying or not the "translate me" links
[8:59] <Luke771> then it looks like a bug, because it's not being used
[8:59] <Luke771> but it can be me doing something wrong, I'm gonna cross check
[8:59] <nextgens> Luke771> could you give me one l10n key that acts as you describe please ?
[9:00] <Luke771> sure
[9:00] <nextgens> anyway, if you can confirm it then fill a bug in
[9:00] <Luke771> now wait, what you mean by a l10n key? what I can do is to send you the l10n override file that is not being loaded
[9:00] <Luke771> yes
[9:00] <Luke771> I'll file a bug if it's confirmed
[9:01] <Luke771> I can pastebin the l10n file
[9:01] <nextgens> Luke771> ok, that would be fine
[9:01] <Luke771> would that be OK?
[9:01] <Luke771> ok
[9:04] <Luke771> nextgens: http://pastebin.ca/1073153
[9:14] * Luke771 (n=luke@) Quit ("Leaving")
[9:22] <nextgens> eh
[9:22] <nextgens> Luke is gone :(
[9:22] <nextgens> QueueToadlet.downloadFiles=Download multipli
[9:22] <nextgens> that works for me
[9:24] * DaffyDuck_ (n=jan@) has joined #freenet
[9:24] * Luke771 (n=luke@) has joined #freenet
[9:25] <nextgens> Luke771> ah
[9:25] <nextgens> Luke771> it works for me
[9:25] <nextgens> I mean, the override file works as expected
[9:25] <Luke771> k
[9:25] <Luke771> then it must be some local problem
[9:25] <Luke771> (I wonder what that can be)
[9:25] <nextgens> if it's called freenet.l10n.it.override.properties, located in freenet's dir and is readable, it's loaded
[9:26] <Luke771> oh, THERE we have it
[9:26] <nextgens> (it has to be utf8 encoded too)
[9:26] <Luke771> mine is called italiano instead of it
[9:26] <nextgens> hrrrmm
[9:26] <Luke771> freenet.l10n.italiano.override.properties
[9:26] <Luke771> ok I'll rename it
[9:26] <nextgens> did you call it that way ?
[9:26] <Luke771> but it was generated like that
[9:26] <Luke771> no
[9:27] <nextgens> or did I screw up when I implemented long names?
[9:27] <nextgens> okay
[9:27] <nextgens> so it's a bug :|
[9:28] <Luke771> well, it generates a file with the long name of the language but it still wants the short name for loading, so yes, it's a bug
[9:28] <Luke771> should I file it, or you will?
[9:28] <nextgens> I'll fix it
[9:28] <Luke771> k
[9:29] <Luke771> hmpf
[9:29] <Luke771> forgot the & and now I can't close the terminal... or use it
[9:29] <Luke771> brb
[9:29] * Luke771 (n=luke@) Quit (Client Quit)
[9:30] * Luke771 (n=luke@) has joined #freenet
[9:38] <Luke771> Er, after all, someone DID screw up my trasnlation. Someone submitted a l10n file that has been accepted without me reveiwing it. As a result, "include local address in noderef" has been trasnlated to the equivalent of "noderef includers local addresses" not mentioning that the word 'noderef' has not been trasnlated)
[9:38] <Luke771> now, either I do the ITA trasnlation, ort someone else does. I'm getting tired of seeing my work (and my language) tortured like that
[9:39] <Luke771> i mean it's cool that more people contribute but let me review them before commiting, I think I've done well so far...
[9:42] <nextgens> Luke771> use svn blame and see who commited it
[9:43] <tsp> Luke771: if you forgot an &, just suspend the process with ^z and run bg to background it
[9:43] <nextgens> alternatively give me the l10n-key of the string and I'll have a look
[9:44] <Luke771> no biggie, it's only one string so far, and I'll send the new updated file within one week or little more.
[9:44] <Luke771> thx tsp
[9:44] <tsp> np
[9:46] <Luke771> my idea is that I would like to review the work of new italian translators if there are any. If I find that one particular guy is doing a good job, I'd say OK you can commit stuff from this guy without asking me from now on. I think this is necessary to keep a the ITA l10n to a quality at least equal to what I've done so far
[9:47] <Luke771> from now on, that is. What's done is done.
[9:48] <nextgens> Luke771> try r21140 out please
[9:49] <Luke771> is that latest trunk?
[9:49] <nextgens> yes, trunk from 1sec ago
[9:49] <Luke771> k
[9:49] <nextgens> Luke771> and give me the corresponding l10n key so that I can check who commited it
[9:50] <nextgens> (the key is the part on the left of the '=')
[9:50] <Luke771> the corresponding... to what? the wrongly trasnlated string?
[9:50] <nextgens> yes
[9:51] <FreenetLogBot> r21140 (1153) was built successfully on emu, mirrors are updating
[9:51] <Luke771> ah, forget that, it's even possible that it was in the latests l10n that I sent. What happened is that a couple of l10n updates ago, Toad commited some strings from a guy who translated them poorly. I corrected them and talked to the guy, old him to ask me before sending more strings)
[9:52] <Luke771> it is possible that the wrongly stranslated string was in that bunch and I missed it when I corrected them
[9:52] <nextgens> well, we could have reverted the commit altogether if that helped
[9:52] <Luke771> no need
[9:53] <Luke771> I'm going thru the l10n right now and it looks good, that string was the only problem so far, and that can wait to the end of the month
[9:53] <Luke771> if I find something so big that it needs to be fixed right away, I'll send a new update
[9:53] <Luke771> an extra update I men
[9:55] <Luke771> mean*
[9:59] <Luke771> nextgens: OK the override file is loaded now, with long language name
[9:59] <nextgens> huh?
[9:59] <nextgens> it should use short names everywhere
[9:59] <Luke771> well maybe bioth work...
[10:00] <Luke771> what happened is
[10:00] <Luke771> that while you were fixing the bug, I was editing the l10n, which generated a new file with long language name
[10:00] <Luke771> then I updated from truck, and the override file was loaded, with long name and everything, just as it was generated
[10:01] <Luke771> I('ll change another string and see if it generates a shot name file
[10:01] <Luke771> it does
[10:02] <Luke771> so now it generates a file with the short language name but it will also load a file with the long name... maybe it shouldnt, I don't know
[10:02] <Luke771> but it does
[10:06] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror3.freenetproject.org/ : it has been disabled
[10:08] <nextgens> raaaaahhhh
[10:12] <FreenetLogBot> r21141 (1153) was built successfully on emu, mirrors are updating
[10:13] * Luke771 is now known as luke771_afk
[10:15] <FreenetLogBot> r21143 (1153) was built successfully on emu, mirrors are updating
[10:15] * toad_ (n=toad@) has joined #freenet
[10:15] * ChanServ sets mode +o toad_
[10:18] <nextgens> hey toad_ !
[10:20] <toad_> hi
[10:20] <FreenetLogBot> The monitoring script has detected that http://mirror3.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[10:20] <toad_> what's up?
[10:21] * nextgens is fixing a few bugs in the l10n framework
[10:21] <toad_> cool
[10:22] * toad_ is reasonably convinced that the db4o layer can work
[10:22] <nextgens> and I'm back in France :)
[10:22] <toad_> but it needs more debugging
[10:22] <toad_> cool, i suppose that means you'll be getting a Real Job soon?
[10:22] <FreenetLogBot> r21144 (1153) was built successfully on emu, mirrors are updating
[10:22] <nextgens> not before September
[10:23] <toad_> ok
[10:23] <nextgens> at least I don't plan to
[10:23] <toad_> any freenet plans for the near to mid term future?
[10:24] <nextgens> I might implement the friends-of-myfriends location broadcasting
[10:24] <toad_> that would be great
[10:24] <nextgens> appart from that nothing special
[10:25] <nextgens> it shouldn't be merged at the same time as other big chunks of work
[10:25] <nextgens> I still suspect that sdiz's random replacement based DS will have effects on the network
[10:26] <nextgens> and that your branch will lead to an increase of the join/leave churn
[10:26] <toad_> why? the simulations look good
[10:26] <nextgens> both being likely to f*ck things up
[10:26] * FreeAndy2 (n=Andy@) has joined #freenet
[10:26] <FreeAndy2> hi
[10:26] <toad_> my branch will lead to more join/leave churn because there will be less of a startup spike?
[10:27] <nextgens> last time we talked about it we agreed on the fact that the simulations were partial
[10:27] <nextgens> and covering only the store
[10:27] <toad_> and more background disk i/o
[10:27] <nextgens> whereas most of hits come from the cache :/
[10:27] <FreeAndy2> is it normal that I have to load the freemail plugin after each and every restart of my node again?
[10:27] <nextgens> toad_> because there will be less incentives to keep nodes up
[10:27] <nextgens> FreeAndy2> no
[10:27] <toad_> nextgens: i think users are stupid, they don't take the hint
[10:28] <nextgens> I am convinced that they are stupid too
[10:28] <FreeAndy2> why are we stupid?
[10:28] <toad_> FreeAndy2: you're not, you just don't understand freenet until you've used it for years
[10:28] <nextgens> hence I'm against providing them tools which will encourage bad behaviours network-wise
[10:29] <FreeAndy2> and what can I do in order not to have to reload the plugin each time? All the other plugins remain stable (though I don't use them) but freemail always disappears and needs to be reloaded
[10:29] <toad_> FreeAndy2: hmmm
[10:29] <FreeAndy2> nextgens: ah, ok, I see
[10:29] <toad_> FreeAndy2: file a bug report on the bug tracker: https://bugs.freenetproject.org/
[10:30] <toad_> nextgens: a certain subset of users will run their nodes 24x7 if we ask them to/tell them it will give better performance
[10:30] <toad_> most users however can't/won't run their nodes 24x7 for various reasons ... and if we try to force them to they will simply dump freenet
[10:30] <toad_> that's my view
[10:30] <toad_> users are stupid *and* stubborn
[10:31] <toad_> :)
[10:31] <FreeAndy2> yes
[10:31] <FreeAndy2> me for example
[10:31] <FreeAndy2> my comp is in the living room and makes a lot of noise, as CPU load is always high with freenet
[10:31] <toad_> i am somewhat worried about the amount of disk i/o the db4o branch is going to cause ... but sdiz's work will save us a ton of disk i/o
[10:31] <FreeAndy2> and it keeps crashing a lot
[10:31] <toad_> FreeAndy2: hmmm
[10:32] <FreeAndy2> but i try to keep it alive as long as possible
[10:32] <toad_> FreeAndy2: how powerful is your computer? how much RAM is freenet allowed? how much data is on your queue?
[10:32] <toad_> FreeAndy2: and does your whole computer crash or just freenet?
[10:32] <toad_> oh and how much RAM do you have overall?
[10:33] <nextgens> toad_> IOs are expensive on lots of desktop-friendly OSes
[10:33] <toad_> nextgens: IOs are expensive full stop
[10:33] <nextgens> if you take macos, windows or even ubuntu; they use a IO scheduler which is a PITA
[10:33] <toad_> 1 disk seek = 1 million ram seeks
[10:33] <nextgens> well, depending on how you order them they might not be
[10:33] <nextgens> sure
[10:34] <nextgens> but that's time-wise
[10:34] <nextgens> the problem with "low-latency" schedulers is that they tend to monopolize other ressources while waiting for the IO task to complete
[10:35] <toad_> nextgens: I *want* prioritised scheduling etc ... I want freenet to not DoS the rest of the system, even if it means freenet slows down
[10:35] <nextgens> say you want to move a file on macos: that's a bulk-io operation ...
[10:35] <nextgens> but it will still drag your CPU down to 100% usage until it's finished
[10:35] * mrdmx is now known as mrdmx-away
[10:35] <nextgens> whereas there is no reason to
[10:35] <toad_> nextgens: why?
[10:35] <FreeAndy2> toad: I run an AMD64Athlon with 1GB Ram and WinXP
[10:36] <FreeAndy2> 512 RAM is given to Freenet
[10:36] <nextgens> because it's identified by the scheduler as a high priority job: it's ordered by the user
[10:36] <toad_> FreeAndy2: how many cores?
[10:36] <FreeAndy2> it's only Freenet that crashes and starts up again, or does not
[10:36] <FreeAndy2> one core
[10:36] <toad_> FreeAndy2: how much data on the queue?
[10:36] <toad_> nextgens: why does that cause high cpu usage?
[10:37] <FreeAndy2> I used to have some files in the queue but since last week there all vanished mysteriously
[10:37] <nextgens> toad_> the scheduler does: it does active-polling I guess
[10:37] <toad_> high load sure but high cpu usage?!
[10:37] <toad_> i doubt it very much
[10:37] <FreeAndy2> so zero files in queue now
[10:37] <nextgens> toad_> try it for yourself
[10:37] <nextgens> it's just like on windows:
[10:38] <toad_> FreeAndy2: no files in the queue and it still crashes? I have no idea what causes that...
[10:38] <toad_> FreeAndy2: are you running games at the same time?
[10:38] <nextgens> most of the time your mouse cursor is going to lag when you're moving a file across hard drives
[10:38] <FreeAndy2> nope but other things
[10:38] <FreeAndy2> namely: i2p
[10:38] <FreeAndy2> and boinc
[10:38] <nextgens> moving your mouse cursor doesn't involve any "realted" IO (no disk access)
[10:38] <toad_> nextgens: why would even windows use PIO on modern SATA disks?
[10:38] <nextgens> but it will still sucks
[10:39] <toad_> FreeAndy2: boinc is at a lower priority than freenet, right?
[10:40] <FreeAndy2> I thought that if I have my comp on for such a long time for i2p and freenet, why not do the world some good and have boinc running at the same time, which does some distributed calculations
[10:40] <toad_> nextgens: anyway, if we have plenty of RAM, the database will be entirely in cache, so we only have the commits
[10:40] <FreeAndy2> toad: I'm pretty sure it's very low
[10:40] <FreeAndy2> I'll check
[10:40] <toad_> FreeAndy2: is boinc at a lower priority than freenet?
[10:41] <nextgens> if the wrapper decides it doesn't get enough cpu time, it will restart the jvm
[10:41] <toad_> if we have to read as well then we may have a problem
[10:41] <FreeAndy2> that might well be the reason
[10:41] <toad_> FreeAndy2: :) :)
[10:41] <nextgens> FreeAndy2> have a look to your wrapper.log file
[10:41] <FreeAndy2> boinc is lower in priority than freenet, but maybe i2p and freenet conflict
[10:41] * luke771_afk is now known as Luke771
[10:41] <nextgens> it will tell you why it restarts
[10:41] <toad_> i doubt it...
[10:42] <FreeAndy2> it tells?
[10:42] <FreeAndy2> where?
[10:42] <toad_> yeah, post the last 200 lines or so of your wrapper.log to a pastebin
[10:42] <FreeAndy2> normally I see it coz the sites don't load or there comes the starting site saying "freenet is starting up..."
[10:42] <FreeAndy2> ok, I'll search a wrapper.log
[10:43] <toad_> it's in the directory you installed freenet into
[10:43] <toad_> you don't need to search the whole disk for it
[10:45] <nextgens> toad_> is there anything else I should put into my freenet-related todo ?
[10:45] <nextgens> toad_> earlier on I saw in the logfiles that Luke771 and kork were planning to do silly things like symlinking random to urandom in order to make their nodes start up ...
[10:46] <nextgens> what can we do against that?
[10:46] <nextgens> improve the error message?
[10:46] <nextgens> generate IOs for them?
[10:46] <toad_> presumably we have to generate IOs?
[10:46] <toad_> or give them the option to generate IOs?
[10:47] <toad_> many systems have entropy problems
[10:47] <nextgens> mine doesn't :p
[10:48] <toad_> you have a /dev/hwrng ?
[10:48] <nextgens> awa
[10:48] <nextgens> no
[10:49] <nextgens> well I do on my server
[10:49] <nextgens> but on my laptop I use wifi and listen to music
[10:49] <FreeAndy2> http://de.pastebin.ca/1073219
[10:49] <nextgens> both of them are helping the kernel to generate a fair amount of entropy
[10:50] <nextgens> even if I cat /dev/random my system doesn't run out of it
[10:50] <FreeAndy2> finally I know what this HelloWord-Plugin is for....
[10:50] <nextgens> INFO | jvm 1 | 2008/07/16 10:34:33 | Caused by: java.io.IOException: Es steht nicht genug Speicherplatz auf dem Datenträger zur Verfügung
[10:50] <nextgens> ...
[10:50] <nextgens> FreeAndy2> that's the best way to f*ck a freenet node up
[10:51] <FreeAndy2> oh, my hdd is frequently full
[10:51] <nextgens> it uses 100% cpu because it's constantly recovering the datastore
[10:51] <FreeAndy2> then I think I need to get myself a new one soon
[10:52] <FreeAndy2> right now I have 32MB hdd free
[10:52] <toad_> why is it recovering the datastore?
[10:52] <nextgens> that's not enough
[10:52] <toad_> why does it crash?
[10:52] <nextgens> toad_> because his system runs out of free space
[10:52] <nextgens> toad_> that's what the german ioe is about
[10:52] <FreeAndy2> how much free dpace do I need?
[10:53] <nextgens> that depends on your log-level and what you do with your node
[10:53] <nextgens> but 32M is not enough for "normal" operations
[10:53] <FreeAndy2> I thought, once i give freenet a certain amount of hdd for its sotre it reserves it and doesn't need a single byte more than that...
[10:53] <nextgens> resize your store if you really have space-related problems
[10:53] <FreeAndy2> my store is 5 GB
[10:53] <nextgens> that's not how it works
[10:53] <nextgens> anyway; bbiab
[10:54] <toad_> FreeAndy2: freenet doesn't allocate temporary space out of the store
[10:54] <FreeAndy2> ok, then now at least I know what the problem is
[10:54] <toad_> FreeAndy2: at the moment
[10:54] <FreeAndy2> thank you nextgens
[10:54] <FreeAndy2> and toads
[10:54] <FreeAndy2> toad
[10:55] <FreeAndy2> and is there a risk posting wrapper.log on a public pastebin, or even noderefs?
[10:55] <FreeAndy2> for I would like to turn to darknet, but I know nobody personally who runs freenet.
[10:56] <FreeAndy2> Tried to convince some friends, but without success
[10:56] <toad_> exchanging noderefs with random strangers is bad, don't do it
[10:56] <FreeAndy2> Only one is interested in free speech and anonymity but now he uses TOR (before that, he used nothing at al)
[10:57] <FreeAndy2> but still better than running completely opennet, isn't it?
[10:57] <toad_> Tor and Freenet are complementary
[10:57] <toad_> Tor now has some support for running in places where it may be blocked, which is good
[10:57] <toad_> Tor is very poor for running anonymously hosted websites for various reasons
[10:57] <FreeAndy2> yea, but he only wants to surf and chat anonymously without being subject to data retention here in Germany and he's not into blogs and filesharing...
[10:58] <toad_> it *can* be secure, but it's a lot of work, and you have intersection attacks
[10:58] <toad_> whereas posting stuff on freenet is easy and (relatively) safe
[10:58] <FreeAndy2> I use i2p at the same time
[10:58] <FreeAndy2> but i2p is not f2f and might one day become prohibited
[10:58] <toad_> w.r.t. filesharing imho freenet could be faster than bittorrent over tor or bittorrent over i2p
[10:59] <toad_> whether it is now is another question, for some things maybe it is
[10:59] <Luke771> when our "democracies" will make all anonymous networking illegal because "terrorists could use it", Tor and i2p will be gone pretty fast, Freenet has the potential to keep running forever, in the face of al the opressive governments
[10:59] <FreeAndy2> what dl rates can you achieve in Freenet?
[10:59] <FreeAndy2> I never got reasonably good dl rates
[10:59] <toad_> Luke771: exactly. Freenet *may* be survivable even if there's nowhere to run. Tor never will be
[11:00] <toad_> HOWEVER, I'm not sure about that: if we insist on a 24x7 uptime network which is real time and over the internet, they can screw us no matter what stego we try to use, with traffic flow analysis
[11:00] <toad_> FreeAndy2: what download rates do you get for bittorrent over tor? :)
[11:00] <toad_> i bet they're rubbish too :)
[11:01] <toad_> if you can run your node 24x7, with a reasonable bandwidth limit, then freenet can be fast enough to be interesting
[11:01] <FreeAndy2> and maybe all encrypted traffic for ordinary people will be prohibited one day and maybe there'll be a way to tell if it's encrypted data you transfer or just a webcam
[11:01] <FreeAndy2> never tried over TOR
[11:01] <toad_> FreeAndy2: with traffic flow analysis they could identify it anyway :(
[11:02] <FreeAndy2> i2p is slower than real internet, but if it's on for several hours, you get some files.
[11:02] <toad_> i mean, there's this huge great f2f network of people constantly looking at each other's webcams, 24 hours a day ... how likely is that?
[11:02] <Luke771> FreeAndy2: people who keep comparing Tor to Freenet miss the whole point. They do different things. For instance, Freenet can't be used to anonymously access the regular web, and Tor can be used to anonymously host websites, but they will only stay up as long as your PC stays up
[11:02] <Luke771> On Freenet on the other hand, your freesite will stay up even if you're arrested or dead
[11:02] <toad_> anyway, first we make it work :)
[11:02] <FreeAndy2> Right now I have 40kB/s dl in i2p-bt and 15kB/s on iMule (also i2p) but I must stop them now, for no more disk space left
[11:03] <toad_> there's been a debate going on for a while on the lists about how far we should go to support non-realtime transports etc
[11:03] <Luke771> i2p has a huge problem: can only run as long as the giovernment allows it to run
[11:03] <Luke771> government
[11:03] <toad_> FreeAndy2: you should be able to run them in parallel, if you reduce the disk space limits
[11:04] <toad_> Luke771: freenet too if the ISPs deploy traffic flow analysis :(
[11:04] <Luke771> I could run freenet and i2p on a duron1600 box with only 1GB of RAM installed, and both ran fine
[11:04] <Luke771> well, we can still go around the ISP...
[11:04] <toad_> at least in its current real-time-focused 24x7-assumption form
[11:04] <toad_> Luke771: right, that means sneakernet and so on
[11:05] <toad_> most of the senior project people think that's a separate project, i disagree
[11:05] <Luke771> but I guess once we move to guerilla networking, i2p can be made to run there too...
[11:05] <toad_> a separate and much less ambitious project
[11:05] <toad_> no, i2p can't run over sneakernet
[11:05] <FreeAndy2> sneakernet?
[11:05] <toad_> it can maybe run over wifi, but only if you can make a whole wireless underground internet, which imho is unrealistic
[11:06] <toad_> FreeAndy2: swapping a pair of 8G USB sticks with a friend once a day yields around a megabit bandwidth to just that friend
[11:06] <Luke771> FreeAndy2: sneakernet is a bunch of guerilla networks held together by people physiocally moving sotrage around
[11:06] <toad_> if i remember correctly
[11:07] <FreeAndy2> ah, ok
[11:07] <toad_> 99,420 bytes per second on average
[11:07] <FreeAndy2> hope we'll never need that
[11:07] <toad_> they do it now in Cuba
[11:07] <toad_> not with freenet, of course
[11:08] <toad_> there are half-way points like stealing the video stream on VoIP and so on
[11:08] <FreeAndy2> so there's nothing else that could be done against traffic flow analysis?
[11:08] <Luke771> guerilla netwoking can be done in citioes and medium/big town, the problem IMHO is how to connect those cities/twons to each other... unless someone finds a way to freeload on the existing infrastructures
[11:08] <toad_> or even faking the timing of VoIP and placing completely bogus calls ... but that sort of stego is very hard
[11:08] <toad_> Luke771: sneakernet probably
[11:08] <toad_> not in all cases but in many cases
[11:09] <Luke771> that's the obvious solution, yes, but it also has an obvious latency problem
[11:09] <toad_> indeed
[11:09] <FreeAndy2> will freenet ever be able to provide real-time services such as im?
[11:09] <toad_> you can try to do long range wireless, but it must be pretty easy to find
[11:09] <toad_> FreeAndy2: that also depends on the outcome of ongoing discussions
[11:09] <Luke771> anyhow, getting city/town nets up and working would be a good start
[11:10] <toad_> there is a sort of real time chat system...
[11:10] <toad_> i dunno how well it works, and i bet it abuses the network by polling constantly
[11:10] <toad_> Luke771: maybe
[11:10] <toad_> Luke771: most likely they will be centralised and owned by a community co-op or by the local council or by a contractor to the local council
[11:10] <toad_> Luke771: because that's what works best performance wise
[11:11] <toad_> price/performance wise
[11:11] <FreeAndy2> toad: you don't know how good it works? I mean you're the freenet dev, I thought you know everything on freenet and that you tired everything...
[11:11] <Luke771> my idea would be more like
[11:12] <Luke771> each user owns his or her own node, and everyone connects to each other
[11:12] <toad_> Luke771: that doesn't work well on wireless
[11:12] <toad_> it wastes so much bandwidth as to be unusable for larger networks
[11:12] <Luke771> in a city like where I live, if everyone opened their wifi routers to anyone, without ISP's, we'd be all connected to each other
[11:12] <toad_> especially if you use a tiny omni in a window, which of course most people would
[11:13] <Luke771> the porblem is how to make people accept this idea
[11:13] <toad_> the problem is routing isn't feasible on an ad hoc network with 100,000 nodes
[11:13] <Luke771> theyre being told that they shouldn't communicate with anyone but ISP's...
[11:13] <toad_> and even if it were it would cut your bandwidth by a factor of 10
[11:13] <Luke771> well yes
[11:14] <Luke771> it would have high latency and low bandwidth
[11:14] <Luke771> very low
[11:14] <toad_> exactly, and then you want to run freenet on top?!?!?!?!?!?!?!
[11:14] <Luke771> some kind of expensive hardware is necessary... and that's where groups of people come in the picture...
[11:14] <toad_> not gonna work!
[11:14] <toad_> right, what you need is dedicated, hidden, directional links between darknet peers
[11:15] <toad_> omnis if they're close together maybe
[11:15] <FreeAndy2> but you should have your wireless onyl admit connections to freenet and not to the rest of your computer or your regular internet connection
[11:15] <FreeAndy2> provided that was possible, I think, people would accept the idea of a non encrypted open wireless
[11:15] <Luke771> ok, so we're stuck with ISP's for now
[11:15] <toad_> we're stuck with ISPs, but long term we won't be able to run freenet over ISPs, at least not 24x7
[11:16] <toad_> and perhaps not at all or not in any useful way
[11:16] <Luke771> you're thinking about net neutrality issues?
[11:16] <toad_> i'm thinking about government forcing ISPs to block p2p
[11:17] <toad_> which is very much on the agenda in europe
[11:17] <Luke771> yes, I guess it is
[11:17] <FreeAndy2> is it?
[11:17] <Luke771> and in the who /HA HA) "free" world
[11:17] <toad_> and will likely come to the US as well
[11:17] <toad_> FreeAndy2: certainly it is in the UK
[11:17] <toad_> whether IPRED2 will have that effect EU-wide is uncertain
[11:18] <FreeAndy2> I don't know that.
[11:18] <Luke771> and where US and UK go, europe will follow
[11:18] <Luke771> a couple of years later, possibly, but it will
[11:18] <toad_> but it is likely that p2p will be blocked from 2010ish in the UK ... ISPs and the media industry are negotiating under the thread of primary legislation
[11:18] <FreeAndy2> I mena ok, against copyright infringements they are, but I thought, only if proven. So that prohibiting ALL p2p is on the agenda is new to me.
[11:18] <toad_> sorry threat not thread
[11:19] <Luke771> so we have, illegal p2p + depp packet inspections that makes disguising freenet packets into some 'legal' stream virtually impossible... all is left is stego
[11:19] <Luke771> deep*
[11:19] <FreeAndy2> damn world!
[11:19] <toad_> FreeAndy2: if the media industry and the ISPs do not come to an agreement, then the government will pass legislation making ISPs responsible for illegal filesharing on their networks
[11:19] <Luke771> thanks "democracy"
[11:20] <toad_> Luke771: deep packet inspection is irrelevant. stateful traffic flow analysis is the next step up and is a big worry.
[11:20] <Luke771> ask any person in the west and over 90% will tell you that they believe they live in a free country
[11:20] <Luke771> with a straight face!
[11:20] <toad_> FreeAndy2: they have said as much
[11:20] <Luke771> anyhow, there's no way we can change that
[11:20] <Luke771> all we can do is try to make some free speech tool available before it's too late
[11:21] <toad_> we can try
[11:21] <Luke771> and that's what freenet is for...
[11:21] <Luke771> sure
[11:21] <toad_> but for it to work we need a huge darknet, and long term we need it to be mostly off-grid
[11:22] <FreeAndy2> off-grid you mean again swapping USB-devices?
[11:22] <toad_> still, when people realise they need it it should become a little easier to get f2f connections :)
[11:22] <toad_> FreeAndy2: or wifi links, cables between friends, or wireless rendezvous with PDAs etc, or any number of other transports
[11:22] <toad_> some can probably go over the internet, but it certainly wouldn't be constant real time connections
[11:23] <Luke771> at that point a major problem would be to convince people that they need alternative non-isp connections, and also that they need free speech tools! Itì's very difficult to sell umrella if 95% of the people don't believe that it will ever rain, depsite the black clouds
[11:23] <toad_> anyway, ian, michael and some other senior folk think that's not freenet, and some simple broadcast system will suffice... :(
[11:23] <toad_> but we'll see
[11:24] <Luke771> anyhow, I'm repeating this stuff for people who already get it... if I only could make that clear for those who 'dont' get it...
[11:24] <FreeAndy2> it's always good to have a solution for all possible problems
[11:24] <toad_> heh
[11:24] <FreeAndy2> even if they might never arise (the probs)
[11:24] <Luke771> they are already here
[11:24] <toad_> freenet 2.0 will feature ... world peace!
[11:25] <Luke771> lol
[11:25] <Luke771> it will, if it will run in a totally human-free world
[11:25] <Luke771> as long as three people are left on earth, there will be war
[11:25] <toad_> nah, artificial intelligences would probably have wars too :)
[11:25] <Luke771> hm.
[11:25] <Luke771> possible
[11:26] <Luke771> but they couldnt kill anyone, so no real war
[11:26] <Luke771> more like vandalism
[11:26] <FreeAndy2> depends on thir algorithms
[11:26] <toad_> they could kill each other
[11:26] <Luke771> they're not alive to being with
[11:26] <Luke771> begin
[11:26] <Luke771> gtg
[11:26] <Luke771> bbl
[11:26] <FreeAndy2> byeluke
[11:26] <FreeAndy2> maybe they're alife
[11:27] <FreeAndy2> depends on your definition of live and on the artificial intelligence of course
[11:27] <toad_> FreeAndy2: are you intending to run your node 24x7?
[11:27] <FreeAndy2> toad: no :-(
[11:27] <toad_> FreeAndy2: or are there practical problems (other than it crashing) that would prevent that?
[11:27] <FreeAndy2> but as close as possible to this
[11:28] <FreeAndy2> my comp is in the living room and constant noise is bad for my nerves
[11:28] <toad_> okay so realistically?
[11:28] <FreeAndy2> but when I sleep or when I'm away or when I work on pc it's on
[11:28] <toad_> 12 hours a day?
[11:28] <toad_> 15?
[11:29] <toad_> on average?
[11:29] <FreeAndy2> well, my i2p says "uptime 3d"
[11:29] <FreeAndy2> theoretically (if no crashes or hassles occur) freenet and i2p are supposed to run concurrently here
[11:29] <FreeAndy2> mayb 15 on average will be the future
[11:30] <toad_> second question: at the moment, if you have downloads queued, and you restart freenet, they have to restart from scratch from the datastore, which takes ages and uses a lot of disk access
[11:30] <FreeAndy2> right now it's a little more but I already got shouted by my wife
[11:30] <FreeAndy2> I have ZERO downloads in freenet
[11:31] <toad_> the code i'm working on would make them resume immediately, would this cause you to download more stuff? would it cause you to turn off the computer more because there's less reason to keep it on?
[11:31] <FreeAndy2> i can imagine that some people would turn it off sooner
[11:32] <FreeAndy2> I would not. but I know many people who enter, get what they want and quickly disappear
[11:32] * toad_ is of the opinion that we will lose more through unhappy users than through happy users turning the computer off
[11:32] <toad_> that's not really feasible on freenet
[11:32] <toad_> at least not for large downloads, they take time
[11:32] <toad_> and freenet starts off fairly slow
[11:32] <toad_> but gets faster
[11:33] <FreeAndy2> my tor firend used normal emule before, dl a file, didn't share a single file and as soon as it was downloaded, he switched off
[11:33] <FreeAndy2> but he's an extreme example
[11:33] <toad_> is he an extreme example of a complete leech, or is he the norm?
[11:33] <FreeAndy2> I agree, that most would rather stay tuned and download more things if dl expereince is good (fast, flawless)
[11:34] <toad_> well it will never be flawless :)
[11:34] <toad_> and it may not be fast either ... :|
[11:34] <toad_> faster, but faster still if you have high uptime...
[11:35] <FreeAndy2> I see the advantages of i2p in the fact that it's a data store and that it's f2f, but apart from that...
[11:35] <FreeAndy2> ... is it more or less anonymous than i2p?
[11:35] <toad_> good question, i haven't studied i2p in detail
[11:35] <phrosty> i2p isn't a datastore
[11:35] <toad_> opennet freenet is way too easy to attack, if you can be completely f2f, you have some proection
[11:36] <toad_> protection
[11:36] <FreeAndy2> I'm very happy with i2p, good dl rates, Instant messenger that works, bittorrent, Gnutella network, imule...
[11:36] <phrosty> i have yet to try it out
[11:36] <FreeAndy2> but if you told me i2p was not secure or anonymous enough I'd hesitat to use it extensively
[11:36] <toad_> FreeAndy2: there are $150-$200 home servers being sold by various places ... generally for bittorrent and http downloads
[11:37] <toad_> would you get one if it could run freenet?
[11:37] <FreeAndy2> But I see the need for a data store like freenet and of f2f for when i2p becomes prohibited
[11:37] <toad_> well i2p is probably a lot smaller than freenet is at the moment, no?
[11:38] <toad_> if you sort out your disk space issues then freenet should deliver reasonable download rates
[11:38] <phrosty> i used it back when it was just anon IRC - had quite a bit of users at the time
[11:38] <FreeAndy2> yes, about 800 nodes connected at any given time
[11:38] <FreeAndy2> but fast downloads, real time anonymopus IRC and IM
[11:38] <toad_> hmmm well freenet probably isn't much more than that
[11:39] <toad_> can I have some answers w.r.t. the hardware box?
[11:39] <FreeAndy2> they had a ig loss of users after jrandom left and the central download server crashed
[11:39] <FreeAndy2> before that user base was even above 1300
[11:40] <toad_> imho freenet with the database changes should be capable of running on something like a c7 with hardware crypto and flash disk...
[11:40] <toad_> FreeAndy2: total users, or online simultaneously?
[11:40] <toad_> freenet has 3000+ over 24 hours, but a lot of churn, online simultaneously might only be 1000
[11:41] <FreeAndy2> toad: home servers for freenet? why not. Ideal would be if it could be put into some closed and simply run and run and run
[11:41] <toad_> it's just an idea that i'm playing with, that's all...
[11:41] <FreeAndy2> lemmmi check right now
[11:41] <toad_> a possible partial solution to the downtime problem
[11:41] <phrosty> toad_: what are the goals of the new store?
[11:41] <toad_> phrosty: sdiz's store?
[11:42] <phrosty> yea
[11:42] * Luke771 is now known as luke771_afk
[11:42] <toad_> not corrupting itself, reducing unnecessary disk accesses, stuff like that
[11:42] <phrosty> ah
[11:42] <toad_> random replacement has been shown in simulations to be slightly better network wise than LRU ... but more simulations may be needed
[11:42] <toad_> less RAM usage too
[11:43] <FreeAndy2> the stats eepsite is down
[11:44] <phrosty> try implementing ARC
[11:44] <phrosty> it's given me much better perf than LRU, for some b+tree code
[11:44] <toad_> my branch (db4o) on the other hand is all about the client layer: put everything (queued downloads, queued uploads) in a database, so that the node doesn't take a long time to restart them on startup, and doesn't need them all in RAM, so you can have as big a queue as you want (in theory) and not use insane amounts of RAM
[11:45] * TheBishop_ (n=TheBisho@) Quit (Read error: 110 (Connection timed out))
[11:45] <toad_> phrosty: depends on what it's used for, just because it works well as a web cache doesn't necessarily mean it'll work well for freenet
[11:45] <phrosty> ARC is basically just a scan-resistant LRU with memory
[11:46] <toad_> scan-resistant meaning...?
[11:46] <phrosty> a block read once might push out a more often read block in LRU
[11:46] <phrosty> scan as in, reading a lot of uncommon blocks at the same time
[11:47] <toad_> well, propose it on the mailing list ... but we will need simulations ...
[11:47] <toad_> the great thing about the salted hash datastore is we don't need an index structure at all
[11:48] <toad_> it's basically a data-carrying bloom filter
[11:49] <phrosty> ah, found a way to have efficient removal?
[11:49] <toad_> why do we need to?
[11:49] <toad_> oh i see
[11:49] <toad_> no, it's an associative array or something ... we have the key, and we get the location in the store where it should be
[11:49] <toad_> like a hashtable
[11:49] <phrosty> ah
[11:49] <toad_> but not chained
[11:50] <toad_> we have a bloom filter as well to minimize accesses, sdiz is working on that at the moment
[11:51] <toad_> so it should be significantly less hassle, disk i/o and memory usage ... which is just as well as my branch will produce a significant amount of disk access
[11:51] <FreeAndy2> site is up again
[11:52] <toad_> granted that's in exchange for hopefully less ram usage (although more ram churn), and no real ram vs queue size issues
[11:52] <FreeAndy2> currently there are 710 nodes connected to i2p
[11:52] <toad_> well that's respectable enough as these things go :)
[11:52] <phrosty> reducing thread count would probably help bring resource usage down
[11:53] <toad_> maybe
[11:53] <phrosty> on windows, by at least 64K/thread
[11:53] <toad_> so what, how many threads do we have?
[11:53] <toad_> even if we have 500 threads, that's only 32MB
[11:53] <toad_> and most of the time we have a lot less than that
[11:54] <toad_> atm i have 108 threads
[11:54] <phrosty> it tends to hover between 200-400 for me
[11:54] <toad_> what are most of those doing?
[11:54] <toad_> can you show me your thread usage box?
[11:54] <phrosty> not at the moment, freenet is stopped
[11:54] <toad_> :|
[11:54] <phrosty> never remember to restart it when pc boots
[11:55] <toad_> well post it sometime
[11:55] <toad_> it would be interesting
[11:56] <FreeAndy2> here when I reboot freenet starts automatically
[11:56] <FreeAndy2> well, gotta go now
[11:56] <phrosty> it's on an encrypted drive for me, service autostart doesn't work :)
[11:56] <toad_> cya
[11:56] <FreeAndy2> see ya folks
[11:56] <toad_> phrosty: meaning you only run it when you are physically present?
[11:56] * FreeAndy2 (n=Andy@) Quit (" HydraIRC -> http://www.hydrairc.com <- Nine out of ten l33t h4x0rz prefer it")
[11:57] <phrosty> yes, but due to working from home that means a large amount of the day
[11:57] <toad_> do you have any opinions on the downtime question? or on my db4o branch?
[11:58] <phrosty> downtime question?
[11:58] <toad_> what is the average uptime likely to be, and what can we do to maximize performance given that?
[11:59] <toad_> and how far should we go to help users who can't run 24x7?
[11:59] <phrosty> well, any increase in perf for non-24/7 users would probably help 24/7 users too, right? or at least leave them their experience the same?
[12:00] <toad_> maybe
[12:00] <phrosty> there are probably a lot of people out there who turn their PCs off at night, their experience should definitely be considered
[12:01] <toad_> yeah... but if they only turn it on in the evening?
[12:01] <toad_> if we can make freenet play nice with games etc then maybe it can run for a reasonable number of hours ...
[12:01] <toad_> but even so, can we make 6 hours a day work? especially with darknet!
[12:01] <phrosty> i don't have any friends on freenet, i run opennet :/
[12:02] <toad_> yeah.. well long term we have to assume darknet
[12:02] <toad_> although short term we just want to do whatever it takes to get more users
[12:03] <phrosty> perf tends to peak after an hour or so for me
[12:03] <phrosty> so if people run 6 hours a day, maybe it's not so bad
[12:04] <toad_> well on opennet .. if people understand that it isn't like [ insert classic p2p here ] and that waiting an hour is a good thing ...
[12:04] <toad_> but what about darknet?
[12:05] <phrosty> well, the reason none of my friends run freenet is because of resource usage
[12:05] * toad_ starts thinking about delay tolerance and whether passive requests can be adapted to do this, and people accuse him of wanting to build an egg-laying woolmilkpig
[12:05] <toad_> phrosty: ahh, so they've tried it?
[12:05] <phrosty> yea
[12:05] <toad_> phrosty: can you be specific about resource usage?
[12:05] <nextgens> rehi
[12:06] <toad_> CPU, RAM?
[12:06] <toad_> something else?
[12:06] <phrosty> yea, cpu/ram
[12:06] <toad_> nextgens: hi
[12:06] <toad_> phrosty: after queueing lots of downloads?
[12:06] <toad_> phrosty: or just running in the background completely idle?
[12:06] <toad_> do they have low-end (<1GB RAM) PCs?
[12:07] <phrosty> the threads cause crazy context switching, cache invalidation, etc. which slows the whole system down, and ram prevents them from getting more FPS in games or whatever they do.
[12:07] <toad_> well that's your theory .. what are the observed facts?
[12:08] <nextgens> phrosty> find out why it doesn't start on the encrypted drive so that I can fix it upstream
[12:08] <phrosty> the observed facts is it doesn't run unnoticeably enough, in both cpu and ram, for them to want it running all the time
[12:08] <phrosty> nextgens: not a freenet issue, drive doesn't auto-mount so there's nothing to start
[12:08] <nextgens> okay :/
[12:09] <toad_> phrosty: is there high CPU usage?
[12:09] <toad_> how much RAM do they have?
[12:09] <phrosty> 1 or 2 gigs, most of them
[12:09] <toad_> hmmm...
[12:10] <toad_> well, 0.8 should run reasonably happily in 128MB ...
[12:10] <nextgens> phrosty> as long as games are mono-threaded and that processor development follow the current trend, we should be fine: more cores will help
[12:10] <toad_> if they have 2GB then it's not likely to be a problem with RAM ...
[12:10] <phrosty> nextgens: there is such a thing as using the hardware, and using the hardware as an excuse
[12:10] <phrosty> toad_: yea, ram isn't such an issue as cpu
[12:10] <nextgens> granted
[12:10] <toad_> otoh with BDBFS we do a significant amount of disk I/O, and the new code will likely do even more
[12:11] <toad_> phrosty: well ... maybe ...
[12:11] <nextgens> but on the other hand, developer's time is expensive; so is reducing the amount of threads we use
[12:11] <nextgens> toad_> we should auto-calibrate that
[12:11] <toad_> did they have stuff queued? typically in the past when freenet has constantly used 100% cpu it has generally been because of running out of memory and constantly garbage collecting
[12:11] <nextgens> atm we don't addapt bdb's cache size depending on the size of the datastore
[12:11] <nextgens> that's a mistake
[12:11] <toad_> nextgens: imho that's impossible
[12:12] <phrosty> i get about 2-5% cpu usage on a quad core, that's 8-20% on a single core. with cache thrashing slowing everything else down, it's not something you want to run while playing games.
[12:12] <nextgens> 10M of cache is fine for 1-10G datastores, not bigger ones
[12:12] <toad_> nextgens: we'd have to auto-calibrate the total memory limit too, that way lies madness
[12:12] <nextgens> granted
[12:12] <toad_> nextgens: imho we should just merge sdiz's store and forget about bdbje, although maybe keep it as an option for those who want it
[12:12] <nextgens> but I don't see any alternative
[12:13] <toad_> with sdiz's store plus my db4o branch we should be able to just set the limit to 128M by default
[12:14] <nextgens> that's the current default, isn't it?
[12:14] <phrosty> i have no experience with Java, but in C/C++ or C# it is quite easy to get by with a single thread per logical processor
[12:14] <toad_> phrosty: what makes you think cache thrashing is a big problem? how long does it take to load the cache on a modern system, RAM is fast, no?
[12:14] <toad_> nextgens: no the current default is 192M
[12:14] <nextgens> toad_> it's not about cache but pipelining imho
[12:15] <toad_> nextgens: that's even more negligible
[12:15] <toad_> who cares about the odd 40 cycle hit on a context switch? a context switch costs way more than that
[12:15] <nextgens> no, it's not on RISK processors
[12:15] <phrosty> every thread that switches in invalidates data cache (for stack space) that could be used for something else
[12:16] <nextgens> anyway, we shouldn't worry about such details
[12:16] <nextgens> we are coding a java app
[12:16] <toad_> phrosty: does the entire on-chip cache get flushed?
[12:16] <nextgens> widely guessing what the jvm will do is silly
[12:16] <toad_> phrosty: what exactly is the problem?
[12:16] <toad_> nextgens: we care about usability and performance. that means we care about such things.
[12:17] <toad_> nextgens: any factor that prevents a large number of people from running freenet AT ALL let alone 24x7 is a big deal; regardless of what layer it may originate on, there may be a fix which can be implemented at an accessible layer
[12:17] <nextgens> toad_> they are other things we should do first in terms of optimization
[12:17] <toad_> well, do you think that threads are a problem? there are certainly things we could do to reduce the thread count further
[12:17] <toad_> i'm not convinced though at this stage
[12:18] <toad_> nextgens: such as?
[12:18] <nextgens> we have obvious performances bottlenecks
[12:18] <toad_> for example?
[12:18] <nextgens> logging
[12:19] <nextgens> that would be the obvious one
[12:19] <toad_> FEC decoding/encoding on amd64 isn't a big issue, most systems which actually run in 64 bit have multiple cores
[12:19] <nextgens> yeah, that's why I haven't bothered working on it
[12:19] <toad_> nextgens: how so?
[12:19] <nextgens> our logger is damn slow
[12:19] <toad_> why is logging an issue with the normal priority settings?
[12:19] <nextgens> it does way too many IOs
[12:20] <toad_> yeah but have you seen logs at NORMAL? they're tiny
[12:20] <nextgens> it's not when everything works fine; but we are looking for border cases here
[12:20] <nextgens> arguably I don't run my node at NORMAL, no
[12:20] <toad_> well they're tiny except when something messes up and we end up resending a packet a zillion times....
[12:20] <nextgens> but NORMAL isn't useful :/
[12:21] <nextgens> or detecting ip addresses zillion times ;)
[12:21] <toad_> nextgens: we are talking about users, not devs
[12:22] <toad_> hmmm
[12:22] * toad_ greps a 5 minute slot ... 2216 normal 106 error
[12:22] <toad_> granted this is on the db4o branch
[12:22] <toad_> maybe normal is a bit too chatty
[12:23] <toad_> as regards the logger itself, we only flush() when we switch files or the queue is empty ...
[12:24] <nextgens> it's flushing much more than that here
[12:24] <toad_> but on a normal user's node the queue will be empty a lot of the time
[12:24] <nextgens> lsof shows that the node does 5IOs/sec on average here
[12:24] <toad_> 5?
[12:24] <nextgens> yes, on an idle node
[12:24] <toad_> why is that a problem?
[12:25] <toad_> ah so no queue?
[12:25] <toad_> well a lot of that is probably bdbje
[12:25] <nextgens> no queue, loglevel at normal
[12:25] * zerwas (i=zerwas@) has joined #freenet
[12:25] <toad_> sdiz's store will help a lot with that
[12:25] <toad_> nextgens: can you cat logs/freenet-previous.log | wc -l ?
[12:26] * toad_ sets mode +v zerwas
[12:26] <nextgens> 731 logs/freenet-latest.log
[12:26] <phrosty> toad_: freenet's got 250 threads going with 30min uptime now
[12:26] <nextgens> that's weird
[12:26] <nextgens> toad_> no need to voice gpg-tor users :)
[12:26] <toad_> nextgens: how long a period does it cover?
[12:26] <zerwas> thank you toad_
[12:26] <toad_> what's gpg-tor ?
[12:27] <zerwas> toad_, see my hostmask
[12:27] <nextgens> the ban is on %*!*@gateway/tor/*
[12:29] <phrosty> toad_: what does RequestSender do?
[12:29] <toad_> nextgens: okay so there are two issues here anyway: 1) we shouldn't flush more than every X ms, and 2) we should log less at NORMAL in the first place
[12:29] <toad_> nextgens: X should be customizable, but default to say 1000
[12:29] <toad_> or even 10*1000
[12:30] <toad_> phrosty: you have the thread status box for me? :)
[12:30] <toad_> thread usage
[12:30] <toad_> can you pastebin it?
[12:31] <phrosty> http://rafb.net/p/ktfZyi65.html
[12:31] <nextgens> luke771_afk> is the ita translation posted on devl fine? can it be merged?
[12:31] <nextgens> toad_> are you up to date with your code review process?
[12:31] <toad_> nowhere near
[12:32] <nextgens> fair enough :)
[12:32] <toad_> nextgens: would you be interested in fixing issue 1) ?
[12:32] <nextgens> I'll have a look
[12:32] <toad_> that'd be great
[12:32] <toad_> would you mind filing a bug if you don't get around to fixing it yourself?
[12:33] <nextgens> I'll have a look soon, possibly today
[12:33] <toad_> thanks
[12:33] <toad_> phrosty: the total is around 130, right?
[12:33] <toad_> no, 240ish
[12:33] <phrosty> around there
[12:34] <toad_> RequestSender does what it says, it's the sending portion of a request
[12:34] <toad_> the receiving portion is asynchronous
[12:34] <toad_> the sending portion isn't, to simplify the code
[12:34] <phrosty> is there a reason it couldn't be async too?
[12:34] <phrosty> ah
[12:34] <toad_> RequestStarter$SenderThread could be asynchronous
[12:35] <toad_> RequestSender itself *could* be asynchronous, but the code would probably be less intelligible if it was ...
[12:35] <toad_> Finish CHK transfer should be asynchronous
[12:35] <luke771_afk> nick Luke771
[12:35] <luke771_afk> arghù
[12:35] * luke771_afk is now known as Luke771
[12:35] <toad_> data transfers should be asynchronous
[12:36] <Luke771> nextgens: I havent's sent anything. I'll send next update at the end of the moth as I usually do, or earlier of I find somethying that needs to be fixed right away
[12:36] <Luke771> mont
[12:37] <toad_> Luke771: any opinions on the posted translations?
[12:37] <Luke771> some day I'll re-enable the spell checker
[12:37] <Luke771> what posted trasnlations?
[12:38] <phrosty> the best way i've found to do async is to have a callback queue - start async i/o with a completion callback, then have it push the callback onto the queue when it's done, with a small pool of threads popping them off the queue. does Java do it that way?
[12:38] <nextgens> Luke771> a mail from luscalf
[12:38] <toad_> 21.5+8.3+0.8+0.8 = 31.4% ...
[12:38] <nextgens> Luke771> Wed, 02 Jul 2008 00:27:24 +0200
[12:38] <Luke771> the ITA translation that comes with the latest versions are almost good, a couple of string left to translate, I'll send those + some typos with next update
[12:38] <toad_> hmmm no
[12:38] <toad_> not the 21.5%
[12:39] <Luke771> wed 02 jul, that's last one I sent yes
[12:39] <toad_> 8.3+0.8+0.8 = ~ 10%
[12:39] <toad_> with asynchronous block/bulk data transfer
[12:39] <toad_> and asynchronous SenderThread
[12:40] * nOgAnOo (i=nOgAnOo@) has joined #freenet
[12:40] <Luke771> yes nextgens you can commit the 07-02 trasnlation, next update will come in a couple of weeks
[12:40] <nextgens> Luke771> okay so that's your email address ?
[12:40] <Luke771> yes
[12:40] <Luke771> luscalf, that's me
[12:40] <toad_> sorry, /me is miscounting again ... 8.3% for RequestStarter$SenderThread (relatively easy), 5.4%+1.2%+0.8%+0.8% for Finish CHK transfer, and a couple of data transfer related ones
[12:40] <Luke771> also luke771 at gmail
[12:41] <nextgens> really, you should sign your emails
[12:41] <nextgens> "LS" is way too far from Luke771 for me to reconize it
[12:41] <Luke771> usually I senty to @devl using luke771, I was sure I'd done that this tiome too, and I tjhought it wasnt necressary to sign because you knew the address... except that I sent it from the wrong address
[12:42] <Luke771> yeah, that was a mistake, sorry, I intended to send it from luke771
[12:43] * caytchen (n=caytchen@) has joined #freenet
[12:44] <toad_> we could probably get rid of Send throttled SSK data (0.8%), Background block encoder (0.4%), BlockTransmitter.sendAsync() (0.4%, should be in the above w.r.t. data transfer), maybe FT offers exec (0.4%), IP redetector, ...
[12:45] <toad_> so 20% with moderately significant changes
[12:45] <FreenetLogBot> r21145 (1153) was built successfully on emu, mirrors are updating
[12:47] <phrosty> preoccupied writing an email for dev list, don't mean to be ignoring you
[12:47] <toad_> but that's really not such a huge gain
[12:47] <toad_> phrosty: about thread usage?
[12:47] <toad_> it's an interesting theory
[12:47] <phrosty> yea
[12:48] <toad_> bear in mind that java is generally built to be fairly heavily threaded, that there is no asynchronous disk i/o and that asynchronous network i/o apart from UDP is a major PITA
[12:48] <phrosty> ah :(
[12:48] <toad_> of course we do use callbacks extensively
[12:49] <toad_> however, we only use TCP to talk to fproxy and fcp clients
[12:49] <toad_> so that's not a big part of the total anyway
[12:50] <toad_> rewriting the request senders as state machines isn't completely out of the question, but i'd need to be convinced the code is not only faster but also clearer
[12:50] <toad_> the old 0.5 state machines were by no means clear
[12:50] <toad_> it is possible to import a continuations library to keep more or less the same code and still have them asynchronous, but i'm not sure it's warranted
[12:50] <toad_> and it has a lot of limitations
[12:51] <phrosty> language support like closures definitely help clarity a lot, but after using pure async for a while the design is pretty easy to follow.. just different
[12:51] * nextgens isn't sure he gets the gist of toad's deadlock-fix to the updater
[12:51] <toad_> nextgens: ummm, something to do with evil nested locking??
[12:52] <nextgens> first patch is arguably okay
[12:52] <nextgens> second definitly isn't
[12:52] <nextgens> "IMHO reading a boolean doesn't need to be synchronized.
[12:52] <nextgens> "
[12:52] <nextgens> that's a false statement
[12:52] <toad_> really?
[12:52] <nextgens> yes
[12:52] <nextgens> unless the boolean is volatile and we are post-java1.5 it's false
[12:52] <toad_> why? i mean, java will do a re-lookup anyway because it's inside a function call, won't it?
[12:53] <nextgens> no, because of auto-inlining
[12:53] <toad_> hmmm
[12:53] <toad_> well you figure out how to avoid deadlock then :)
[12:53] <nextgens> do you still have the deadlock notice ?
[12:54] <toad_> no
[12:54] <nextgens> hmm, okay
[12:54] <toad_> on one path we lock the NodeUpdateManager and then the NodeUpdater, on the other path we lock the NodeUpdater and then the NodeUpdateManager
[12:54] <toad_> that's the basic problem
[12:55] <toad_> it might be possible to have them both lock on the NUM
[12:55] <nextgens> or to use a volatile boolean for the getter
[12:55] <toad_> iirc that's the tip of the iceberg
[12:55] <nextgens> and simplify the whole locking stuffs
[12:56] <toad_> which boolean were we fetching?
[12:56] <nextgens> isRunning
[12:57] <nextgens> from what I can see here it's not even called
[12:57] <nextgens> I don't see how it could cause a deadlock: it's dead code
[12:57] <toad_> it is called from at least 3 places in NUM
[12:57] <nextgens> or netbeans is playing me some dark magic
[12:57] <toad_> enable, isEnabled and setURI
[12:58] <nextgens> the getter ?
[12:58] <toad_> however in isEnabled and setURI, we unlock NUM before we call it
[12:58] <nextgens> in r21090 you unsynchronize the getter
[12:58] <nextgens> I don't see where it's called
[12:58] <toad_> in enable() we call isRunning() while locked on NUM
[12:59] <nextgens> okay
[13:00] * toad_ wonders if we could get rid of isRunning
[13:01] <nextgens> I don't think we can
[13:01] <nextgens> iirc its purpose is to allow a gracefull "unload"
[13:02] <toad_> well, why do we need to access it from NUM?
[13:02] <toad_> can't we just null out the NodeUpdater when we call kill() ?
[13:03] <toad_> there's onChangeURI(), but that doesn't work anyway, isRunning is never set back to true
[13:03] <toad_> in blow() and enable() we'd just null it out
[13:04] <toad_> so we can probably eliminate isRunning
[13:04] <toad_> well
[13:04] <toad_> isRunning()