Timestamps are in GMT/BST.
[0:03] * Mathiasdm (n=Mathias@) Quit ("bye")
[0:16] * ahuxley is now known as ahuxley_afk
[0:22] <FreenetLogBot> r22750 (1164) was built successfully on emu, mirrors are updating
[0:25] <FreenetLogBot> r22751 (1164) was built successfully on emu, mirrors are updating
[0:25] <FreenetLogBot> r22752 (1164) was built successfully on emu, mirrors are updating
[0:28] <FreenetLogBot> r22753 (1164) was built successfully on emu, mirrors are updating
[0:39] * salazar151 (i=salazar@) has joined #freenet
[0:45] * ordex (n=linuxaro@) Quit (Read error: 110 (Connection timed out))
[0:48] * salazar151 (i=salazar@) Quit ("Leaving")
[0:52] * PhrostByte (n=phrosty@) has joined #freenet
[0:52] * phrosty (n=phrosty@) Quit (Read error: 113 (No route to host))
[1:10] * toad_ (n=toad@) Quit (Remote closed the connection)
[1:15] * NEOatNHNG1 (n=neo@) has joined #freenet
[1:17] * zerwas (n=zerwas@) Quit (Read error: 110 (Connection timed out))
[1:31] * n0ob (n=travis@) has joined #freenet
[1:34] * NEOatNHNG (n=neo@) Quit (Read error: 113 (No route to host))
[1:47] * Luke771 (n=luke@) has joined #freenet
[1:47] * ChanServ sets mode +v Luke771
[2:19] * gasi_ (n=gasi@) has joined #freenet
[2:24] * ahuxley_afk is now known as ahuxley
[2:33] * gasi (n=gasi@) Quit (Read error: 110 (Connection timed out))
[2:43] * pmpp (n=pmpp@) has joined #freenet
[2:46] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror2.freenetproject.org/ : it has been disabled
[2:58] <FreenetLogBot> The monitoring script has detected that http://mirror2.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[3:14] * TheBishop_ (n=TheBisho@) Quit (Read error: 110 (Connection timed out))
[3:14] * TheBishop_ (n=TheBisho@) has joined #freenet
[3:27] * Spooof (n=spoof@) has joined #freenet
[3:30] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror2.freenetproject.org/ : it has been disabled
[3:35] * Zarggg (n=z@) Quit ()
[3:37] * Zarggg (n=z@) has joined #freenet
[3:43] * Zarggg (n=z@) Quit ()
[3:44] * Ratchet__ (n=xxxx@) has joined #freenet
[3:49] * Zarggg (n=z@) has joined #freenet
[3:52] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror1.freenetproject.org/ : it has been disabled
[3:54] * Luke771 (n=luke@) Quit ("[add smart/funny quote here]")
[3:55] * Luke771 (n=luke@) has joined #freenet
[3:55] * ChanServ sets mode +v Luke771
[4:00] * Ratchet (n=xxxx@) Quit (Read error: 110 (Connection timed out))
[4:22] <FreenetLogBot> The monitoring script has detected that http://mirror1.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[4:27] * Zarggg (n=z@) Quit ()
[4:37] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror1.freenetproject.org/ : it has been disabled
[4:37] * TheBishop_ (n=TheBisho@) Quit ("Verlassend")
[4:38] * n0ob (n=travis@) Quit ("Now onto something more interesting")
[4:52] <FreenetLogBot> The monitoring script has detected that http://mirror1.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[5:06] * FrinkC (n=FrinkC@) Quit (Read error: 54 (Connection reset by peer))
[5:06] * FrinkC (n=FrinkC@) has joined #freenet
[5:07] * Luke771 (n=luke@) Quit ("[add smart/funny quote here]")
[5:27] <FreenetLogBot> The monitoring script has detected that http://mirror2.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[5:39] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror1.freenetproject.org/ : it has been disabled
[5:54] <FreenetLogBot> The monitoring script has detected that http://mirror1.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[5:55] * TheBishop_ (n=TheBisho@) has joined #freenet
[7:02] * Mathiasdm (n=Mathias@) has joined #freenet
[7:05] * Ratchet__ (n=xxxx@) Quit ("Reconnecting")
[7:05] * Ratchet (n=xxxx@) has joined #freenet
[7:18] * TheBishop_ (n=TheBisho@) Quit ("Verlassend")
[7:19] * mikusr (n=mikusrak@) has joined #freenet
[7:32] * Mathiasdm (n=Mathias@) Quit ("bye")
[7:37] * ordex (n=linuxaro@) has joined #freenet
[7:47] * saces (n=saces@) Quit (Read error: 110 (Connection timed out))
[7:48] * saces (n=saces@) has joined #freenet
[7:48] * ChanServ sets mode +o saces
[7:56] * PhrostByte is now known as phrosty
[8:04] <ordex> hi
[8:09] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror1.freenetproject.org/ : it has been disabled
[8:21] * Mathiasdm (n=Mathias@) has joined #freenet
[8:25] * kalkin- (n=kalkin-@) has joined #freenet
[8:28] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror2.freenetproject.org/ : it has been disabled
[8:38] <FreenetLogBot> The monitoring script has detected that http://mirror1.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[8:53] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror1.freenetproject.org/ : it has been disabled
[9:08] <FreenetLogBot> The monitoring script has detected that http://mirror1.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[9:29] <FreenetLogBot> The monitoring script has detected that http://mirror2.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[9:34] * Mathiasdm (n=Mathias@) Quit (Read error: 113 (No route to host))
[9:38] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror1.freenetproject.org/ : it has been disabled
[9:44] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror2.freenetproject.org/ : it has been disabled
[9:49] * infinity0 (n=infinity@) has joined #freenet
[9:52] <FreenetLogBot> The monitoring script has detected that http://mirror1.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[10:13] <FreenetLogBot> The monitoring script has detected that http://mirror2.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[10:17] * Johan^mlg (n=bllarf@) has joined #freenet
[10:26] <FreenetLogBot> r22754 (1164) was built successfully on emu, mirrors are updating
[10:26] <FreenetLogBot> r22756 (1164) was built successfully on emu, mirrors are updating
[10:26] <FreenetLogBot> r22757 (1164) was built successfully on emu, mirrors are updating
[10:26] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror2.freenetproject.org/ : it has been disabled
[10:40] <ordex> do you think is a good idea to hold freenet cache on a crypted partition??
[10:40] <ordex> mh..i hink it is giving too overhead to disk i/o operation
[10:45] <FreenetLogBot> r22758 (1164) was built successfully on emu, mirrors are updating
[11:00] <FreenetLogBot> The monitoring script has detected that http://mirror2.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[11:03] <FreenetLogBot> r22759 (WoT) was built successfully on emu, mirrors are updating
[11:12] <FreenetLogBot> r22760 (WoT) was built successfully on emu, mirrors are updating
[11:12] <FreenetLogBot> r22761 (WoT) was built successfully on emu, mirrors are updating
[11:27] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror2.freenetproject.org/ : it has been disabled
[11:39] <FreenetLogBot> r22762 (WoT) was built successfully on emu, mirrors are updating
[11:39] <FreenetLogBot> r22763 (WoT) was built successfully on emu, mirrors are updating
[11:56] * Luke771 (n=luke@) has joined #freenet
[11:56] * ChanServ sets mode +v Luke771
[11:59] <FreenetLogBot> The monitoring script has detected that http://mirror2.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[11:59] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror4.freenetproject.org/ : it has been disabled
[12:00] <Luke771> networkSizeEstimateSession: 105 nodes
[12:00] <Luke771> nodeUptime: 3h30m
[12:00] <Luke771> Other node:
[12:01] <Luke771> networkSizeEstimateSession: 133 nodes
[12:01] <Luke771> nodeUptime: 21h46m
[12:01] <Luke771> wtf is happening?
[12:02] * Mathiasdm (n=Mathias@) has joined #freenet
[12:03] <Luke771> anyone else having similar problems?
[12:06] <Luke771> bug 2629
[12:10] * toad_ (n=toad@) has joined #freenet
[12:10] * ChanServ sets mode +o toad_
[12:13] * mikusr (n=mikusrak@) has left #freenet
[12:15] <Ratchet> Luke771: after restart, my node too only shows 114 node estimate
[12:15] <Ratchet> after 21h uptime
[12:15] <toad_> hi
[12:15] <toad_> imho that's normal
[12:16] <toad_> only the 5% or so of nodes that are pure darknet will initiate swaps now
[12:16] <toad_> maybe we should get rid of the indicator?
[12:20] <Ratchet> would be better, yes
[12:21] <toad_> does it show in non-advanced mode?
[12:21] <Ratchet> toad_: there's a discussion going on on fms if you're really planing to switch the default database to salted hash in the next release
[12:21] <Ratchet> peaple seem confused :-)
[12:22] <Ratchet> *people
[12:22] <toad_> it depends partly on what feedback we get on its stability
[12:22] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror1.freenetproject.org/ : it has been disabled
[12:22] <Ratchet> I don't think this is the main concern
[12:22] <toad_> what is?
[12:22] <Ratchet> it's the migration taking up double disk space
[12:22] <toad_> we will not force auto-migration in any case
[12:22] <toad_> if we make it the default it will only be the default FOR NEW NODES
[12:22] <Ratchet> an losing datastore completeley on full disk
[12:22] <Ratchet> ah! :-)
[12:23] <toad_> this is easy cos the datastore type is always written to disk anyway iirc
[12:23] <toad_> if not we'll change that first
[12:23] <Ratchet> you're not using fms any more?
[12:23] <toad_> what makes you think that?
[12:24] <Ratchet> haven't seen any posts of you since I'm back.
[12:24] <Luke771> OK, so it's normal. Good to know.
[12:24] <toad_> awww fsck
[12:24] <Ratchet> and I thought this discussion would be worth an answer of you :-)
[12:24] <Luke771> what about the guy who had zero?
[12:24] <toad_> it's not an alwaysWriteToDisk option
[12:25] <Ratchet> since even sdiz wasn't sure what you're planing to do
[12:25] <toad_> okay
[12:25] <toad_> well the next build will make storetype an always-write-to-disk option
[12:25] <toad_> that's the first thing
[12:25] <toad_> meaning if we change the default it only affects new nodes
[12:26] * phrosty (n=phrosty@) Quit (wolfe.freenode.net irc.freenode.net)
[12:26] * edt (n=Ed@) Quit (wolfe.freenode.net irc.freenode.net)
[12:26] * FrinkC (n=FrinkC@) Quit ("Bye")
[12:28] <FreenetLogBot> r22764 (1164) was built successfully on emu, mirrors are updating
[12:28] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror2.freenetproject.org/ : it has been disabled
[12:28] <FreenetLogBot> The monitoring script has detected that http://mirror4.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[12:28] <FreenetLogBot> r22765 (1164) was built successfully on emu, mirrors are updating
[12:28] * edt (n=Ed@) has joined #freenet
[12:28] * phrosty (n=phrosty@) has joined #freenet
[12:29] * phrosty (n=phrosty@) Quit (Connection reset by peer)
[12:29] <toad_> You don't need 200GB for migrate,
[12:29] <toad_> you just need the size of CHK-store.* plus some overhead.
[12:29] <toad_> ahhh fair point
[12:30] <toad_> well only with sparse files :|
[12:30] <Luke771> maybe it would possible to pop up a one-time warning that tells the user about the new datastore being ready for use non non-experimental nodes, and also warn that the migration will take time and will use 2x the size of the current store, and that it is possible to keep using the old kind of datastore.
[12:30] <toad_> nah, i'd just leave them on bdb, and recommend they switch but warn them
[12:31] <Luke771> OK if the very low number of nodes in the net size estimate is normal, you can close the bug 2629, that's about that problem-
[12:32] <Luke771> why is it counting so few nodes anyway? Have you changed the way they're counted?
[12:33] <Luke771> nah, i'd just leave them on bdb, and recommend they switch but warn them <== that's pretty much what I said. Warn them, not run the migration automatically
[12:33] <Luke771> anyhow I have to go for a while bbl
[12:35] <Luke771> (answer anyway, I'll read when I come back, or I'll read the logs on emu)
[12:35] * Luke771 is now known as luke771_afk
[12:35] <Ratchet> toad_: I have my datastore for testing purposes on a USB flash drive. I can turn off bloomfilter with the new store?
[12:36] <toad_> Ratchet: yes, it's not needed on a flash disk
[12:36] <Ratchet> ok
[12:36] * Mathiasdm (n=Mathias@) Quit (Read error: 110 (Connection timed out))
[12:37] * Mathiasdm (n=Mathias@) has joined #freenet
[12:43] <Ratchet> after migration, will the old db files automatically deleted?
[12:43] <FreenetLogBot> The monitoring script has detected that http://mirror2.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[12:43] <toad_> Ratchet: i assume so
[12:43] <Ratchet> ok
[12:45] * toad_ caught up on FMS
[12:45] <toad_> but to be fair i posted some stuff on saturday
[12:45] * toad_ wonders if it got lost somehow
[12:46] <Ratchet> perhaps it fell in a sparse file hole? :-)
[12:46] <toad_> :|
[12:48] <toad_> what's the latest version of Thaw ?
[12:49] * StefftheEmperor (n=Steffthe@) has joined #freenet
[12:52] <ordex> Thaw 0.8.4 r18453
[12:52] <ordex> this probably
[12:52] <ordex> i dunno if it is the last revision too
[12:55] <FreenetLogBot> The monitoring script has detected that http://mirror1.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[12:56] <toad_> w.r.t. the new store, have people tried shrinking a store when there are 0 bytes free? what happens?
[12:56] <toad_> it ought to work but it would be good to have confirmation of that
[12:56] <toad_> luke771_afk: ???
[13:01] <FreenetLogBot> r22766 (1164) was built successfully on emu, mirrors are updating
[13:03] <ordex> never tried
[13:07] * Zero3 (i=5550d67b@) has joined #freenet
[13:08] <Zero3> toad_: The browse.cmd bug is fixed! Just made a fresh install, and it works great now. I can't mark the bug as "resolved" in the bugtracker though.
[13:18] * kalkin- (n=kalkin-@) Quit (Read error: 113 (No route to host))
[13:21] <toad_> Zero3: thanks
[13:21] <toad_> Zero3: which bug?
[13:24] <Zero3> You got it :)
[13:35] * Johan^mlg (n=bllarf@) Quit (Read error: 104 (Connection reset by peer))
[13:35] * Johan^mlg (n=bllarf@) has joined #freenet
[13:36] * caytchen (n=caytchen@) has joined #freenet
[13:51] <FreenetLogBot> r22767 (1164) was built successfully on emu, mirrors are updating
[13:51] <FreenetLogBot> r22768 (1164) was built successfully on emu, mirrors are updating
[13:57] <luke771_afk> luke771_afk: ??? toad_ : couldnt do anything but nuke the whole store. sorry
[13:58] * sdiz (n=sdiz@) has joined #freenet
[13:58] * ChanServ sets mode +o sdiz
[13:59] <luke771_afk> now it looks like the other one: 400+ used, 400+ free, in a 500GB disk. but you said that was OK (or I got it wrong)
[14:01] <toad_> luke771_afk: oh because it couldn't write freenet.ini ?
[14:01] <toad_> luke771_afk: and it didn't manage to start up so you couldn't do it on the config page?
[14:01] <toad_> sdiz: hey
[14:02] <toad_> sdiz: we need to talk about "release criteria" for the salted hash store - critical bugs, etc
[14:03] * ahuxley (n=ahuxley@) Quit ()
[14:03] <toad_> sdiz: clearly we can't auto-migrate old bdbje stores because of the disk space issues ... but for new nodes it's *very* interesting
[14:04] <luke771_afk> toad_: I tried as hard as I could but I couldn't even edit freenet.ini. I tried deleting less important files like logs etc, and also moving out of the way anything that I could: nothing helped. The free space was still zero byte despite at least 20MB moved, probably more. When I saw that I din't have anything left I could move and the space left was still zero and I couldn't edit freenet.ini, I decided to delete the store
[14:04] <toad_> luke771_afk: so it was still running, but you couldn't get to the config page?
[14:05] * NEOatNHNG1 (n=neo@) Quit ("Leaving.")
[14:06] <luke771_afk> no I couldn't do anything, it was running but it was hung waiting for free space
[14:06] <toad_> you mean it was in the middle of migration? or something?
[14:06] <toad_> was the web interface responsive? was it still starting up?
[14:06] <luke771_afk> I could come to the config page before I restarted it (I don't remember exactly why I did that) but after restarting it refused to start
[14:07] <luke771_afk> yes it was in the moddle of something
[14:07] <luke771_afk> datastore mainteinance IIRC
[14:07] <toad_> ok so because it was doing datastore maintenance the web ui wasn't accessible?
[14:07] <luke771_afk> the web interface was up and acessible
[14:07] <toad_> then why couldn't you tell it to shrink from the config page?
[14:07] <luke771_afk> but the log talked about problems because of lack of disk space
[14:08] <toad_> i suppose because it was busy, it wouldn't shrink until after it had finished maintenance?
[14:08] <luke771_afk> so I did something, I dont remeber exactly what, then I tried to restart but it wouldn't start because of lack of disk space
[14:08] <luke771_afk> it wouldn't shrink because I made a mistake
[14:08] <toad_> hmmm wrapper problem?
[14:08] * luke771_afk is now known as Luke771
[14:09] <Luke771> it hit the disk size limit
[14:09] <toad_> oh wait a minute...
[14:09] <Luke771> the physical disk limit
[14:09] <toad_> maintenance, shrinking, all of it ... will un-hole the holes
[14:09] <toad_> because we read a blob of data, and then write it
[14:10] <Luke771> because I'd set the max size to 475GB, which was interpreted as 475GiB and I didt know that my 500GB HDD wasn't really 500GiB but more like 500000MB or something
[14:10] <toad_> we don't write it if nothing has changed, but if there is one non-hole entry in it, and that changes (which it will even if it's only updating the bloom filters) ... the holes will be written, and therefore turned into non-holes
[14:10] <Luke771> really, I don't understand this part
[14:10] <toad_> Luke771: yeah, we should check the disk space on setting the size, if the JVM is 1.6
[14:11] * toad_ wonders if preallocating space would in fact be the most user-friendly option
[14:11] <Luke771> all I can say is that after deleteing the store, the new store got blown to an unbelieveable 450+GB in a few minutes, but the free space was still 400GB +
[14:11] <Luke771> but you said that it's OK, the holes thing etc
[14:11] <toad_> Luke771: that's normal, ignore it
[14:11] <Luke771> yeah
[14:12] <toad_> but if maintenance is gonna use all that space up, we have a problem
[14:12] <toad_> sdiz: any thoughts?
[14:12] <Luke771> I think you should warn users about this 'side effect', some people could freak out and uninstall
[14:12] <Luke771> oh btw
[14:12] <toad_> side effect of what?
[14:12] <Luke771> since I switched to salt-hash, the node has always being doing some mainatinance
[14:13] <toad_> rebuilding the bloom filter because your node crashed??
[14:13] <Luke771> the side effect that sometimes the used space and the free space in the freenet partition may not add up
[14:13] <Luke771> hm...
[14:13] <toad_> Luke771: i don't see why anyone would care
[14:13] <Luke771> possible
[14:13] <Luke771> well it got me worried
[14:13] <Luke771> anyhow
[14:13] <toad_> well we have to rebuild the bloom filter if the node crashes
[14:13] <toad_> but i think it will say that if it does
[14:14] <toad_> i dunno about other kinds of maintenance
[14:14] <toad_> sdiz: either start talking or go away :)
[14:14] <Luke771> if I just reboot my box without shutting down freenet first, would the node count that as a crash and rebuild the bloom filter? (I guess that's what's happening with my continuous maintainance)
[14:14] <toad_> yes
[14:15] <Luke771> in that case, that's another thing that users should be warned about: lots of them just shut down
[14:15] <toad_> well it depends on how exactly the shutdown happens
[14:15] <Luke771> OK, I'll shut down the node properly next time
[14:15] <toad_> on windows hopefully the node will be cleanly shut down
[14:15] <Luke771> what happens is that
[14:15] <Luke771> most nodes autostart at startup, login or whenever they start up
[14:16] <toad_> on linux if it's an init service, it should be shutdown cleanly
[14:16] <toad_> i dunno about with a cron job though
[14:16] <Luke771> so users just shut down or reboot their box, without caring abouot shutting down the node properly
[14:16] <Luke771> now, with salt-hash, that's gonna be a problem
[14:16] <Luke771> maybe we need a shut down script cponnected to the reboot action
[14:16] <toad_> ok so there are 10 million problems all of them insoluble that prevent us from using saltedhash by default :|
[14:17] <Luke771> sadly so
[14:17] <toad_> sdiz will be happy :)
[14:17] <toad_> anyway, we need to solve them: for example, unclean shutdown is bad in MANY ways
[14:17] <toad_> it breaks the connectivity detection
[14:17] <sdiz> the store size in config page does not include overhead.
[14:17] <Luke771> but the good part is that I'm reporting lots of things that have to be done so you can start organizing the work, and hopefully everything will be ready to go default soon
[14:17] <sdiz> it's just the chk/ssk/pubkey header+data size.
[14:18] <toad_> on both windows and linux, the OS will try to shut down services properly ... cron jobs on linux i have no idea
[14:18] <toad_> sdiz: iirc we include overhead on the bdbje store
[14:18] <Luke771> i don't even had a cron job
[14:18] <toad_> sdiz: what is maintenance?
[14:18] <sdiz> `shutdown` would send SIGTERM before it kill
[14:18] <toad_> Luke771: well then you must have an init script, right?
[14:19] <Luke771> I installed my nodes way before you included that, and I never bothered adding one manually
[14:19] <sdiz> toad_: it don't include overhead in bdbje store
[14:19] <toad_> Luke771: so you start it manually?
[14:19] <sdiz> toad_: maintenance == rebuild bloom filter
[14:19] <toad_> sdiz: ok
[14:19] <toad_> yeah but shutdown only gives a few seconds from "Sending all processes the TERM signal" to "Sending all process the KILL signal"
[14:20] <Luke771> all I did was to add a symlink to the start script in my home dir, and use the ubuntu feature sessions > startup (simiar to the windows startup folder) to run the startup script at login
[14:20] <toad_> we may not successfully shut down the store in that time if we're unlucky, overloaded etc
[14:20] <toad_> Luke771: ugh
[14:20] <Luke771> technically, it's the equivalent of starting manually
[14:20] <Luke771> er, manually running the startup script, I mean
[14:20] <toad_> Luke771: check your wrapper.log
[14:20] <sdiz> if you shutdown the node before the rebuild done, it would start from zero on next node start
[14:21] <Luke771> k
[14:21] <toad_> Luke771: it will be clear from the wrapper.log
[14:21] <Luke771> what am I looking for?
[14:21] <toad_> whether the node shutdown successfully
[14:21] <Luke771> k
[14:21] <toad_> sdiz: hmmm ok
[14:21] <toad_> sdiz: can that be fixed easily? assuming a clean shutdown?
[14:22] <sdiz> hmmm, not start from zero ..... each store/cache is independant.
[14:22] <toad_> hmmm
[14:22] * HyperDimensions (n=HyperDim@) has joined #freenet
[14:22] <toad_> and they're all the same size, we're only munching metadata here
[14:23] <toad_> so it's not completely busted
[14:23] <sdiz> hmmm
[14:23] * cdent (n=cdent@) has joined #freenet
[14:23] * THEapophis (n=apophis@) has joined #freenet
[14:23] <toad_> it would be best if we could save the progress in the manifest file on every checkpoint though, or on some checkpoints
[14:23] <Luke771> toad_: no clean shutdown. Abrupt interruption, then start line
[14:23] <toad_> Luke771: ugh
[14:23] <toad_> Luke771: does it try to start to shut down?
[14:24] <Luke771> ?
[14:24] <Luke771> oh
[14:24] <Luke771> no
[14:24] <toad_> it should say so
[14:24] <toad_> hmmm
[14:24] * kryptos23_ (n=kryptos@) has joined #freenet
[14:24] <toad_> so why does ubuntu not send it the TERM signal?
[14:24] <Luke771> no it doesnt say anything about shut down procedures
[14:24] <sdiz> hmm.. the problem is, the "new" bloom filter is not written to disk until it is done. if you shutdown during the process, it is lost.
[14:25] <Luke771> it just starts with a --> wrapper started as a aemon
[14:25] <toad_> sounds like it's impatient... :(
[14:25] <toad_> Luke771: so it just stops, then it starts
[14:25] <toad_> Luke771: aaaaaaaaargh
[14:25] <Luke771> maybe because I don't havew the cronjob
[14:25] <Luke771> let it test by someone who has it
[14:25] <toad_> I suppose it's a reason to put in an init script ... but init scripts are hard to put in in a cross platform way...
[14:26] <toad_> Luke771: that would be you ... put the cronjob in and see if it still breaks in the same way
[14:26] <sdiz> you can't rename a file opened with mappedbytebuffer on windows,.
[14:26] <Luke771> meh
[14:26] <Luke771> is there a pre-packaged cronjob to download?
[14:27] * Zero3 (i=5550d67b@) Quit ("http://www.mibbit.com ajax IRC Client")
[14:28] <toad_> Luke771: @reboot /home/freenet/run.sh start
[14:28] <THEapophis> ha: thats a good one: There isn't enough entropy available on your system... Freenet won't start until it can gather enough.
[14:29] <THEapophis> How do I do that without a keyboard and a mouse? ;)
[14:29] <toad_> and of course there's no @shutdown
[14:29] <toad_> THEapophis: freenet should deal with it itself
[14:29] <THEapophis> it does :) ty
[14:29] <toad_> THEapophis: if it doesn't, just do find /
[14:29] <toad_> hard disk timings are another source of entropy
[14:30] <toad_> sdiz: am i right about rebuilding and de-holing ?
[14:30] <THEapophis> http://apophis.ch:8888/ is back, for the ones who asked lately
[14:30] <toad_> sdiz: what do you think about preallocating as soon as possible after making a salted hash store?
[14:30] <sdiz> toad_: de-holing ?
[14:30] <sdiz> toad_: on windows, it preallocate
[14:31] <toad_> sdiz: you read 128 entries. one of them is used, the rest are all zeros. so you increment the generation on that one, write it all back. writing it all back clobbers the holes, since holes are only created by seek().
[14:31] <toad_> sdiz: NTFS has hole support - is it off by default?
[14:32] <sdiz> the only way to preallocate on unix is write the whole file once, --- it take very long time
[14:32] <sdiz> java don't support ntfs sparse file
[14:32] <toad_> it takes exactly the same time as it would on windows
[14:32] <sdiz> it need special API to create
[14:32] <toad_> cos it's the same disk operation
[14:32] <toad_> sdiz: eh?
[14:32] <toad_> sdiz: seek() creates a hole
[14:32] <sdiz> special API to create sparse file on windows
[14:32] <toad_> you're probably thinking of something else
[14:32] <toad_> there are lots of different filesystem things that are called "sparse"
[14:33] <sdiz> seek() on windows would preallocate the blocks
[14:33] <toad_> without actually zeroing them? hmmm
[14:33] <toad_> so very fast
[14:34] <toad_> well, we could do background preallocation, fill with lots of random data from a mersenne twister?
[14:35] * Luke771 (n=luke@) Quit ("[add smart/funny quote here]")
[14:35] <sdiz> hmm.. will try.
[14:35] <toad_> configurable of course
[14:35] <Ratchet> ok, what's the setting for freenet.ini if I want my old datastore type back?
[14:35] <toad_> Ratchet: what's wrong with the new one? :|
[14:35] <Ratchet> it's taking too long on the usb drive
[14:35] <Ratchet> I have to do that later on harddisk
[14:35] <toad_> Ratchet: the .hd change should be the last catastrophic format change
[14:36] <toad_> Ratchet: everything is too slow on a USB drive
[14:36] <toad_> Ratchet: most of them are like 2MB/sec
[14:36] <toad_> Ratchet: migration will take ages sure, is that the problem?
[14:37] <Ratchet> yes, I'll do it later
[14:37] <Ratchet> so what's the setting for the old storetype?
[14:37] <toad_> Ratchet: look for storeType
[14:37] <Ratchet> i know the key, I need the value :-)
[14:37] <toad_> bdb-index
[14:37] <Ratchet> thx
[14:38] <toad_> sdiz: it would improve security ... and predictability ... predictability is a good thing, so i think we should probably have preallocation turned on by default
[14:39] <toad_> sdiz: assuming we're not expecting more incompatible store changes that require moving tons of data around?
[14:47] <sdiz> hmm, i hope we don't have to
[14:49] <toad_> i hope so too
[14:50] <toad_> so what needs to be done before we can make salted hash the default for new nodes?
[14:50] <toad_> as regards bloom filters, imho 64MB is reasonable, so in our disk space autodetection code we can just not create a store bigger than 128GB
[14:51] <toad_> is it possible to make the store more tolerant of low disk space?
[14:53] <sdiz> it just crash on out of disk space, the data is not corrupted
[14:53] <toad_> yeah but people end up deleting their store to make it work ... i suppose all we can realistically do is preallocate, and check disk space before accepting a new store size?
[14:55] <toad_> hmmm w.r.t. bloom filters, 100GB divided by 32K+32K+2K+2K+1K+1K times 128 times 6 is around 1.1GB of entries for 100GB of data...
[14:56] <toad_> so after an unclean shutdown we need to read 1.1GB of data for every 100GB in the store
[14:56] <toad_> but it's only 190MB per store, and we mark each one after it's done, so it can be happily restarted...
[14:57] <toad_> so it's not a big deal, unless we're unable to prevent large numbers of unclean shutdowns
[14:58] <toad_> anything else we need to do? apart from migration, testing, and bloom reconstruction, feedback has been almost entirely positive...
[14:58] <toad_> err apart from migration, out of disk space, and bloom reconstruction
[14:59] <toad_> (when I say we i mean it - let me know if you need me to debug something)
[14:59] <ordex> my java process is still getting 160% of cpu :/
[14:59] <toad_> ordex: uptime?
[14:59] <ordex> process started about a day ago
[15:00] <toad_> sdiz: one other thing I wanted to ask ... how much slower is saltedhash with bloom turned off? how does it compare to bdbje?
[15:00] <toad_> ordex: get another thread dump :|
[15:00] <ordex> ok :\
[15:01] <ordex> wait
[15:01] <ordex> fpc says
[15:01] <ordex> # nodeUptime: 36m42s
[15:01] <ordex> is it normal?
[15:01] <toad_> if you have a lot of stuff queued it could still be resuming
[15:01] <toad_> ah well then that explains it
[15:01] <ordex> why does the uptime be resetted?
[15:01] <toad_> look at your wrapper.log, why did your node restart?
[15:01] <ordex> ok
[15:04] <ordex> toad_: could be this exception
[15:04] <ordex> Caught exception, closing database: pubkey-store- (com.sleepycat.je.DatabaseException: (JE 3.3.69) Keys in database: 0 but keys in file: 2190)
[15:04] <toad_> ordex: no, that's part of the startup process
[15:04] <ordex> i'm not understanding in which point it restarted
[15:04] <ordex> ok
[15:04] <toad_> ordex: send me your wrapper.log
[15:04] <ordex> ok
[15:05] <ordex> java.lang.OutOfMemoryError: GC overhea
[15:05] <ordex> d limit exceeded
[15:05] <ordex> what do you thin kabout this?
[15:05] <toad_> eeeek
[15:05] <toad_> that's not good
[15:06] <toad_> have you tried increasing your memory limit?
[15:06] <ordex> i could try
[15:06] <ordex> at the moment maximum memory is 512 MB
[15:07] * p0s (n=1mn57@) has joined #freenet
[15:07] * ChanServ sets mode +o p0s
[15:07] <ordex> i thought it was good enough
[15:07] <toad_> hmmm it ought to be
[15:07] <toad_> but then why so much GC?
[15:07] <toad_> what's your logging set to?
[15:07] <toad_> and how many gigabytes do you have in total on your queue?
[15:08] <toad_> and how big is your datastore?
[15:08] <ordex> let me have a look
[15:08] <toad_> also check your datastore max memory setting - if it's set to 0 that can cause this
[15:08] <toad_> brb
[15:09] <ordex> store is 10GB
[15:09] <ordex> max memory for datastore is 10MB
[15:10] <ordex> 20 sorry
[15:10] * sdiz (n=sdiz@) Quit ("leaving")
[15:10] <Ratchet> toad_: on file insert, shouldn't all blocks be stored in my store (not cache)?
[15:11] <ordex> my download queue is about 4/5GB
[15:12] <Ratchet> toad_: I inserted a ~100Mb file and the store only grew about 60Mb
[15:12] <FreenetLogBot> r22769 (1164) was built successfully on emu, mirrors are updating
[15:13] * Luke771 (n=luke@) has joined #freenet
[15:13] * ChanServ sets mode +v Luke771
[15:15] <toad_> Ratchet: no
[15:15] <toad_> Ratchet: it doesn't work like that
[15:15] <Ratchet> ah ok
[15:15] <toad_> ordex: ok i dunno then
[15:15] <toad_> ordex: did you send me your wrapper.log?
[15:19] <Luke771> toad_: I couldn't get it to work, Added job with crontab -e, saved with ctrl-o but it didnt work. And now I got to go, more details (if needed) later. Please get someone else to test the crash-shutdown if you can. I'll be back in a few hours
[15:20] * Luke771 is now known as luke771_afk
[15:21] <toad_> bbiab
[15:21] <toad_> luke771_afk: couldn't get *what* to work?
[15:21] <toad_> luke771_afk: oh i see
[15:22] <toad_> luke771_afk: yeah talk later about that, nag me
[15:22] <toad_> bbiab
[15:25] <ordex> toad_: sent
[15:34] <ordex> mh
[15:44] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror2.freenetproject.org/ : it has been disabled
[15:44] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror4.freenetproject.org/ : it has been disabled
[15:57] * kryptos23_ (n=kryptos@) Quit ("ChatZilla 0.9.81 [Firefox 2.0.0.16/2008070205]")
[16:00] * kalkin- (n=kalkin-@) has joined #freenet
[16:00] <THEapophis> Anyone has some experience with the FMS-NNTP Bridge? :) would like to have some answers...
[16:03] <toad_> THEapophis: i'm sure there have been scripts floating around
[16:03] <toad_> probably nothing fully functional and two way
[16:05] <THEapophis> well it works one way... But I am sure it doesnt work to post without having the identities on the server, which is not a good way for a public NNTP bridge,... but maybe someone has some experience ;)
[16:05] <THEapophis> And I dont understand the way how the Bridge adds new boards...
[16:05] <THEapophis> but probably all boards which are from trusted ppl... which is kind of... strange
[16:08] <ordex> toad_: did you receive my email?
[16:08] <ordex> i think the attchment could be a bit big
[16:09] <toad_> wrapper.log ? yes
[16:10] <ordex> yeah
[16:10] <ordex> ok
[16:11] <toad_> ordex: hmmm
[16:11] <toad_> ordex: it definitely looks like some memory related issue
[16:11] <ordex> mh
[16:11] <toad_> "GC overhead limit exceeded" means it's doing too much garbage collection
[16:11] <toad_> and not making enough progress
[16:12] <ordex> mh
[16:12] <toad_> how big is your queue, in gigabytes queued for upload/download?
[16:12] <toad_> and how big is your store?
[16:12] <ordex> 10GB
[16:12] <ordex> thaw also print some warning
[16:12] <ordex> like this: [WARNING] thaw.fcp.FCPMessage: Setting amount of data waiting to 0 ?! Abnormal !
[16:12] <toad_> 10GB store?
[16:13] <toad_> how much downloads?
[16:13] <ordex> yeah
[16:13] <ordex> now i reduced them
[16:13] <toad_> what was the total volume?
[16:13] <toad_> before it crashed?
[16:13] <ordex> about 5gb of download
[16:13] <toad_> hmmm
[16:13] <toad_> that should be fine
[16:13] <toad_> strange...
[16:13] <ordex> what do you think about clearing all downloads and readd them?
[16:14] <toad_> it'll take ages
[16:14] <toad_> to resume them
[16:14] <ordex> mh
[16:14] <ordex> but hte downloaded pieces should be in cache right?
[16:14] <toad_> yeah but it'll take ages to resume them
[16:14] <ordex> ah ok
[16:15] <toad_> hmmm
[16:15] <toad_> bizarre
[16:15] <ordex> [NOTICE ] thaw.plugins.index.Comment$CommentHandler: Signature validation failed !
[16:15] <ordex> could this messages from thaw help?
[16:15] <toad_> no
[16:15] <toad_> that's related to indexes
[16:15] <ordex> ok
[16:16] <toad_> probably db4o+salted hash would solve all your problems... or db4o on its own maybe...
[16:16] <ordex> mh ok
[16:16] <toad_> sadly db4o doesn't work right now
[16:16] <ordex> i'll wiat..
[16:16] <ordex> but what could i do for now?
[16:16] <toad_> of course that gets the overhead limit thing too
[16:16] <toad_> hmmm
[16:16] <toad_> you can disable the max GC overhead detection logic
[16:17] <toad_> then it will probably OOM outright... but it'd be useful to know...
[16:17] <ordex> mh
[16:17] <ordex> if i delete ll my downloads?
[16:17] <toad_> no
[16:17] <toad_> wait
[16:17] <ordex> k
[16:17] <toad_> edit your wrapper.conf
[16:17] <toad_> add
[16:17] <toad_> wrapper.java.additional.5=-XX:-UseGCOverheadLimit
[16:18] <toad_> change "5" to whatever is the next number after your existing wrapper.java.additional's
[16:18] <toad_> shutdown the node
[16:18] <toad_> and then start it back up
[16:18] <ordex> ok
[16:19] <ordex> doint it
[16:19] <ordex> doing it
[16:20] <ordex> node doesn't want to stop itself :D
[16:20] <ordex> restarted
[16:20] <toad_> did you actually shut down?
[16:20] <toad_> or just restart?
[16:20] <toad_> it needs to be shut down explicitly, otherwise it might not reload the config
[16:21] <ordex> yeah ./run stop ./run start
[16:22] <toad_> ok
[16:22] <ordex> my harddisk is working like never :P
[16:22] <toad_> :|
[16:22] <toad_> keep me up to date
[16:22] <ordex> k
[16:22] * HyperDimensions (n=HyperDim@) Quit (Read error: 110 (Connection timed out))
[16:22] <ordex> wrapper.log is stopped at the classic exception
[16:25] <ordex> will my hard disk stop to work?
[16:25] <toad_> what exception?
[16:25] <ordex> :P
[16:26] <ordex> Failed to load native FEC: java.lang.UnsatisfiedLinkError: com.onionnetworks.fec.Native8Code.nativeNewFEC(II)I
[16:26] <ordex> INFO | jvm 1 | 2008/09/23 18:22:01 | java.lang.UnsatisfiedLinkError: com.onionnetworks.fec.Native8Code.nativeNewFEC(II)I
[16:26] <toad_> ah
[16:26] <toad_> that's normal, ignore it
[16:26] <ordex> anyway fec is installed
[16:26] <ordex> where can i have a look of why is my hardisk working?
[16:27] <toad_> hmmm?
[16:27] <ordex> where can i look to know what is doing freenet now?
[16:27] <FreenetLogBot> The monitoring script has detected that http://mirror2.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[16:27] <ordex> to know why my harddisk is working so much
[16:27] <toad_> it's probably resuming the requests
[16:28] <ordex> mh, yeah , infact in thaw i see rate of 888KBps
[16:36] <FreenetLogBot> r22770 (1164) was built successfully on emu, mirrors are updating
[16:36] <FreenetLogBot> r22771 (1164) was built successfully on emu, mirrors are updating
[16:36] <FreenetLogBot> r22772 (1164) was built successfully on emu, mirrors are updating
[16:39] <FreenetLogBot> r22773 (1164) was built successfully on emu, mirrors are updating
[16:39] * popcorn99 (n=joelymoe@) has joined #freenet
[16:40] <popcorn99> hi all
[16:42] <toad_> hi
[16:42] <popcorn99> i'm back. I'm trying to install/connect to FMS but it crashes about 5 min after I solve puzzles to announce my identity. the MS error message gives: ModName: msver80.dll ModVer: 8.0.50727.762 Offset:00008a8c. Does this mean anything to anybody here?
[16:43] <toad_> eeek
[16:43] <p0s> popcorn99: i also have the problem.
[16:43] <p0s> popcorn99: i suppose someone found a bug in FMS and is exploiting it.
[16:43] <toad_> well, this is bad, but i can't fix it
[16:44] <toad_> well... i hope not, or we have big problems
[16:44] <p0s> toad_: this was quite expectable :)
[16:44] <toad_> segfaults do suggest something that might be exploitable
[16:44] * p0s installed a VM to try FMS, now he is glad that he did this
[16:44] <toad_> p0s: well we need the java FMS port asap
[16:45] <toad_> it won't have segfaults! :)
[16:45] <p0s> toad_: it could need one more developer i suppose. saces said that he's quite busy right now with other stuff.
[16:46] <toad_> I agree :)
[16:46] <popcorn99> I appreciate the incredibly quick response. Realistically, how soon would this be fixed so I can use FMS--your best guess?
[16:47] <p0s> popcorn99: the FMS developer is anonymous so you could post it on FMS ... and wait for the FMS freesite to be updated.
[16:47] <popcorn99> ummm I don't have an identity so do I have the ability still to read and write messages?
[16:48] <popcorn99> no id announced, i mean
[16:48] <p0s> popcorn99: you can read them but you cannot write :)
[16:48] <toad_> p0s: this is assuming the FMS dev isn't behind the buffer overflows ... which hopefully he isn't
[16:48] <p0s> popcorn99: BUT: the FMS guy has published his freemail adress i guess.
[16:48] <p0s> toad_: i also hope so.
[16:49] <toad_> hehe and the spam too!!! jhajaajajajajajajajhah
[16:49] <toad_> at least FMS runs on a dedicated user here
[16:50] <p0s> its funny that there is already ELIZA on fms, isnt it? :)
[16:50] <toad_> there is?
[16:50] <toad_> never bothered me
[16:50] <popcorn99> this is all new to me, I'll try to read the FMS messages, find the guy and set up freemail to write to him--all before it crashes on me. Is that what you are suggesting?
[16:50] <toad_> well there's no offline support for it
[16:50] <toad_> you could try a different version perhaps
[16:51] <popcorn99> ok, where would I find one?
[16:51] <toad_> back-issues of the freesite
[16:51] * Artefact2 (n=Romain@) has joined #freenet
[16:51] <popcorn99> ok i'll try that. thanks guys/gals!
[16:51] <toad_> USK@blah.../name/number/path can be rewritten to SSK@blah.../name-number/path
[16:51] <p0s> toad_: well eliza's posts wont arrive because it's got no trust. but its funny that the guy who ran it on frost really took the time for implementing it on fms..
[16:51] <toad_> and then you can reduce the number
[16:51] <ordex> my hard disk stopped trashing :P
[16:52] <toad_> ordex: yay
[16:53] <Artefact2> hi, i (again) have little questions on salted hash. my node alwars do datastore maintenance, I have a lot of false positives on my cache, and it returned to 0 keys used 2 days ago ...
[16:53] <toad_> ok, do we need round-robin-between-transfers in 1165? probably ...
[16:53] <toad_> Artefact2: how do you start your node? what OS?
[16:54] <popcorn99> sorry-ignorant me--how do i find a "back-issues" free site to get older FMS version? sorry to interrupt
[16:54] <toad_> [17:51] <toad_> USK@blah.../name/number/path can be rewritten to SSK@blah.../name-number/path
[16:54] <toad_> [17:51] <toad_> and then you can reduce the number
[16:54] <toad_> Artefact2: ubuntu?
[16:54] <Artefact2> Windows Vista 64 bits, 2 GB ram (512 mb given to my node, only 100 mb used), I start it as a system service, I have a 250 Gb datastore
[16:54] <toad_> hmmm
[16:54] <toad_> post your wrapper.log to a pastebin so we can see whether it cleanly shut down
[16:55] <toad_> or email me if you're worried it might have your IP in it
[16:55] <toad_> toad at amphibian.dyndns.org
[16:55] <Artefact2> I'm not so worried about that, i live in europe
[16:55] <toad_> hehe that's what they all say...
[16:55] <Artefact2> Freenet is not yet illegal in my country
[16:55] <toad_> ... right before they get rendered... :)
[16:56] <popcorn99> toad, was the USK... response for me--not sure i know freenet well enough to follow that..
[16:56] <toad_> popcorn99: i was talking about freesites
[16:56] <toad_> popcorn99: specifically the FMS freesite
[16:56] <toad_> popcorn99: you can access back-issues of the site, which will have links to older versions of FMS
[16:56] <toad_> Artefact2: well post it to a pastebin then
[16:57] <toad_> Artefact2: http://code.bulix.org/
[16:57] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror2.freenetproject.org/ : it has been disabled
[16:57] <FreenetLogBot> The monitoring script has detected that http://mirror4.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[16:57] <popcorn99> i'll look around on the FMS freesite some more..thanks!
[16:57] <Artefact2> toad_: be careful, lots of exceptions
[16:57] <Artefact2> toad_: http://code.bulix.org/hb6t2j-68444
[16:58] <Artefact2> damn, the paste was cutted
[16:58] <toad_> STATUS | wrapper | 2008/09/21 11:19:29 | JVM requested a restart.
[16:58] <toad_> ERROR | wrapper | 2008/09/21 11:21:34 | Shutdown failed: Timed out waiting for the JVM to terminate.
[16:58] <toad_> hmmm
[16:58] <toad_> well if that's typical then it explains the constant rebuilding the bloom filters :|
[16:59] <Artefact2> they do need to be constantly rebuild ? that's very slow :(
[16:59] <toad_> only if the node always crashes
[16:59] <Artefact2> here's the end : http://code.bulix.org/vwn8qf-68445
[16:59] <toad_> if it shuts down cleanly they don't need to be rebuilt
[17:00] <toad_> but windows doesn't give us enough time to shut down properly ... or we take too long to shut down ... depending on your point of view
[17:00] <toad_> how big's your datastore?
[17:00] <Artefact2> 250 GiB
[17:00] <toad_> hmmm
[17:00] <Artefact2> I do have outages, sometimes
[17:01] <toad_> hmmm
[17:01] <toad_> it doesn't actually say whether it managed to close the store ... hmmm
[17:01] <Artefact2> does it helps to increase the bloom filter's size ?
[17:01] <toad_> what's the bloom size now? -1?
[17:02] <Artefact2> yes (=> 128 MiB)
[17:02] * TheBishop_ (n=TheBisho@) has joined #freenet
[17:02] <toad_> well that's about right
[17:03] <toad_> the problem seems to be that the node never shuts down cleanly...
[17:03] <toad_> hmmm
[17:03] <FreenetLogBot> r22774 (1164) was built successfully on emu, mirrors are updating
[17:03] <toad_> can you do update.cmd testing?
[17:03] <Artefact2> Already done this morning
[17:03] <p0s> toad_: what about increasing wrapper.jvm_exit.timeout
[17:03] <p0s> toad_: that could help him
[17:04] <Artefact2> What value should i put ? Actual is 120
[17:05] <p0s> Artefact2: depends on your patience. i'd try 300, thats 5 minutes
[17:05] <toad_> for which?
[17:05] <toad_> wait
[17:05] <toad_> p0s: maybe
[17:05] <toad_> p0s: we need to verify the cause first
[17:05] <toad_> Artefact2: please could you run it again now
[17:05] <toad_> Artefact2: and verify that you've got 22774
[17:05] <Artefact2> I re-run update.cmd ?
[17:05] <toad_> yeah
[17:05] <Artefact2> Ok, i'm doing this now.
[17:06] <toad_> what is the typical cause for the node to be shutdown? a reboot?
[17:06] <Artefact2> Yes, but i try to avoid rebooting as much as possible
[17:06] <Artefact2> My node runs 24/7
[17:07] <toad_> well clearly it does get shut down quite frequently, or it'd complete the maintenance
[17:07] <toad_> STATUS | wrapper | 2008/09/21 11:19:29 | JVM requested a restart.
[17:07] <toad_> ERROR | wrapper | 2008/09/21 11:21:34 | Shutdown failed: Timed out waiting for the JVM to terminate.
[17:07] <toad_> two minutes and five seconds? aiiieeeee
[17:07] <toad_> Artefact2: in that particular instance was it because of a reboot or because you manually restarted it to update.cmd testing??
[17:08] <Artefact2> Yes, it was for the update.cmd
[17:08] <toad_> okay
[17:08] <toad_> and that was on sunday at 11:21
[17:09] <Artefact2> Oh no, sunday I deleted bloom filters, to see if it does improve anything
[17:09] <toad_> Artefact2: that might explain why it had to rebuild them? :)
[17:09] <Artefact2> Yes, but it rebuild it during the freenet start, and it goes much faster
[17:10] <toad_> I don't understand
[17:10] <toad_> it shows it rebuilding it just after it started up
[17:11] <toad_> did it finish the rebuild?
[17:11] <toad_> your log only goes up to 11:46 on sunday
[17:11] <Artefact2> It rebuilds it while Freenet starts : i can't browse freenet, i always get the page "Your freenet node is starting up, please hold on..."
[17:11] <toad_> that's not what the logfile shows
[17:13] <toad_> Artefact2: i don't understand
[17:13] <toad_> Artefact2: if it's slow now, then presumably it's rebuilding after startup, not during startup, right?
[17:13] <FreenetLogBot> The monitoring script has detected that http://mirror2.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[17:14] <toad_> Artefact2: hello??
[17:16] <toad_> Artefact2: when you get this message, i need to know whether it manages to shut down the datastores before the 2 minute timeout on shutdown
[17:17] <toad_> Artefact2: so update, make sure it's running at least r22774 (on the web interface page), then post your wrapper.log again - the recent part of it, not the beginning, we need it to include the shutdown, the timeout, and the restart
[17:17] <toad_> Artefact2: thanks
[17:20] <popcorn99> the applications site has a link to an older version (0.3.23) of FMS but when I try to download the Win bin file it can't seem to find it...after a long wait i get a data not found type message..anyone have another link?
[17:20] <popcorn99> ok, back
[17:21] * popcorn99 (n=joelymoe@) has left #freenet
[17:21] * popcorn99 (n=joelymoe@) has joined #freenet
[17:21] <toad_> popcorn99: try different editions of the FMS site
[17:22] <popcorn99> i don't know how to get them..sorry very new here
[17:22] <toad_> i told you twice already
[17:22] * zerwas (n=zerwas@) has joined #freenet
[17:22] <toad_> take the URL for the FMS site and change the USK into an SSK and <site name>/<number> into <site name>-<number>
[17:23] <toad_> and then change the number
[17:23] <popcorn99> now i think i understand. i'll try thanks
[17:23] <toad_> anyone here got a big saltedhashstore datastore?
[17:24] <toad_> sadly mine is only 2GB for some reason
[17:24] <toad_> probably because I accidentally used the wizard on the wrong node at some point
[17:25] * p0s (n=1mn57@) Quit ()
[17:25] <toad_> what I need to know is whether rebuilding the bloom filters *at full speed* has a big impact on performance
[17:25] <toad_> in SVN r22775 we rebuild at full speed
[17:25] <FreenetLogBot> r22775 (1164) was built successfully on emu, mirrors are updating
[17:26] <toad_> this will be vastly faster in terms of time to completion ... we previously delayed 100ms after every 128 blocks, because rebuilding caused a big performance hit
[17:26] <toad_> i don't think that's true since some changes went in relating to only writing to the secondary bloom filter in RAM ...
[17:27] <toad_> so I need a volunteer with a big saltedhashstore datastore... anyone?
[17:27] <toad_> all you need to do is update to r22775, and do an unclean shutdown
[17:28] <toad_> and restart the node, and see whether the bloom rebuild causes major performance problems
[17:28] <toad_> imho it shouldn't
[17:28] <toad_> since it's effectively just a linear read now
[17:29] <toad_> unclean shutdown = killall -9 java (for example)
[17:31] <FreenetLogBot> r22776 (FMSPlugin) was built successfully on emu, mirrors are updating
[17:34] <saces> toad_: i have added a txt file to FMSPlugin source tree, i hope its helpful.
[17:37] <toad_> ok
[17:37] <FreenetLogBot> r22777 (1164) was built successfully on emu, mirrors are updating
[17:37] * toad_ will look at it when he gets around to reviewing the commit
[17:38] <zerwas> how long does it take on your nodes until a message in FMS arrives?
[17:43] <toad_> how would we tell?
[17:54] <Tommy[D]> from minutes to hours (or days)? :)
[17:58] <popcorn99> toad, i am able to find prior versions of FMS, but just like i said happened with the 2.23 version, the 3.14 version also wasn't found once i clicked on the .zip file here is the message freenet gave: * Filename: fms-win32bin-0.3.14.zip.bin
[17:58] <popcorn99> * Size: unknown
[17:58] <popcorn99> * MIME type: unknown
[17:58] <popcorn99> Explanation
[17:58] <popcorn99> Freenet was unable to retrieve this file.
[17:58] <popcorn99> Not enough data found; some data was fet
[18:00] <popcorn99> going to lunch but will check back in 45 min. just trying to get an older version of FMS since the current one crashes..
[18:01] <zerwas> Tommy[D]: OK, thanks, i experience the same delays
[18:10] <Tommy[D]> toad_: Was the estimated network size somehow related to location swaps?
[18:16] <Artefact2> i'm back. my computer is terribly sdlow as soon as i started the nodes (after running update.cmd testing). I see everything (including my mouse) at 1 frame per minute and the disk thrashing is very, very intense
[18:16] <Artefact2> i can't lower the process priority in the task manager : it doesn't load
[18:16] <Ratchet> did you have any downloads running before restart which are now resuming?
[18:17] <toad_> Artefact2: that's bizarre
[18:17] <toad_> it's just a linear read ... i suppose it writes stuff too, at random, to the bloom filter ... but the whole bloom filter is only 128MB you said???
[18:18] <Ratchet> ah, you upgraded your store :-)
[18:18] <Artefact2> no, i doesn't have any downloads
[18:18] <Artefact2> *dint'
[18:19] <Artefact2> yes, it is 128 mb. but disk usage is very, very intensive ! I don't know what's happening right now and I can't kill the node
[18:20] <toad_> sorry 1sec
[18:21] <toad_> okay
[18:23] <toad_> I don't understand why it is so disk intensive - it's not like seeking across 128MB is going very far on the disk surface!
[18:25] <FreenetLogBot> r22778 (1164) was built successfully on emu, mirrors are updating
[18:25] <toad_> also it's frankly amazing that vista doesn't have I/O priorities, linux has had it for years and commercial unixes have had it for decades
[18:26] <toad_> you don't know what the CPU usage is because task manager won't load?
[18:26] <toad_> if I had a free choice i'd say it's entirely due to your OS and your computer :) but we do need it to run on Vista on systems with <4GB of RAM, because unfortunately they're very common
[18:27] <Artefact2> My disk thrashes for at least 1 hour niow...Should I restart manually my computer ?
[18:27] <toad_> plainly it IS a problem with Vista - NO application should be able to prevent Task Manager from loading
[18:28] <toad_> why can't you just tell the node to shut down?
[18:28] <Artefact2> I cant' launch any application; I can't even move my mouse...
[18:28] <toad_> how is that my fault?
[18:28] <toad_> seriously, an OS that is that easy to DoS isn't worth using
[18:29] <toad_> rebooting may not fix the problem
[18:29] <Artefact2> I didn't say it was your fault :)
[18:29] <toad_> your computer is fucked and it's a toss up between me and microsoft who's fault it is :|
[18:29] <toad_> sorry :|
[18:29] <Artefact2> No, it will work fine, I have to fix the problem
[18:30] <toad_> rebooting will not fix the problem because freenet is installed as a service
[18:30] <toad_> you have to terminate it somehow
[18:30] <toad_> if you can't move your mouse then that will be difficult
[18:32] <toad_> let me know how it goes :|
[18:33] * toad_ has reverted the commit which causes all the I/O btw
[18:33] <toad_> so if you can run update.cmd you can get rid of it
[18:37] * Artefact21 (n=Romain@) has joined #freenet
[18:38] <Artefact21> I restarted in safe mode, and disabled the freenet daemon
[18:38] <toad_> Artefact21: welcome back
[18:38] <toad_> Artefact21: yay
[18:38] <toad_> now update it to get rid of the dangerous jar file
[18:38] <Artefact21> I'm looking for the logs first
[18:38] <toad_> and then post the last part of your wrapper.log
[18:39] <Artefact21> Ohh, I was migrating to the new salted-hash appearently
[18:39] <toad_> wtf?
[18:39] <Artefact21> INFO | jvm 1 | 2008/09/23 19:11:59 | Migrating .header/.data -to-> .hd
[18:39] <toad_> ugh
[18:39] <toad_> i thought you said you'd already updated today?
[18:39] <Artefact21> It explains everything
[18:39] <toad_> that came in days ago
[18:39] <toad_> no it doesn't
[18:40] <toad_> that again is more or less linear read of two files and linear write to another one
[18:40] <toad_> hmmm well it's going backwards...
[18:40] <Artefact21> http://code.bulix.org/fp9jcz-68446
[18:40] <toad_> i suppose it will have some seeking
[18:41] <toad_> do you by any chance run vista on less than 4GB of RAM?
[18:41] <Artefact21> Maybe because I enabled NTFS compression ? SDiZ tell me on FMS to do this, because it enables sparse files
[18:41] <Artefact21> Yes, I only have 2 GB
[18:41] <Artefact21> of ram
[18:41] <toad_> ok well ... i'm not gonna say what i was going to say, and have already said several times, any more times :)
[18:41] <toad_> it's obvious and i'll leave it at that :)
[18:42] <toad_> hmmm so when you killed it it was 2/3rds of the way through migrating the chk store
[18:42] <toad_> but the CHK-cache is just as big
[18:42] <toad_> and then there are four smaller things
[18:42] <Artefact21> But, why was the computer completely frozen ?
[18:43] <toad_> because vista is a pile of shit on less than 4GB of RAM
[18:43] <toad_> that's the only explanation I can immediately provide
[18:43] <Artefact21> Ok... So what can I do now to solve this ? I wipe my all store and start the new salted hash from nothing ?
[18:43] <toad_> it's reading from two files, granted it's reading from the back end, and it's writing to a new file, also from the back end ...
[18:43] <toad_> hmmm
[18:44] <toad_> so it's going to have to allocate a huge amount of disk space when it first starts
[18:44] <toad_> cos windows preallocates on seek, instead of creating a hole (imho this is one of a small number of *good* things about windows)
[18:44] <Artefact21> Huh, damn
[18:44] <toad_> but after that, it's just moving data from two files to one file, it ought not to be catastrophic
[18:45] <toad_> anyway yeah it does look that way ... or let it run like it was for maybe 5 or 6 hours
[18:45] <toad_> and it should sort itself out
[18:46] <toad_> you're not out of disk space, correct?
[18:46] <Artefact21> No, I'm not. Freenet is almost alone on a 400 GB drive :)
[18:46] <toad_> it took an hour to reach 2/3rds of the CHK store ... it should take 3 hours to complete the CHK store and CHK cache, and maybe another hour to do the rest
[18:46] <toad_> Artefact21: and you allowed plenty of slack
[18:47] <toad_> I mean you didn't set the limit to 400GB
[18:47] <toad_> iirc you said 250
[18:47] <toad_> so it should work ... in theory
[18:47] <toad_> how much of that is used apart from the datastore?
[18:47] <toad_> it's going to need temp space of around 125GB
[18:48] <Artefact21> No, i use sparse files. My "big" 125 GB file doesn't fill 125 gb of the disk
[18:48] <toad_> okay
[18:48] <toad_> in that case it will definitely complete
[18:48] <toad_> unfortunately your PC may not be usable during that period
[18:48] <toad_> but if you can leave it for 5 hours it should be fine
[18:48] <Artefact21> Yes, that night
[18:49] <toad_> so leave it overnight
[18:49] <toad_> in the meantime don't use it
[18:49] <Artefact21> Yes, i will be sleeping :)
[18:49] <toad_> :)
[18:49] <toad_> come back tomorrow and we can look at the other issue again
[18:49] <toad_> and btw this is doubly my fault
[18:49] <toad_> I actually requested that the format be changed in the first place
[18:50] <toad_> which is what the data+headers -> .hd migration is doing
[18:50] <toad_> so 2/3rds my fault 1/3rd microsoft's :)
[18:50] <toad_> thanks for the testing
[18:50] <Artefact21> It doesn't annoy me, I run freenet a bit for the technical exploit too ;)
[18:50] <toad_> :)
[18:50] <toad_> must ... overload ... system ...
[18:51] * toad_ recommends near-line capable drives and 24x7 operation :)
[18:51] <Artefact21> http://img48.imageshack.us/img48/205/snag0002hi1.png here is what happens with NTFS compression on big .data files
[18:51] * Artefact2 (n=Romain@) Quit (Read error: 113 (No route to host))
[18:51] <toad_> i have heard some issues about compression, i think there may be a problem with big files
[18:51] <Artefact21> Theorical size -> 119 gb, size on disk : 1.83 gb
[18:51] <toad_> if you run into it, you will need to nuke the store and start again
[18:51] <toad_> lets see while i fish out the url...
[18:52] <Artefact21> We will see that tomorrow, bye :)
[18:52] <toad_> ok, turn it on before you go
[18:52] * Artefact21 (n=Romain@) Quit ("Leaving.")
[18:52] <popcorn99> good luck Art. I gotta run and will come back later to find out where to get an older FMS that actually will load.
[18:52] * popcorn99 (n=joelymoe@) has left #freenet
[18:54] * FrinkC (n=FrinkC@) has joined #freenet
[18:54] <toad_> http://support.microsoft.com/kb/927912/
[18:55] <toad_> yay, that particular bug doesn't happen on Vista, so hopefully Artefact's store-mash will finish
[18:55] <toad_> I bet the non-responsive-mouse-etc was caused by the data compression too
[19:00] <FreenetLogBot> r22779 (1164) was built successfully on emu, mirrors are updating
[19:01] <toad_> if anyone here has a large salted hash store datastore, please do an unclean shutdown and try trunk
[19:02] <toad_> let me know whether it results in catastrophic performance problems :)
[19:09] <FreenetLogBot> r22780 (1164) was built successfully on emu, mirrors are updating
[19:12] <FreenetLogBot> r22781 (1164) was built successfully on emu, mirrors are updating
[19:12] <FreenetLogBot> r22782 (1164) was built successfully on emu, mirrors are updating
[19:15] <FreenetLogBot> r22783 (1164) was built successfully on emu, mirrors are updating
[19:40] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror1.freenetproject.org/ : it has been disabled
[19:43] * NEOatNHNG (n=neo@) has joined #freenet
[19:59] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror4.freenetproject.org/ : it has been disabled
[20:00] * sanity_ (n=ian@) Quit ()
[20:10] * phrosty (n=phrosty@) has joined #freenet
[20:16] * mikusr (n=mikusrak@) has joined #freenet
[20:23] * caytchen (n=caytchen@) Quit ("Verlassend")
[20:25] <FreenetLogBot> The monitoring script has detected that http://mirror1.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[20:29] * ordex is now known as ordex_
[20:30] * ordex_ is now known as ordex
[20:33] <THEapophis> So whats HOT on Freenet these days? what is the thing every freenetter needs to know about? :)
[20:40] <FreenetLogBot> The monitoring script has detected a network glitch on http://mirror1.freenetproject.org/ : it has been disabled
[20:41] <ordex> toad_: # nodeUptime: 3m9s it reboots again!
[20:43] <FreenetLogBot> The monitoring script has detected that http://mirror4.freenetproject.org/ is eligible to be back on the main mirror rotation list. Welcome Back!
[20:44] <ordex> damn
[20:44] <toad_> hi
[20:44] <ordex> jvm hang up
[20:44] <ordex> hi
[20:45] <ordex> look at this
[20:45] <ordex> no, too long
[20:45] <ordex> pastebin
[20:45] <ordex> http://pasteb
