From bo-le at web.de  Tue Jan  2 14:54:49 2007
From: bo-le at web.de (bo-le at web.de)
Date: Tue, 2 Jan 2007 15:54:49 +0100
Subject: [Tech] EvilDude, Round 2
Message-ID: <200701021554.50318.bo-le@web.de>

hi, 

just in the frost-board 'frost'
----- EvilDude ----- 2007.01.02 - 07:22:29GMT -----

If I wanted to download a key but it was rare, could I increase my chances of 
getting it if I inserted a KSK redirecting to it, forcing every frost user to 
download that file?

Similarly, could I create a denial of service attack by inserting a bunch of 
redirects to all the large files I can find? I'm not going to attempt it this 
time around, I'm just wondering.

-----

The fcp client should be able to detect the redirects if wanted.

My suggestion: a new progress, that show the redirects.

   static final int showRedirects = 128;

   verbose=verbose | showRedirects


Now the ClientGet returns a new progress message 

   FollowingRedirect 
   Identifier=hello
   RequestedURI=freenet:KSK at something-1-2
   RedirectTargetURI=freenet:CHK at jdncdj,hdcbd,a8
   EndMessage

Now the client app can decide how to handle it.

thanks.

-- 
Mfg
saces


From bbackde at googlemail.com  Wed Jan  3 14:19:59 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Wed, 3 Jan 2007 15:19:59 +0100
Subject: [Tech] EvilDude, Round 2
In-Reply-To: <200701021554.50318.bo-le@web.de>
References: <200701021554.50318.bo-le@web.de>
Message-ID: <d6149a790701030619i66d19abaxf4f57e6db63be8c1@mail.gmail.com>

I think he means that a client should be able to determine if a
KSK/SSK was inserted as a manual redirect to another key, or if the
KSK/SSK was inserted with data and a redirect to the content was
automatically created by the node.

Is it possible to separate this in the node? This would be a great and
simple solution.

On 1/2/07, bo-le at web.de <bo-le at web.de> wrote:
> hi,
>
> just in the frost-board 'frost'
> ----- EvilDude ----- 2007.01.02 - 07:22:29GMT -----
>
> If I wanted to download a key but it was rare, could I increase my chances of
> getting it if I inserted a KSK redirecting to it, forcing every frost user to
> download that file?
>
> Similarly, could I create a denial of service attack by inserting a bunch of
> redirects to all the large files I can find? I'm not going to attempt it this
> time around, I'm just wondering.
>
> -----
>
> The fcp client should be able to detect the redirects if wanted.
>
> My suggestion: a new progress, that show the redirects.
>
>    static final int showRedirects = 128;
>
>    verbose=verbose | showRedirects
>
>
> Now the ClientGet returns a new progress message
>
>    FollowingRedirect
>    Identifier=hello
>    RequestedURI=freenet:KSK at something-1-2
>    RedirectTargetURI=freenet:CHK at jdncdj,hdcbd,a8
>    EndMessage
>
> Now the client app can decide how to handle it.
>
> thanks.
>
> --
> Mfg
> saces
>


From dbkr at freenetproject.org  Wed Jan  3 16:30:52 2007
From: dbkr at freenetproject.org (Dave Baker)
Date: Wed, 3 Jan 2007 16:30:52 +0000
Subject: [Tech] EvilDude, Round 2
In-Reply-To: <d6149a790701030619i66d19abaxf4f57e6db63be8c1@mail.gmail.com>
References: <200701021554.50318.bo-le@web.de>
	<d6149a790701030619i66d19abaxf4f57e6db63be8c1@mail.gmail.com>
Message-ID: <200701031630.52885.dbkr@freenetproject.org>

I don't really see as that's relevant, since it would be spoofable anyway. 
Surely the point is setting a limit on the size of a key when it's requested 
(frosy must do this, surely..?) and limiting the number of redirects (which 
the ClientGet command doesn't seem to have an option for, but I assume the 
node has a sensible limit and detects redirect loops).

This would prevent the second attack. I don't really see that the first is a 
problem anyway, since the risk of downloading arbitrary content is implicit 
when using frost. You could potentially only allow keys that are non-redirect 
KSKs (thus limiting the size of a frost post to 1k, which may or may not be 
desirable). That would prevent it, but as I say, I don't really see it as a 
problem assuming that there's a filesize limit.


Dave


On Wednesday 03 January 2007 14:19, bbackde at googlemail.com wrote:
> I think he means that a client should be able to determine if a
> KSK/SSK was inserted as a manual redirect to another key, or if the
> KSK/SSK was inserted with data and a redirect to the content was
> automatically created by the node.
>
> Is it possible to separate this in the node? This would be a great and
> simple solution.
>
> On 1/2/07, bo-le at web.de <bo-le at web.de> wrote:
> > hi,
> >
> > just in the frost-board 'frost'
> > ----- EvilDude ----- 2007.01.02 - 07:22:29GMT -----
> >
> > If I wanted to download a key but it was rare, could I increase my
> > chances of getting it if I inserted a KSK redirecting to it, forcing
> > every frost user to download that file?
> >
> > Similarly, could I create a denial of service attack by inserting a bunch
> > of redirects to all the large files I can find? I'm not going to attempt
> > it this time around, I'm just wondering.
> >
> > -----
> >
> > The fcp client should be able to detect the redirects if wanted.
> >
> > My suggestion: a new progress, that show the redirects.
> >
> >    static final int showRedirects = 128;
> >
> >    verbose=verbose | showRedirects
> >
> >
> > Now the ClientGet returns a new progress message
> >
> >    FollowingRedirect
> >    Identifier=hello
> >    RequestedURI=freenet:KSK at something-1-2
> >    RedirectTargetURI=freenet:CHK at jdncdj,hdcbd,a8
> >    EndMessage
> >
> > Now the client app can decide how to handle it.
> >
> > thanks.
> >
> > --
> > Mfg
> > saces
>
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech


From bbackde at googlemail.com  Wed Jan  3 17:05:51 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Wed, 3 Jan 2007 18:05:51 +0100
Subject: [Tech] EvilDude, Round 2
In-Reply-To: <200701031630.52885.dbkr@freenetproject.org>
References: <200701021554.50318.bo-le@web.de>
	<d6149a790701030619i66d19abaxf4f57e6db63be8c1@mail.gmail.com>
	<200701031630.52885.dbkr@freenetproject.org>
Message-ID: <d6149a790701030905g3ff470c0je195b987dc0de3bf@mail.gmail.com>

It would help to prevent the usage of specific keys that were inserted
as a redirect target of a KSK. If a client could determine that the
inserted KSK was inserted as redirect to a specific key, we could
easily drop and ignore this KSK. We would only allow KSKs that were
inserted as data. This prevents attackers from inserting alot of KSK
redirects (which is very fast, compared to the insert of a KSK with
the data) in a short time, and it prevents spreading of specific CHK
keys over KSKs.
So I see some advantages there. It should'nt not too hard IF the node
knows when a KSK was inserted as redirect to a specific CHK. If not I
think we will have some more problems with KSK attacks than under 0.5.
On 0.5 this kind of attack is not possible such easily...

On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> I don't really see as that's relevant, since it would be spoofable anyway.
> Surely the point is setting a limit on the size of a key when it's requested
> (frosy must do this, surely..?) and limiting the number of redirects (which
> the ClientGet command doesn't seem to have an option for, but I assume the
> node has a sensible limit and detects redirect loops).
>
> This would prevent the second attack. I don't really see that the first is a
> problem anyway, since the risk of downloading arbitrary content is implicit
> when using frost. You could potentially only allow keys that are non-redirect
> KSKs (thus limiting the size of a frost post to 1k, which may or may not be
> desirable). That would prevent it, but as I say, I don't really see it as a
> problem assuming that there's a filesize limit.
>
>
> Dave
>
>
> On Wednesday 03 January 2007 14:19, bbackde at googlemail.com wrote:
> > I think he means that a client should be able to determine if a
> > KSK/SSK was inserted as a manual redirect to another key, or if the
> > KSK/SSK was inserted with data and a redirect to the content was
> > automatically created by the node.
> >
> > Is it possible to separate this in the node? This would be a great and
> > simple solution.
> >
> > On 1/2/07, bo-le at web.de <bo-le at web.de> wrote:
> > > hi,
> > >
> > > just in the frost-board 'frost'
> > > ----- EvilDude ----- 2007.01.02 - 07:22:29GMT -----
> > >
> > > If I wanted to download a key but it was rare, could I increase my
> > > chances of getting it if I inserted a KSK redirecting to it, forcing
> > > every frost user to download that file?
> > >
> > > Similarly, could I create a denial of service attack by inserting a bunch
> > > of redirects to all the large files I can find? I'm not going to attempt
> > > it this time around, I'm just wondering.
> > >
> > > -----
> > >
> > > The fcp client should be able to detect the redirects if wanted.
> > >
> > > My suggestion: a new progress, that show the redirects.
> > >
> > >    static final int showRedirects = 128;
> > >
> > >    verbose=verbose | showRedirects
> > >
> > >
> > > Now the ClientGet returns a new progress message
> > >
> > >    FollowingRedirect
> > >    Identifier=hello
> > >    RequestedURI=freenet:KSK at something-1-2
> > >    RedirectTargetURI=freenet:CHK at jdncdj,hdcbd,a8
> > >    EndMessage
> > >
> > > Now the client app can decide how to handle it.
> > >
> > > thanks.
> > >
> > > --
> > > Mfg
> > > saces
> >
> > _______________________________________________
> > Tech mailing list
> > Tech at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
>


From dbkr at freenetproject.org  Wed Jan  3 17:32:31 2007
From: dbkr at freenetproject.org (Dave Baker)
Date: Wed, 3 Jan 2007 17:32:31 +0000
Subject: [Tech] EvilDude, Round 2
In-Reply-To: <d6149a790701030905g3ff470c0je195b987dc0de3bf@mail.gmail.com>
References: <200701021554.50318.bo-le@web.de>
	<200701031630.52885.dbkr@freenetproject.org>
	<d6149a790701030905g3ff470c0je195b987dc0de3bf@mail.gmail.com>
Message-ID: <200701031732.31334.dbkr@freenetproject.org>

> If a client could determine that the inserted KSK was inserted
> as redirect to a specific key, 

That's the problem though - I don't think that's possible. Even if it were 
marked somehow on insertion, it would only require some minor hackery of the 
node's source to make a user-created redirect look like a node-created 
redirect.

Again though, I'm still not convinced it's worth worrying about.


Dave

On Wednesday 03 January 2007 17:05, bbackde at googlemail.com wrote:
> It would help to prevent the usage of specific keys that were inserted
> as a redirect target of a KSK. If a client could determine that the
> inserted KSK was inserted as redirect to a specific key, we could
> easily drop and ignore this KSK. We would only allow KSKs that were
> inserted as data. This prevents attackers from inserting alot of KSK
> redirects (which is very fast, compared to the insert of a KSK with
> the data) in a short time, and it prevents spreading of specific CHK
> keys over KSKs.
> So I see some advantages there. It should'nt not too hard IF the node
> knows when a KSK was inserted as redirect to a specific CHK. If not I
> think we will have some more problems with KSK attacks than under 0.5.
> On 0.5 this kind of attack is not possible such easily...
>
> On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > I don't really see as that's relevant, since it would be spoofable
> > anyway. Surely the point is setting a limit on the size of a key when
> > it's requested (frosy must do this, surely..?) and limiting the number of
> > redirects (which the ClientGet command doesn't seem to have an option
> > for, but I assume the node has a sensible limit and detects redirect
> > loops).
> >
> > This would prevent the second attack. I don't really see that the first
> > is a problem anyway, since the risk of downloading arbitrary content is
> > implicit when using frost. You could potentially only allow keys that are
> > non-redirect KSKs (thus limiting the size of a frost post to 1k, which
> > may or may not be desirable). That would prevent it, but as I say, I
> > don't really see it as a problem assuming that there's a filesize limit.
> >
> >
> > Dave
> >
> > On Wednesday 03 January 2007 14:19, bbackde at googlemail.com wrote:
> > > I think he means that a client should be able to determine if a
> > > KSK/SSK was inserted as a manual redirect to another key, or if the
> > > KSK/SSK was inserted with data and a redirect to the content was
> > > automatically created by the node.
> > >
> > > Is it possible to separate this in the node? This would be a great and
> > > simple solution.
> > >
> > > On 1/2/07, bo-le at web.de <bo-le at web.de> wrote:
> > > > hi,
> > > >
> > > > just in the frost-board 'frost'
> > > > ----- EvilDude ----- 2007.01.02 - 07:22:29GMT -----
> > > >
> > > > If I wanted to download a key but it was rare, could I increase my
> > > > chances of getting it if I inserted a KSK redirecting to it, forcing
> > > > every frost user to download that file?
> > > >
> > > > Similarly, could I create a denial of service attack by inserting a
> > > > bunch of redirects to all the large files I can find? I'm not going
> > > > to attempt it this time around, I'm just wondering.
> > > >
> > > > -----
> > > >
> > > > The fcp client should be able to detect the redirects if wanted.
> > > >
> > > > My suggestion: a new progress, that show the redirects.
> > > >
> > > >    static final int showRedirects = 128;
> > > >
> > > >    verbose=verbose | showRedirects
> > > >
> > > >
> > > > Now the ClientGet returns a new progress message
> > > >
> > > >    FollowingRedirect
> > > >    Identifier=hello
> > > >    RequestedURI=freenet:KSK at something-1-2
> > > >    RedirectTargetURI=freenet:CHK at jdncdj,hdcbd,a8
> > > >    EndMessage
> > > >
> > > > Now the client app can decide how to handle it.
> > > >
> > > > thanks.
> > > >
> > > > --
> > > > Mfg
> > > > saces
> > >
> > > _______________________________________________
> > > Tech mailing list
> > > Tech at freenetproject.org
> > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> >
> > _______________________________________________
> > Tech mailing list
> > Tech at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
>
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech


From bbackde at googlemail.com  Wed Jan  3 19:42:18 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Wed, 3 Jan 2007 20:42:18 +0100
Subject: [Tech] EvilDude, Round 2
In-Reply-To: <200701031732.31334.dbkr@freenetproject.org>
References: <200701021554.50318.bo-le@web.de>
	<200701031630.52885.dbkr@freenetproject.org>
	<d6149a790701030905g3ff470c0je195b987dc0de3bf@mail.gmail.com>
	<200701031732.31334.dbkr@freenetproject.org>
Message-ID: <d6149a790701031142r4d747171p6599b31c50f1c9a2@mail.gmail.com>

What is wrong with the described advantages?

Scenario:

board_1 and board_2 exists. An attacker tries to spam them to make the
retrieval of messages nearly impossible.

On board_1 he inserts 100 KSK keys with redirects to CHK keys that
were randomly created and point to no existing content. The insert
completes _very_ quickly.

On board_2 he inserts 100 KSKs with random data behind it. The insert
takes a _much_ longer time to complete.

Now a Frost client updates this board.

On board_1 the client requests the 1st KSK. The node redirects
transparently to the CHK in the KSK redirect, and needs a _long_ time
until he gives up and reports code=28 to the client. The client does
not know if the request failed because the CHK does'nt exist at all,
or if the CHK could not be retrieved this time, but later. The client
steps over all 100 KSK keys (each one failing after a long search
time) and after this the client have to request all keys again during
next update, because some CHK might be retrievable now.

On board_2 the client also requests the 1st KSK key. The node
redirects to the CHK key, and after a _shorter_ time the request is
successful or not. If it _is_ successful the client marks the slot
used (tried never again) and inspects the retrieved data, which are
random and therefore invalid. If the request was _not_ successful, the
client gets code=28, but this time he knows that the redirect points
to valid CHK key! So a retry will be successful later, and its worth
to retry.

So why does the node even allow to insert a redirect to not-existing
data? He should check if the data is retrievable from the inserting
node. I can't understand why someone should insert a redirect to
content that cannot be retrieved by himself.

I think an indication if the CHK is valid would be better than
without. Even with patched nodes, each insert node on the htl chain
could check if the CHK is known, and it is known if at least e.g. 2
nodes (on the chain or neighbors) say it is valid.

I don't know if my points are valid, but I want some explanations that
help me to understand this. A simple 'its not worth to think about',
without to provide a working circumvention for this, its not enough
for me.

And why is there no type of key in 0.7 that is designed for
applications like Frost that provide a global messaging feature? There
are many improvements for freesites (e.g. USK) and easy filetransfer
(although to less detailed for clients, currently).

On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > If a client could determine that the inserted KSK was inserted
> > as redirect to a specific key,
>
> That's the problem though - I don't think that's possible. Even if it were
> marked somehow on insertion, it would only require some minor hackery of the
> node's source to make a user-created redirect look like a node-created
> redirect.
>
> Again though, I'm still not convinced it's worth worrying about.
>
>
> Dave
>
> On Wednesday 03 January 2007 17:05, bbackde at googlemail.com wrote:
> > It would help to prevent the usage of specific keys that were inserted
> > as a redirect target of a KSK. If a client could determine that the
> > inserted KSK was inserted as redirect to a specific key, we could
> > easily drop and ignore this KSK. We would only allow KSKs that were
> > inserted as data. This prevents attackers from inserting alot of KSK
> > redirects (which is very fast, compared to the insert of a KSK with
> > the data) in a short time, and it prevents spreading of specific CHK
> > keys over KSKs.
> > So I see some advantages there. It should'nt not too hard IF the node
> > knows when a KSK was inserted as redirect to a specific CHK. If not I
> > think we will have some more problems with KSK attacks than under 0.5.
> > On 0.5 this kind of attack is not possible such easily...
> >
> > On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > > I don't really see as that's relevant, since it would be spoofable
> > > anyway. Surely the point is setting a limit on the size of a key when
> > > it's requested (frosy must do this, surely..?) and limiting the number of
> > > redirects (which the ClientGet command doesn't seem to have an option
> > > for, but I assume the node has a sensible limit and detects redirect
> > > loops).
> > >
> > > This would prevent the second attack. I don't really see that the first
> > > is a problem anyway, since the risk of downloading arbitrary content is
> > > implicit when using frost. You could potentially only allow keys that are
> > > non-redirect KSKs (thus limiting the size of a frost post to 1k, which
> > > may or may not be desirable). That would prevent it, but as I say, I
> > > don't really see it as a problem assuming that there's a filesize limit.
> > >
> > >
> > > Dave
> > >
> > > On Wednesday 03 January 2007 14:19, bbackde at googlemail.com wrote:
> > > > I think he means that a client should be able to determine if a
> > > > KSK/SSK was inserted as a manual redirect to another key, or if the
> > > > KSK/SSK was inserted with data and a redirect to the content was
> > > > automatically created by the node.
> > > >
> > > > Is it possible to separate this in the node? This would be a great and
> > > > simple solution.
> > > >
> > > > On 1/2/07, bo-le at web.de <bo-le at web.de> wrote:
> > > > > hi,
> > > > >
> > > > > just in the frost-board 'frost'
> > > > > ----- EvilDude ----- 2007.01.02 - 07:22:29GMT -----
> > > > >
> > > > > If I wanted to download a key but it was rare, could I increase my
> > > > > chances of getting it if I inserted a KSK redirecting to it, forcing
> > > > > every frost user to download that file?
> > > > >
> > > > > Similarly, could I create a denial of service attack by inserting a
> > > > > bunch of redirects to all the large files I can find? I'm not going
> > > > > to attempt it this time around, I'm just wondering.
> > > > >
> > > > > -----
> > > > >
> > > > > The fcp client should be able to detect the redirects if wanted.
> > > > >
> > > > > My suggestion: a new progress, that show the redirects.
> > > > >
> > > > >    static final int showRedirects = 128;
> > > > >
> > > > >    verbose=verbose | showRedirects
> > > > >
> > > > >
> > > > > Now the ClientGet returns a new progress message
> > > > >
> > > > >    FollowingRedirect
> > > > >    Identifier=hello
> > > > >    RequestedURI=freenet:KSK at something-1-2
> > > > >    RedirectTargetURI=freenet:CHK at jdncdj,hdcbd,a8
> > > > >    EndMessage
> > > > >
> > > > > Now the client app can decide how to handle it.
> > > > >
> > > > > thanks.
> > > > >
> > > > > --
> > > > > Mfg
> > > > > saces
> > > >
> > > > _______________________________________________
> > > > Tech mailing list
> > > > Tech at freenetproject.org
> > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > >
> > > _______________________________________________
> > > Tech mailing list
> > > Tech at freenetproject.org
> > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> >
> > _______________________________________________
> > Tech mailing list
> > Tech at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
>


From dbkr at freenetproject.org  Wed Jan  3 21:12:14 2007
From: dbkr at freenetproject.org (Dave Baker)
Date: Wed, 3 Jan 2007 21:12:14 +0000
Subject: [Tech] EvilDude, Round 2
In-Reply-To: <d6149a790701031142r4d747171p6599b31c50f1c9a2@mail.gmail.com>
References: <200701021554.50318.bo-le@web.de>
	<200701031732.31334.dbkr@freenetproject.org>
	<d6149a790701031142r4d747171p6599b31c50f1c9a2@mail.gmail.com>
Message-ID: <200701032112.14882.dbkr@freenetproject.org>

On Wednesday 03 January 2007 19:42, bbackde at googlemail.com wrote:
> What is wrong with the described advantages?
> 
> Scenario:
> 
> board_1 and board_2 exists. An attacker tries to spam them to make the
> retrieval of messages nearly impossible.
> 
> On board_1 he inserts 100 KSK keys with redirects to CHK keys that
> were randomly created and point to no existing content. The insert
> completes _very_ quickly.
> 
> On board_2 he inserts 100 KSKs with random data behind it. The insert
> takes a _much_ longer time to complete.
> 
> Now a Frost client updates this board.
> 
> On board_1 the client requests the 1st KSK. The node redirects
> transparently to the CHK in the KSK redirect, and needs a _long_ time
> until he gives up and reports code=28 to the client. The client does
> not know if the request failed because the CHK does'nt exist at all,
> or if the CHK could not be retrieved this time, but later. The client
> steps over all 100 KSK keys (each one failing after a long search
> time) and after this the client have to request all keys again during
> next update, because some CHK might be retrievable now.
> 
> On board_2 the client also requests the 1st KSK key. The node
> redirects to the CHK key, and after a _shorter_ time the request is
> successful or not. If it _is_ successful the client marks the slot
> used (tried never again) and inspects the retrieved data, which are
> random and therefore invalid. If the request was _not_ successful, the
> client gets code=28, but this time he knows that the redirect points
> to valid CHK key! So a retry will be successful later, and its worth
> to retry.
> 
> So why does the node even allow to insert a redirect to not-existing
> data? He should check if the data is retrievable from the inserting 
> node. I can't understand why someone should insert a redirect to
> content that cannot be retrieved by himself.
> 
> I think an indication if the CHK is valid would be better than
> without. Even with patched nodes, each insert node on the htl chain
> could check if the CHK is known, and it is known if at least e.g. 2
> nodes (on the chain or neighbors) say it is valid.

This would surely make the insertion of redirects quite considerably slower? 
It also means that the network could refuse to insert a redirect just because 
some node on the chain can't retrieve the data, for whatever reason. These 
would be my reasons for allowing insertion of arbitrary redirects, but I 
didn't write it!

Come to think of it, surely only the data and the routing key is sent on an 
insert, so the destination nodes can't check the redirect because they can't 
decrypt the data because they don't have the public key. In fact, they can't 
even tell it's a redirect.

> 
> I don't know if my points are valid, but I want some explanations that
> help me to understand this. A simple 'its not worth to think about',
> without to provide a working circumvention for this, its not enough
> for me.
> 
> And why is there no type of key in 0.7 that is designed for
> applications like Frost that provide a global messaging feature? There
> are many improvements for freesites (e.g. USK) and easy filetransfer
> (although to less detailed for clients, currently).

I'm not really sure what you mean by a 'global messaging feature'.



Dave


> 
> On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > > If a client could determine that the inserted KSK was inserted
> > > as redirect to a specific key,
> >
> > That's the problem though - I don't think that's possible. Even if it were
> > marked somehow on insertion, it would only require some minor hackery of 
the
> > node's source to make a user-created redirect look like a node-created
> > redirect.
> >
> > Again though, I'm still not convinced it's worth worrying about.
> >
> >
> > Dave
> >
> > On Wednesday 03 January 2007 17:05, bbackde at googlemail.com wrote:
> > > It would help to prevent the usage of specific keys that were inserted
> > > as a redirect target of a KSK. If a client could determine that the
> > > inserted KSK was inserted as redirect to a specific key, we could
> > > easily drop and ignore this KSK. We would only allow KSKs that were
> > > inserted as data. This prevents attackers from inserting alot of KSK
> > > redirects (which is very fast, compared to the insert of a KSK with
> > > the data) in a short time, and it prevents spreading of specific CHK
> > > keys over KSKs.
> > > So I see some advantages there. It should'nt not too hard IF the node
> > > knows when a KSK was inserted as redirect to a specific CHK. If not I
> > > think we will have some more problems with KSK attacks than under 0.5.
> > > On 0.5 this kind of attack is not possible such easily...
> > >
> > > On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > > > I don't really see as that's relevant, since it would be spoofable
> > > > anyway. Surely the point is setting a limit on the size of a key when
> > > > it's requested (frosy must do this, surely..?) and limiting the number 
of
> > > > redirects (which the ClientGet command doesn't seem to have an option
> > > > for, but I assume the node has a sensible limit and detects redirect
> > > > loops).
> > > >
> > > > This would prevent the second attack. I don't really see that the 
first
> > > > is a problem anyway, since the risk of downloading arbitrary content 
is
> > > > implicit when using frost. You could potentially only allow keys that 
are
> > > > non-redirect KSKs (thus limiting the size of a frost post to 1k, which
> > > > may or may not be desirable). That would prevent it, but as I say, I
> > > > don't really see it as a problem assuming that there's a filesize 
limit.
> > > >
> > > >
> > > > Dave
> > > >
> > > > On Wednesday 03 January 2007 14:19, bbackde at googlemail.com wrote:
> > > > > I think he means that a client should be able to determine if a
> > > > > KSK/SSK was inserted as a manual redirect to another key, or if the
> > > > > KSK/SSK was inserted with data and a redirect to the content was
> > > > > automatically created by the node.
> > > > >
> > > > > Is it possible to separate this in the node? This would be a great 
and
> > > > > simple solution.
> > > > >
> > > > > On 1/2/07, bo-le at web.de <bo-le at web.de> wrote:
> > > > > > hi,
> > > > > >
> > > > > > just in the frost-board 'frost'
> > > > > > ----- EvilDude ----- 2007.01.02 - 07:22:29GMT -----
> > > > > >
> > > > > > If I wanted to download a key but it was rare, could I increase my
> > > > > > chances of getting it if I inserted a KSK redirecting to it, 
forcing
> > > > > > every frost user to download that file?
> > > > > >
> > > > > > Similarly, could I create a denial of service attack by inserting 
a
> > > > > > bunch of redirects to all the large files I can find? I'm not 
going
> > > > > > to attempt it this time around, I'm just wondering.
> > > > > >
> > > > > > -----
> > > > > >
> > > > > > The fcp client should be able to detect the redirects if wanted.
> > > > > >
> > > > > > My suggestion: a new progress, that show the redirects.
> > > > > >
> > > > > >    static final int showRedirects = 128;
> > > > > >
> > > > > >    verbose=verbose | showRedirects
> > > > > >
> > > > > >
> > > > > > Now the ClientGet returns a new progress message
> > > > > >
> > > > > >    FollowingRedirect
> > > > > >    Identifier=hello
> > > > > >    RequestedURI=freenet:KSK at something-1-2
> > > > > >    RedirectTargetURI=freenet:CHK at jdncdj,hdcbd,a8
> > > > > >    EndMessage
> > > > > >
> > > > > > Now the client app can decide how to handle it.
> > > > > >
> > > > > > thanks.
> > > > > >
> > > > > > --
> > > > > > Mfg
> > > > > > saces
> > > > >
> > > > > _______________________________________________
> > > > > Tech mailing list
> > > > > Tech at freenetproject.org
> > > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > > >
> > > > _______________________________________________
> > > > Tech mailing list
> > > > Tech at freenetproject.org
> > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > >
> > > _______________________________________________
> > > Tech mailing list
> > > Tech at freenetproject.org
> > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > _______________________________________________
> > Tech mailing list
> > Tech at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> >
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
>


From bbackde at googlemail.com  Wed Jan  3 21:32:52 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Wed, 3 Jan 2007 22:32:52 +0100
Subject: [Tech] EvilDude, Round 2
In-Reply-To: <200701032112.14882.dbkr@freenetproject.org>
References: <200701021554.50318.bo-le@web.de>
	<200701031732.31334.dbkr@freenetproject.org>
	<d6149a790701031142r4d747171p6599b31c50f1c9a2@mail.gmail.com>
	<200701032112.14882.dbkr@freenetproject.org>
Message-ID: <d6149a790701031332u5284e85ft1373a3014e033949@mail.gmail.com>

> I'm not really sure what you mean by a 'global messaging feature'.

I mean clients that rely on public keys, readable and writable be
everyone. Without private keys (like freemail) because you would have
to check the keys for each known user in this case. So each client
must know to what key it should write to if it inserts a new message,
and each client must know what keys to check to get new messages.

On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> On Wednesday 03 January 2007 19:42, bbackde at googlemail.com wrote:
> > What is wrong with the described advantages?
> >
> > Scenario:
> >
> > board_1 and board_2 exists. An attacker tries to spam them to make the
> > retrieval of messages nearly impossible.
> >
> > On board_1 he inserts 100 KSK keys with redirects to CHK keys that
> > were randomly created and point to no existing content. The insert
> > completes _very_ quickly.
> >
> > On board_2 he inserts 100 KSKs with random data behind it. The insert
> > takes a _much_ longer time to complete.
> >
> > Now a Frost client updates this board.
> >
> > On board_1 the client requests the 1st KSK. The node redirects
> > transparently to the CHK in the KSK redirect, and needs a _long_ time
> > until he gives up and reports code=28 to the client. The client does
> > not know if the request failed because the CHK does'nt exist at all,
> > or if the CHK could not be retrieved this time, but later. The client
> > steps over all 100 KSK keys (each one failing after a long search
> > time) and after this the client have to request all keys again during
> > next update, because some CHK might be retrievable now.
> >
> > On board_2 the client also requests the 1st KSK key. The node
> > redirects to the CHK key, and after a _shorter_ time the request is
> > successful or not. If it _is_ successful the client marks the slot
> > used (tried never again) and inspects the retrieved data, which are
> > random and therefore invalid. If the request was _not_ successful, the
> > client gets code=28, but this time he knows that the redirect points
> > to valid CHK key! So a retry will be successful later, and its worth
> > to retry.
> >
> > So why does the node even allow to insert a redirect to not-existing
> > data? He should check if the data is retrievable from the inserting
> > node. I can't understand why someone should insert a redirect to
> > content that cannot be retrieved by himself.
> >
> > I think an indication if the CHK is valid would be better than
> > without. Even with patched nodes, each insert node on the htl chain
> > could check if the CHK is known, and it is known if at least e.g. 2
> > nodes (on the chain or neighbors) say it is valid.
>
> This would surely make the insertion of redirects quite considerably slower?
> It also means that the network could refuse to insert a redirect just because
> some node on the chain can't retrieve the data, for whatever reason. These
> would be my reasons for allowing insertion of arbitrary redirects, but I
> didn't write it!
>
> Come to think of it, surely only the data and the routing key is sent on an
> insert, so the destination nodes can't check the redirect because they can't
> decrypt the data because they don't have the public key. In fact, they can't
> even tell it's a redirect.
>
> >
> > I don't know if my points are valid, but I want some explanations that
> > help me to understand this. A simple 'its not worth to think about',
> > without to provide a working circumvention for this, its not enough
> > for me.
> >
> > And why is there no type of key in 0.7 that is designed for
> > applications like Frost that provide a global messaging feature? There
> > are many improvements for freesites (e.g. USK) and easy filetransfer
> > (although to less detailed for clients, currently).
>
> I'm not really sure what you mean by a 'global messaging feature'.
>
>
>
> Dave
>
>
> >
> > On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > > > If a client could determine that the inserted KSK was inserted
> > > > as redirect to a specific key,
> > >
> > > That's the problem though - I don't think that's possible. Even if it were
> > > marked somehow on insertion, it would only require some minor hackery of
> the
> > > node's source to make a user-created redirect look like a node-created
> > > redirect.
> > >
> > > Again though, I'm still not convinced it's worth worrying about.
> > >
> > >
> > > Dave
> > >
> > > On Wednesday 03 January 2007 17:05, bbackde at googlemail.com wrote:
> > > > It would help to prevent the usage of specific keys that were inserted
> > > > as a redirect target of a KSK. If a client could determine that the
> > > > inserted KSK was inserted as redirect to a specific key, we could
> > > > easily drop and ignore this KSK. We would only allow KSKs that were
> > > > inserted as data. This prevents attackers from inserting alot of KSK
> > > > redirects (which is very fast, compared to the insert of a KSK with
> > > > the data) in a short time, and it prevents spreading of specific CHK
> > > > keys over KSKs.
> > > > So I see some advantages there. It should'nt not too hard IF the node
> > > > knows when a KSK was inserted as redirect to a specific CHK. If not I
> > > > think we will have some more problems with KSK attacks than under 0.5.
> > > > On 0.5 this kind of attack is not possible such easily...
> > > >
> > > > On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > > > > I don't really see as that's relevant, since it would be spoofable
> > > > > anyway. Surely the point is setting a limit on the size of a key when
> > > > > it's requested (frosy must do this, surely..?) and limiting the number
> of
> > > > > redirects (which the ClientGet command doesn't seem to have an option
> > > > > for, but I assume the node has a sensible limit and detects redirect
> > > > > loops).
> > > > >
> > > > > This would prevent the second attack. I don't really see that the
> first
> > > > > is a problem anyway, since the risk of downloading arbitrary content
> is
> > > > > implicit when using frost. You could potentially only allow keys that
> are
> > > > > non-redirect KSKs (thus limiting the size of a frost post to 1k, which
> > > > > may or may not be desirable). That would prevent it, but as I say, I
> > > > > don't really see it as a problem assuming that there's a filesize
> limit.
> > > > >
> > > > >
> > > > > Dave
> > > > >
> > > > > On Wednesday 03 January 2007 14:19, bbackde at googlemail.com wrote:
> > > > > > I think he means that a client should be able to determine if a
> > > > > > KSK/SSK was inserted as a manual redirect to another key, or if the
> > > > > > KSK/SSK was inserted with data and a redirect to the content was
> > > > > > automatically created by the node.
> > > > > >
> > > > > > Is it possible to separate this in the node? This would be a great
> and
> > > > > > simple solution.
> > > > > >
> > > > > > On 1/2/07, bo-le at web.de <bo-le at web.de> wrote:
> > > > > > > hi,
> > > > > > >
> > > > > > > just in the frost-board 'frost'
> > > > > > > ----- EvilDude ----- 2007.01.02 - 07:22:29GMT -----
> > > > > > >
> > > > > > > If I wanted to download a key but it was rare, could I increase my
> > > > > > > chances of getting it if I inserted a KSK redirecting to it,
> forcing
> > > > > > > every frost user to download that file?
> > > > > > >
> > > > > > > Similarly, could I create a denial of service attack by inserting
> a
> > > > > > > bunch of redirects to all the large files I can find? I'm not
> going
> > > > > > > to attempt it this time around, I'm just wondering.
> > > > > > >
> > > > > > > -----
> > > > > > >
> > > > > > > The fcp client should be able to detect the redirects if wanted.
> > > > > > >
> > > > > > > My suggestion: a new progress, that show the redirects.
> > > > > > >
> > > > > > >    static final int showRedirects = 128;
> > > > > > >
> > > > > > >    verbose=verbose | showRedirects
> > > > > > >
> > > > > > >
> > > > > > > Now the ClientGet returns a new progress message
> > > > > > >
> > > > > > >    FollowingRedirect
> > > > > > >    Identifier=hello
> > > > > > >    RequestedURI=freenet:KSK at something-1-2
> > > > > > >    RedirectTargetURI=freenet:CHK at jdncdj,hdcbd,a8
> > > > > > >    EndMessage
> > > > > > >
> > > > > > > Now the client app can decide how to handle it.
> > > > > > >
> > > > > > > thanks.
> > > > > > >
> > > > > > > --
> > > > > > > Mfg
> > > > > > > saces
> > > > > >
> > > > > > _______________________________________________
> > > > > > Tech mailing list
> > > > > > Tech at freenetproject.org
> > > > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > > > >
> > > > > _______________________________________________
> > > > > Tech mailing list
> > > > > Tech at freenetproject.org
> > > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > > >
> > > > _______________________________________________
> > > > Tech mailing list
> > > > Tech at freenetproject.org
> > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > > _______________________________________________
> > > Tech mailing list
> > > Tech at freenetproject.org
> > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > >
> > _______________________________________________
> > Tech mailing list
> > Tech at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> >
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
>


From dbkr at freenetproject.org  Thu Jan  4 08:58:57 2007
From: dbkr at freenetproject.org (Dave Baker)
Date: Thu, 4 Jan 2007 08:58:57 +0000
Subject: [Tech] Global messaging (was: EvilDude, Round 2)
In-Reply-To: <d6149a790701031332u5284e85ft1373a3014e033949@mail.gmail.com>
References: <200701021554.50318.bo-le@web.de>
	<200701032112.14882.dbkr@freenetproject.org>
	<d6149a790701031332u5284e85ft1373a3014e033949@mail.gmail.com>
Message-ID: <200701040858.57178.dbkr@freenetproject.org>

On Wednesday 03 January 2007 21:32, bbackde at googlemail.com wrote:
> > I'm not really sure what you mean by a 'global messaging feature'.
>
> I mean clients that rely on public keys, readable and writable be
> everyone. Without private keys (like freemail) because you would have
> to check the keys for each known user in this case. So each client
> must know to what key it should write to if it inserts a new message,
> and each client must know what keys to check to get new messages.

Sorry - I still don't understand. A key that's readable and writable to 
everyone - is that not a KSK?


Dave


>
> On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > On Wednesday 03 January 2007 19:42, bbackde at googlemail.com wrote:
> > > What is wrong with the described advantages?
> > >
> > > Scenario:
> > >
> > > board_1 and board_2 exists. An attacker tries to spam them to make the
> > > retrieval of messages nearly impossible.
> > >
> > > On board_1 he inserts 100 KSK keys with redirects to CHK keys that
> > > were randomly created and point to no existing content. The insert
> > > completes _very_ quickly.
> > >
> > > On board_2 he inserts 100 KSKs with random data behind it. The insert
> > > takes a _much_ longer time to complete.
> > >
> > > Now a Frost client updates this board.
> > >
> > > On board_1 the client requests the 1st KSK. The node redirects
> > > transparently to the CHK in the KSK redirect, and needs a _long_ time
> > > until he gives up and reports code=28 to the client. The client does
> > > not know if the request failed because the CHK does'nt exist at all,
> > > or if the CHK could not be retrieved this time, but later. The client
> > > steps over all 100 KSK keys (each one failing after a long search
> > > time) and after this the client have to request all keys again during
> > > next update, because some CHK might be retrievable now.
> > >
> > > On board_2 the client also requests the 1st KSK key. The node
> > > redirects to the CHK key, and after a _shorter_ time the request is
> > > successful or not. If it _is_ successful the client marks the slot
> > > used (tried never again) and inspects the retrieved data, which are
> > > random and therefore invalid. If the request was _not_ successful, the
> > > client gets code=28, but this time he knows that the redirect points
> > > to valid CHK key! So a retry will be successful later, and its worth
> > > to retry.
> > >
> > > So why does the node even allow to insert a redirect to not-existing
> > > data? He should check if the data is retrievable from the inserting
> > > node. I can't understand why someone should insert a redirect to
> > > content that cannot be retrieved by himself.
> > >
> > > I think an indication if the CHK is valid would be better than
> > > without. Even with patched nodes, each insert node on the htl chain
> > > could check if the CHK is known, and it is known if at least e.g. 2
> > > nodes (on the chain or neighbors) say it is valid.
> >
> > This would surely make the insertion of redirects quite considerably
> > slower? It also means that the network could refuse to insert a redirect
> > just because some node on the chain can't retrieve the data, for whatever
> > reason. These would be my reasons for allowing insertion of arbitrary
> > redirects, but I didn't write it!
> >
> > Come to think of it, surely only the data and the routing key is sent on
> > an insert, so the destination nodes can't check the redirect because they
> > can't decrypt the data because they don't have the public key. In fact,
> > they can't even tell it's a redirect.
> >
> > > I don't know if my points are valid, but I want some explanations that
> > > help me to understand this. A simple 'its not worth to think about',
> > > without to provide a working circumvention for this, its not enough
> > > for me.
> > >
> > > And why is there no type of key in 0.7 that is designed for
> > > applications like Frost that provide a global messaging feature? There
> > > are many improvements for freesites (e.g. USK) and easy filetransfer
> > > (although to less detailed for clients, currently).
> >
> > I'm not really sure what you mean by a 'global messaging feature'.
> >
> >
> >
> > Dave
> >
> > > On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > > > > If a client could determine that the inserted KSK was inserted
> > > > > as redirect to a specific key,
> > > >
> > > > That's the problem though - I don't think that's possible. Even if it
> > > > were marked somehow on insertion, it would only require some minor
> > > > hackery of
> >
> > the
> >
> > > > node's source to make a user-created redirect look like a
> > > > node-created redirect.
> > > >
> > > > Again though, I'm still not convinced it's worth worrying about.
> > > >
> > > >
> > > > Dave
> > > >
> > > > On Wednesday 03 January 2007 17:05, bbackde at googlemail.com wrote:
> > > > > It would help to prevent the usage of specific keys that were
> > > > > inserted as a redirect target of a KSK. If a client could determine
> > > > > that the inserted KSK was inserted as redirect to a specific key,
> > > > > we could easily drop and ignore this KSK. We would only allow KSKs
> > > > > that were inserted as data. This prevents attackers from inserting
> > > > > alot of KSK redirects (which is very fast, compared to the insert
> > > > > of a KSK with the data) in a short time, and it prevents spreading
> > > > > of specific CHK keys over KSKs.
> > > > > So I see some advantages there. It should'nt not too hard IF the
> > > > > node knows when a KSK was inserted as redirect to a specific CHK.
> > > > > If not I think we will have some more problems with KSK attacks
> > > > > than under 0.5. On 0.5 this kind of attack is not possible such
> > > > > easily...
> > > > >
> > > > > On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > > > > > I don't really see as that's relevant, since it would be
> > > > > > spoofable anyway. Surely the point is setting a limit on the size
> > > > > > of a key when it's requested (frosy must do this, surely..?) and
> > > > > > limiting the number
> >
> > of
> >
> > > > > > redirects (which the ClientGet command doesn't seem to have an
> > > > > > option for, but I assume the node has a sensible limit and
> > > > > > detects redirect loops).
> > > > > >
> > > > > > This would prevent the second attack. I don't really see that the
> >
> > first
> >
> > > > > > is a problem anyway, since the risk of downloading arbitrary
> > > > > > content
> >
> > is
> >
> > > > > > implicit when using frost. You could potentially only allow keys
> > > > > > that
> >
> > are
> >
> > > > > > non-redirect KSKs (thus limiting the size of a frost post to 1k,
> > > > > > which may or may not be desirable). That would prevent it, but as
> > > > > > I say, I don't really see it as a problem assuming that there's a
> > > > > > filesize
> >
> > limit.
> >
> > > > > > Dave
> > > > > >
> > > > > > On Wednesday 03 January 2007 14:19, bbackde at googlemail.com wrote:
> > > > > > > I think he means that a client should be able to determine if a
> > > > > > > KSK/SSK was inserted as a manual redirect to another key, or if
> > > > > > > the KSK/SSK was inserted with data and a redirect to the
> > > > > > > content was automatically created by the node.
> > > > > > >
> > > > > > > Is it possible to separate this in the node? This would be a
> > > > > > > great
> >
> > and
> >
> > > > > > > simple solution.
> > > > > > >
> > > > > > > On 1/2/07, bo-le at web.de <bo-le at web.de> wrote:
> > > > > > > > hi,
> > > > > > > >
> > > > > > > > just in the frost-board 'frost'
> > > > > > > > ----- EvilDude ----- 2007.01.02 - 07:22:29GMT -----
> > > > > > > >
> > > > > > > > If I wanted to download a key but it was rare, could I
> > > > > > > > increase my chances of getting it if I inserted a KSK
> > > > > > > > redirecting to it,
> >
> > forcing
> >
> > > > > > > > every frost user to download that file?
> > > > > > > >
> > > > > > > > Similarly, could I create a denial of service attack by
> > > > > > > > inserting
> >
> > a
> >
> > > > > > > > bunch of redirects to all the large files I can find? I'm not
> >
> > going
> >
> > > > > > > > to attempt it this time around, I'm just wondering.
> > > > > > > >
> > > > > > > > -----
> > > > > > > >
> > > > > > > > The fcp client should be able to detect the redirects if
> > > > > > > > wanted.
> > > > > > > >
> > > > > > > > My suggestion: a new progress, that show the redirects.
> > > > > > > >
> > > > > > > >    static final int showRedirects = 128;
> > > > > > > >
> > > > > > > >    verbose=verbose | showRedirects
> > > > > > > >
> > > > > > > >
> > > > > > > > Now the ClientGet returns a new progress message
> > > > > > > >
> > > > > > > >    FollowingRedirect
> > > > > > > >    Identifier=hello
> > > > > > > >    RequestedURI=freenet:KSK at something-1-2
> > > > > > > >    RedirectTargetURI=freenet:CHK at jdncdj,hdcbd,a8
> > > > > > > >    EndMessage
> > > > > > > >
> > > > > > > > Now the client app can decide how to handle it.
> > > > > > > >
> > > > > > > > thanks.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Mfg
> > > > > > > > saces
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > Tech mailing list
> > > > > > > Tech at freenetproject.org
> > > > > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > > > > >
> > > > > > _______________________________________________
> > > > > > Tech mailing list
> > > > > > Tech at freenetproject.org
> > > > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > > > >
> > > > > _______________________________________________
> > > > > Tech mailing list
> > > > > Tech at freenetproject.org
> > > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > > >
> > > > _______________________________________________
> > > > Tech mailing list
> > > > Tech at freenetproject.org
> > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > >
> > > _______________________________________________
> > > Tech mailing list
> > > Tech at freenetproject.org
> > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> >
> > _______________________________________________
> > Tech mailing list
> > Tech at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
>
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech


From bbackde at googlemail.com  Thu Jan  4 09:21:54 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Thu, 4 Jan 2007 10:21:54 +0100
Subject: [Tech] Global messaging (was: EvilDude, Round 2)
In-Reply-To: <200701040858.57178.dbkr@freenetproject.org>
References: <200701021554.50318.bo-le@web.de>
	<200701032112.14882.dbkr@freenetproject.org>
	<d6149a790701031332u5284e85ft1373a3014e033949@mail.gmail.com>
	<200701040858.57178.dbkr@freenetproject.org>
Message-ID: <d6149a790701040121l7ad0c9bbja3532a699dca887d@mail.gmail.com>

> Sorry - I still don't understand. A key that's readable and writable to
> everyone - is that not a KSK?

Yes, thats KSK. But you see the problems we have with the 'new' KSK
keys which are 1KB in size and provide transparent redirects to other
keys? KSKs on 0.5 were great for Frosts messaging (32KB size, no
transparent redirects), but the 'new' KSKs introduced the problems we
talk about here (new kind of attacks, ...).

The devs tried to make it easier for clients, but now some clients
have serious problems.
It could help to introduce a new type of freenet key, like KSK but
without redirects and 32KB maximum size (like CHK). This would be
perfect for Frost like clients.


On 1/4/07, Dave Baker <dbkr at freenetproject.org> wrote:
> On Wednesday 03 January 2007 21:32, bbackde at googlemail.com wrote:
> > > I'm not really sure what you mean by a 'global messaging feature'.
> >
> > I mean clients that rely on public keys, readable and writable be
> > everyone. Without private keys (like freemail) because you would have
> > to check the keys for each known user in this case. So each client
> > must know to what key it should write to if it inserts a new message,
> > and each client must know what keys to check to get new messages.
>
> Sorry - I still don't understand. A key that's readable and writable to
> everyone - is that not a KSK?
>
>
> Dave
>
>
> >
> > On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > > On Wednesday 03 January 2007 19:42, bbackde at googlemail.com wrote:
> > > > What is wrong with the described advantages?
> > > >
> > > > Scenario:
> > > >
> > > > board_1 and board_2 exists. An attacker tries to spam them to make the
> > > > retrieval of messages nearly impossible.
> > > >
> > > > On board_1 he inserts 100 KSK keys with redirects to CHK keys that
> > > > were randomly created and point to no existing content. The insert
> > > > completes _very_ quickly.
> > > >
> > > > On board_2 he inserts 100 KSKs with random data behind it. The insert
> > > > takes a _much_ longer time to complete.
> > > >
> > > > Now a Frost client updates this board.
> > > >
> > > > On board_1 the client requests the 1st KSK. The node redirects
> > > > transparently to the CHK in the KSK redirect, and needs a _long_ time
> > > > until he gives up and reports code=28 to the client. The client does
> > > > not know if the request failed because the CHK does'nt exist at all,
> > > > or if the CHK could not be retrieved this time, but later. The client
> > > > steps over all 100 KSK keys (each one failing after a long search
> > > > time) and after this the client have to request all keys again during
> > > > next update, because some CHK might be retrievable now.
> > > >
> > > > On board_2 the client also requests the 1st KSK key. The node
> > > > redirects to the CHK key, and after a _shorter_ time the request is
> > > > successful or not. If it _is_ successful the client marks the slot
> > > > used (tried never again) and inspects the retrieved data, which are
> > > > random and therefore invalid. If the request was _not_ successful, the
> > > > client gets code=28, but this time he knows that the redirect points
> > > > to valid CHK key! So a retry will be successful later, and its worth
> > > > to retry.
> > > >
> > > > So why does the node even allow to insert a redirect to not-existing
> > > > data? He should check if the data is retrievable from the inserting
> > > > node. I can't understand why someone should insert a redirect to
> > > > content that cannot be retrieved by himself.
> > > >
> > > > I think an indication if the CHK is valid would be better than
> > > > without. Even with patched nodes, each insert node on the htl chain
> > > > could check if the CHK is known, and it is known if at least e.g. 2
> > > > nodes (on the chain or neighbors) say it is valid.
> > >
> > > This would surely make the insertion of redirects quite considerably
> > > slower? It also means that the network could refuse to insert a redirect
> > > just because some node on the chain can't retrieve the data, for whatever
> > > reason. These would be my reasons for allowing insertion of arbitrary
> > > redirects, but I didn't write it!
> > >
> > > Come to think of it, surely only the data and the routing key is sent on
> > > an insert, so the destination nodes can't check the redirect because they
> > > can't decrypt the data because they don't have the public key. In fact,
> > > they can't even tell it's a redirect.
> > >
> > > > I don't know if my points are valid, but I want some explanations that
> > > > help me to understand this. A simple 'its not worth to think about',
> > > > without to provide a working circumvention for this, its not enough
> > > > for me.
> > > >
> > > > And why is there no type of key in 0.7 that is designed for
> > > > applications like Frost that provide a global messaging feature? There
> > > > are many improvements for freesites (e.g. USK) and easy filetransfer
> > > > (although to less detailed for clients, currently).
> > >
> > > I'm not really sure what you mean by a 'global messaging feature'.
> > >
> > >
> > >
> > > Dave
> > >
> > > > On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > > > > > If a client could determine that the inserted KSK was inserted
> > > > > > as redirect to a specific key,
> > > > >
> > > > > That's the problem though - I don't think that's possible. Even if it
> > > > > were marked somehow on insertion, it would only require some minor
> > > > > hackery of
> > >
> > > the
> > >
> > > > > node's source to make a user-created redirect look like a
> > > > > node-created redirect.
> > > > >
> > > > > Again though, I'm still not convinced it's worth worrying about.
> > > > >
> > > > >
> > > > > Dave
> > > > >
> > > > > On Wednesday 03 January 2007 17:05, bbackde at googlemail.com wrote:
> > > > > > It would help to prevent the usage of specific keys that were
> > > > > > inserted as a redirect target of a KSK. If a client could determine
> > > > > > that the inserted KSK was inserted as redirect to a specific key,
> > > > > > we could easily drop and ignore this KSK. We would only allow KSKs
> > > > > > that were inserted as data. This prevents attackers from inserting
> > > > > > alot of KSK redirects (which is very fast, compared to the insert
> > > > > > of a KSK with the data) in a short time, and it prevents spreading
> > > > > > of specific CHK keys over KSKs.
> > > > > > So I see some advantages there. It should'nt not too hard IF the
> > > > > > node knows when a KSK was inserted as redirect to a specific CHK.
> > > > > > If not I think we will have some more problems with KSK attacks
> > > > > > than under 0.5. On 0.5 this kind of attack is not possible such
> > > > > > easily...
> > > > > >
> > > > > > On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > > > > > > I don't really see as that's relevant, since it would be
> > > > > > > spoofable anyway. Surely the point is setting a limit on the size
> > > > > > > of a key when it's requested (frosy must do this, surely..?) and
> > > > > > > limiting the number
> > >
> > > of
> > >
> > > > > > > redirects (which the ClientGet command doesn't seem to have an
> > > > > > > option for, but I assume the node has a sensible limit and
> > > > > > > detects redirect loops).
> > > > > > >
> > > > > > > This would prevent the second attack. I don't really see that the
> > >
> > > first
> > >
> > > > > > > is a problem anyway, since the risk of downloading arbitrary
> > > > > > > content
> > >
> > > is
> > >
> > > > > > > implicit when using frost. You could potentially only allow keys
> > > > > > > that
> > >
> > > are
> > >
> > > > > > > non-redirect KSKs (thus limiting the size of a frost post to 1k,
> > > > > > > which may or may not be desirable). That would prevent it, but as
> > > > > > > I say, I don't really see it as a problem assuming that there's a
> > > > > > > filesize
> > >
> > > limit.
> > >
> > > > > > > Dave
> > > > > > >
> > > > > > > On Wednesday 03 January 2007 14:19, bbackde at googlemail.com wrote:
> > > > > > > > I think he means that a client should be able to determine if a
> > > > > > > > KSK/SSK was inserted as a manual redirect to another key, or if
> > > > > > > > the KSK/SSK was inserted with data and a redirect to the
> > > > > > > > content was automatically created by the node.
> > > > > > > >
> > > > > > > > Is it possible to separate this in the node? This would be a
> > > > > > > > great
> > >
> > > and
> > >
> > > > > > > > simple solution.
> > > > > > > >
> > > > > > > > On 1/2/07, bo-le at web.de <bo-le at web.de> wrote:
> > > > > > > > > hi,
> > > > > > > > >
> > > > > > > > > just in the frost-board 'frost'
> > > > > > > > > ----- EvilDude ----- 2007.01.02 - 07:22:29GMT -----
> > > > > > > > >
> > > > > > > > > If I wanted to download a key but it was rare, could I
> > > > > > > > > increase my chances of getting it if I inserted a KSK
> > > > > > > > > redirecting to it,
> > >
> > > forcing
> > >
> > > > > > > > > every frost user to download that file?
> > > > > > > > >
> > > > > > > > > Similarly, could I create a denial of service attack by
> > > > > > > > > inserting
> > >
> > > a
> > >
> > > > > > > > > bunch of redirects to all the large files I can find? I'm not
> > >
> > > going
> > >
> > > > > > > > > to attempt it this time around, I'm just wondering.
> > > > > > > > >
> > > > > > > > > -----
> > > > > > > > >
> > > > > > > > > The fcp client should be able to detect the redirects if
> > > > > > > > > wanted.
> > > > > > > > >
> > > > > > > > > My suggestion: a new progress, that show the redirects.
> > > > > > > > >
> > > > > > > > >    static final int showRedirects = 128;
> > > > > > > > >
> > > > > > > > >    verbose=verbose | showRedirects
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Now the ClientGet returns a new progress message
> > > > > > > > >
> > > > > > > > >    FollowingRedirect
> > > > > > > > >    Identifier=hello
> > > > > > > > >    RequestedURI=freenet:KSK at something-1-2
> > > > > > > > >    RedirectTargetURI=freenet:CHK at jdncdj,hdcbd,a8
> > > > > > > > >    EndMessage
> > > > > > > > >
> > > > > > > > > Now the client app can decide how to handle it.
> > > > > > > > >
> > > > > > > > > thanks.
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Mfg
> > > > > > > > > saces
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > Tech mailing list
> > > > > > > > Tech at freenetproject.org
> > > > > > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > Tech mailing list
> > > > > > > Tech at freenetproject.org
> > > > > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > > > > >
> > > > > > _______________________________________________
> > > > > > Tech mailing list
> > > > > > Tech at freenetproject.org
> > > > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > > > >
> > > > > _______________________________________________
> > > > > Tech mailing list
> > > > > Tech at freenetproject.org
> > > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > > >
> > > > _______________________________________________
> > > > Tech mailing list
> > > > Tech at freenetproject.org
> > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > >
> > > _______________________________________________
> > > Tech mailing list
> > > Tech at freenetproject.org
> > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> >
> > _______________________________________________
> > Tech mailing list
> > Tech at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
>


From drriley at cogeco.ca  Thu Jan  4 09:33:20 2007
From: drriley at cogeco.ca (Dr. R. Riley)
Date: Thu, 04 Jan 2007 04:33:20 -0500
Subject: [Tech] Global messaging
In-Reply-To: <200701040858.57178.dbkr@freenetproject.org>
References: <200701021554.50318.bo-le@web.de>	<200701032112.14882.dbkr@freenetproject.org>	<d6149a790701031332u5284e85ft1373a3014e033949@mail.gmail.com>
	<200701040858.57178.dbkr@freenetproject.org>
Message-ID: <459CC9E0.40209@cogeco.ca>

Sorry that this is rather off-topic, but seeing as there  is lots of 
discussion perhaps I could interject and ask for some volunteers to 
trade node refs with me.  I have only found one willing partner thus 
far, and I understand I need 3 for freenet to work?  If you are 
willing/able, please contact me at drriley at cogeco.ca

Thanks.

Dave Baker wrote:
> On Wednesday 03 January 2007 21:32, bbackde at googlemail.com wrote:
>   
>>> I'm not really sure what you mean by a 'global messaging feature'.
>>>       
>> <SNIP>
>   


From nextgens at freenetproject.org  Thu Jan  4 14:31:51 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re_=28NextGen$=29?=)
Date: Thu, 4 Jan 2007 15:31:51 +0100
Subject: [Tech] Node gets more FEC blocks than needed
In-Reply-To: <d6149a790612291304s416823e3k9a287164bcad014f@mail.gmail.com>
References: <d6149a790612291304s416823e3k9a287164bcad014f@mail.gmail.com>
Message-ID: <20070104143150.GC4525@freenetproject.org>

* bbackde at googlemail.com <bbackde at googlemail.com> [2006-12-29 22:04:07]:

> Following SimpleProgress shows it: some threads in freenet node seems
> to be started and do not finish if we have all the needed blocks:
> 
> SimpleProgress {Total=9606, FatallyFailed=2, FinalizedTotal=true, Failed=287, Su
> cceeded=6411, Identifier=get-11673399350836773, Required=6404} EndMessage
> 
> Its a request with DDA enabled, not in global queue.
> 
> Reasons? Fixes? Ideas?

Well, it's a wholly erroneous interpretation of what "required" means.
It does mean that the download *can't* complete without that "amount" of
blocks... but it doesn't mean it will.

Ever heard of ForwardErrorCorrection and Hamming distance ?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070104/1d3f7af3/attachment.pgp 

From nextgens at freenetproject.org  Thu Jan  4 14:36:52 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re_=28NextGen$=29?=)
Date: Thu, 4 Jan 2007 15:36:52 +0100
Subject: [Tech] SimpleProgress does not show filesize
In-Reply-To: <d6149a790612291309t204ff82cu4bc7071507c2477a@mail.gmail.com>
References: <d6149a790612291309t204ff82cu4bc7071507c2477a@mail.gmail.com>
Message-ID: <20070104143652.GD4525@freenetproject.org>

* bbackde at googlemail.com <bbackde at googlemail.com> [2006-12-29 22:09:04]:

> If a SimpleProgress message is sent with Finalized=true, does'nt the
> node know the final filesize at this time? I assume yes, and I want
> that the size is sent in the SimpleProgress message if it is known.

And I *want* you to fill in a ticket on the bug tracking system using the
category "wishlist or feature" :p

More seriously speaking ... they are more urgent matters to be addressed
; will do but not now.

NextGen$
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070104/de831077/attachment.pgp 

From bbackde at googlemail.com  Thu Jan  4 14:40:24 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Thu, 4 Jan 2007 15:40:24 +0100
Subject: [Tech] Node gets more FEC blocks than needed
In-Reply-To: <20070104143150.GC4525@freenetproject.org>
References: <d6149a790612291304s416823e3k9a287164bcad014f@mail.gmail.com>
	<20070104143150.GC4525@freenetproject.org>
Message-ID: <d6149a790701040640o4dde48afm3f5e3f4891d33b8f@mail.gmail.com>

Of course I heard about FEC, in fact I implemented all this stuff in
the 0.5 part of Frost by myself ;)

And thats why I thought if the node says X blocks are required, then
this amount of blocks is required to decode the file (no matter if
data or check blocks).
So please tell me what Required means in the SimpleProgress message
then? Or do you mean that there were not enough blocks for one
segment, but more than enough for another segment, is it this?

On 1/4/07, Florent Daigni?re (NextGen$) <nextgens at freenetproject.org> wrote:
> * bbackde at googlemail.com <bbackde at googlemail.com> [2006-12-29 22:04:07]:
>
> > Following SimpleProgress shows it: some threads in freenet node seems
> > to be started and do not finish if we have all the needed blocks:
> >
> > SimpleProgress {Total=9606, FatallyFailed=2, FinalizedTotal=true, Failed=287, Su
> > cceeded=6411, Identifier=get-11673399350836773, Required=6404} EndMessage
> >
> > Its a request with DDA enabled, not in global queue.
> >
> > Reasons? Fixes? Ideas?
>
> Well, it's a wholly erroneous interpretation of what "required" means.
> It does mean that the download *can't* complete without that "amount" of
> blocks... but it doesn't mean it will.
>
> Ever heard of ForwardErrorCorrection and Hamming distance ?
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFFnQ/WU/Z/dHFfxtcRAoI9AKDq4rlL0ENI6QpMU9Xb4h+w5hhRKACgwa8M
> 5DE+vbm0kb3I9Ptl41VwjRs=
> =8rvB
> -----END PGP SIGNATURE-----
>
>
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
>
>


From bbackde at googlemail.com  Thu Jan  4 14:41:25 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Thu, 4 Jan 2007 15:41:25 +0100
Subject: [Tech] SimpleProgress does not show filesize
In-Reply-To: <20070104143652.GD4525@freenetproject.org>
References: <d6149a790612291309t204ff82cu4bc7071507c2477a@mail.gmail.com>
	<20070104143652.GD4525@freenetproject.org>
Message-ID: <d6149a790701040641k3bd66d86q49753a07f9efadc9@mail.gmail.com>

Ok I will do. In the meantime Frost computes the appr. size out of the
total blocks reported in the SimpleProgress message....

On 1/4/07, Florent Daigni?re (NextGen$) <nextgens at freenetproject.org> wrote:
> * bbackde at googlemail.com <bbackde at googlemail.com> [2006-12-29 22:09:04]:
>
> > If a SimpleProgress message is sent with Finalized=true, does'nt the
> > node know the final filesize at this time? I assume yes, and I want
> > that the size is sent in the SimpleProgress message if it is known.
>
> And I *want* you to fill in a ticket on the bug tracking system using the
> category "wishlist or feature" :p
>
> More seriously speaking ... they are more urgent matters to be addressed
> ; will do but not now.
>
> NextGen$
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFFnREEU/Z/dHFfxtcRApGMAKDywR4DSCwDj+/CWmTna/7JdRWSYwCghIIN
> pCJIVZdUAzQeR2FZ+hrAHCM=
> =ZQCM
> -----END PGP SIGNATURE-----
>
>
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
>
>


From nextgens at freenetproject.org  Thu Jan  4 14:42:55 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re_=28NextGen$=29?=)
Date: Thu, 4 Jan 2007 15:42:55 +0100
Subject: [Tech] Node gets more FEC blocks than needed
In-Reply-To: <d6149a790701040640o4dde48afm3f5e3f4891d33b8f@mail.gmail.com>
References: <d6149a790612291304s416823e3k9a287164bcad014f@mail.gmail.com>
	<20070104143150.GC4525@freenetproject.org>
	<d6149a790701040640o4dde48afm3f5e3f4891d33b8f@mail.gmail.com>
Message-ID: <20070104144254.GE4525@freenetproject.org>

* bbackde at googlemail.com <bbackde at googlemail.com> [2007-01-04 15:40:24]:

> Of course I heard about FEC, in fact I implemented all this stuff in
> the 0.5 part of Frost by myself ;)
> 
> And thats why I thought if the node says X blocks are required, then
> this amount of blocks is required to decode the file (no matter if
> data or check blocks).
> So please tell me what Required means in the SimpleProgress message
> then? Or do you mean that there were not enough blocks for one
> segment, but more than enough for another segment, is it this?

That's probably it ... it's hard to tell without logs.

> 
> On 1/4/07, Florent Daigni?re (NextGen$) <nextgens at freenetproject.org> wrote:
> > * bbackde at googlemail.com <bbackde at googlemail.com> [2006-12-29 22:04:07]:
> >
> > > Following SimpleProgress shows it: some threads in freenet node seems
> > > to be started and do not finish if we have all the needed blocks:
> > >
> > > SimpleProgress {Total=9606, FatallyFailed=2, FinalizedTotal=true, Failed=287, Su
> > > cceeded=6411, Identifier=get-11673399350836773, Required=6404} EndMessage
> > >
> > > Its a request with DDA enabled, not in global queue.
> > >
> > > Reasons? Fixes? Ideas?
> >
> > Well, it's a wholly erroneous interpretation of what "required" means.
> > It does mean that the download *can't* complete without that "amount" of
> > blocks... but it doesn't mean it will.
> >
> > Ever heard of ForwardErrorCorrection and Hamming distance ?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070104/0bd8b02f/attachment.pgp 

From bbackde at googlemail.com  Thu Jan  4 14:48:09 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Thu, 4 Jan 2007 15:48:09 +0100
Subject: [Tech] Node gets more FEC blocks than needed
In-Reply-To: <20070104144254.GE4525@freenetproject.org>
References: <d6149a790612291304s416823e3k9a287164bcad014f@mail.gmail.com>
	<20070104143150.GC4525@freenetproject.org>
	<d6149a790701040640o4dde48afm3f5e3f4891d33b8f@mail.gmail.com>
	<20070104144254.GE4525@freenetproject.org>
Message-ID: <d6149a790701040648g2869aafdtb4fc970ceb4c19b9@mail.gmail.com>

Ok, so I will open a feature request for this issue. Because with the
informations provided in SimpleProgress a freenet client cannot
display a correct progress to the user. Either we need more
informations about the progress of each segment, or the node
normalizes this by itself and provides a percentage :)

But the current behavior often leads to downloads that have more than
100% progress *g*

On 1/4/07, Florent Daigni?re (NextGen$) <nextgens at freenetproject.org> wrote:
> * bbackde at googlemail.com <bbackde at googlemail.com> [2007-01-04 15:40:24]:
>
> > Of course I heard about FEC, in fact I implemented all this stuff in
> > the 0.5 part of Frost by myself ;)
> >
> > And thats why I thought if the node says X blocks are required, then
> > this amount of blocks is required to decode the file (no matter if
> > data or check blocks).
> > So please tell me what Required means in the SimpleProgress message
> > then? Or do you mean that there were not enough blocks for one
> > segment, but more than enough for another segment, is it this?
>
> That's probably it ... it's hard to tell without logs.
>
> >
> > On 1/4/07, Florent Daigni?re (NextGen$) <nextgens at freenetproject.org> wrote:
> > > * bbackde at googlemail.com <bbackde at googlemail.com> [2006-12-29 22:04:07]:
> > >
> > > > Following SimpleProgress shows it: some threads in freenet node seems
> > > > to be started and do not finish if we have all the needed blocks:
> > > >
> > > > SimpleProgress {Total=9606, FatallyFailed=2, FinalizedTotal=true, Failed=287, Su
> > > > cceeded=6411, Identifier=get-11673399350836773, Required=6404} EndMessage
> > > >
> > > > Its a request with DDA enabled, not in global queue.
> > > >
> > > > Reasons? Fixes? Ideas?
> > >
> > > Well, it's a wholly erroneous interpretation of what "required" means.
> > > It does mean that the download *can't* complete without that "amount" of
> > > blocks... but it doesn't mean it will.
> > >
> > > Ever heard of ForwardErrorCorrection and Hamming distance ?
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFFnRJuU/Z/dHFfxtcRAtxsAJ0b1s3hkAc5U/MVnz5iVBMsI6O2AgCfRwOW
> En/iX2M3+WV8QIpGwJw2tpk=
> =B4sk
> -----END PGP SIGNATURE-----
>
>
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
>
>


From nextgens at freenetproject.org  Thu Jan  4 15:03:22 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re_=28NextGen$=29?=)
Date: Thu, 4 Jan 2007 16:03:22 +0100
Subject: [Tech] Node gets more FEC blocks than needed
In-Reply-To: <d6149a790701040648g2869aafdtb4fc970ceb4c19b9@mail.gmail.com>
References: <d6149a790612291304s416823e3k9a287164bcad014f@mail.gmail.com>
	<20070104143150.GC4525@freenetproject.org>
	<d6149a790701040640o4dde48afm3f5e3f4891d33b8f@mail.gmail.com>
	<20070104144254.GE4525@freenetproject.org>
	<d6149a790701040648g2869aafdtb4fc970ceb4c19b9@mail.gmail.com>
Message-ID: <20070104150321.GG4525@freenetproject.org>

* bbackde at googlemail.com <bbackde at googlemail.com> [2007-01-04 15:48:09]:

> Ok, so I will open a feature request for this issue. Because with the
> informations provided in SimpleProgress a freenet client cannot
> display a correct progress to the user. Either we need more
> informations about the progress of each segment, or the node
> normalizes this by itself and provides a percentage :)
> 
> But the current behavior often leads to downloads that have more than
> 100% progress *g*

Is it *that* important to have an accurate progress bar ? some blocks
can take *ages* to be retrived whereas some will come up
instantaneously... Imho we can't provide a reliable
percentage/ETA/whatever for small files even if we had detailled stats
on what's going on.

Btw, haven't you ever experienced the same "bug" on the queue toadlet on
fproxy ? more than a lack of feedback from fcp, it's a network
charateristic : we can't predict precisely the time needed to fetch
anything and even less waranty it will be delivered on time.

Making FCP more verbose is probably a good idea, but do not think it
will enable you to display a precise ETA from it. Keep in mind what
the 'E' is standing for ;)

> On 1/4/07, Florent Daigni?re (NextGen$) <nextgens at freenetproject.org> wrote:
> > * bbackde at googlemail.com <bbackde at googlemail.com> [2007-01-04 15:40:24]:
> >
> > > Of course I heard about FEC, in fact I implemented all this stuff in
> > > the 0.5 part of Frost by myself ;)
> > >
> > > And thats why I thought if the node says X blocks are required, then
> > > this amount of blocks is required to decode the file (no matter if
> > > data or check blocks).
> > > So please tell me what Required means in the SimpleProgress message
> > > then? Or do you mean that there were not enough blocks for one
> > > segment, but more than enough for another segment, is it this?
> >
> > That's probably it ... it's hard to tell without logs.
> >
> > >
> > > On 1/4/07, Florent Daigni?re (NextGen$) <nextgens at freenetproject.org> wrote:
> > > > * bbackde at googlemail.com <bbackde at googlemail.com> [2006-12-29 22:04:07]:
> > > >
> > > > > Following SimpleProgress shows it: some threads in freenet node seems
> > > > > to be started and do not finish if we have all the needed blocks:
> > > > >
> > > > > SimpleProgress {Total=9606, FatallyFailed=2, FinalizedTotal=true, Failed=287, Su
> > > > > cceeded=6411, Identifier=get-11673399350836773, Required=6404} EndMessage
> > > > >
> > > > > Its a request with DDA enabled, not in global queue.
> > > > >
> > > > > Reasons? Fixes? Ideas?
> > > >
> > > > Well, it's a wholly erroneous interpretation of what "required" means.
> > > > It does mean that the download *can't* complete without that "amount" of
> > > > blocks... but it doesn't mean it will.
> > > >
> > > > Ever heard of ForwardErrorCorrection and Hamming distance ?
> >
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.6 (GNU/Linux)
> >
> > iD8DBQFFnRJuU/Z/dHFfxtcRAtxsAJ0b1s3hkAc5U/MVnz5iVBMsI6O2AgCfRwOW
> > En/iX2M3+WV8QIpGwJw2tpk=
> > =B4sk
> > -----END PGP SIGNATURE-----
> >
> >
> > _______________________________________________
> > Tech mailing list
> > Tech at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> >
> >
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070104/7bc3542d/attachment.pgp 

From listen at mjh.name  Sat Jan  6 07:56:46 2007
From: listen at mjh.name (Milan =?ISO-8859-1?B?SG9seuRwZmVs?=)
Date: Sat, 6 Jan 2007 08:56:46 +0100
Subject: [Tech] Node gets more FEC blocks than needed
In-Reply-To: <20070104150321.GG4525@freenetproject.org>
References: <d6149a790612291304s416823e3k9a287164bcad014f@mail.gmail.com>
	<20070104143150.GC4525@freenetproject.org>
	<d6149a790701040640o4dde48afm3f5e3f4891d33b8f@mail.gmail.com>
	<20070104144254.GE4525@freenetproject.org>
	<d6149a790701040648g2869aafdtb4fc970ceb4c19b9@mail.gmail.com>
	<20070104150321.GG4525@freenetproject.org>
Message-ID: <20070106085646.bdd50735.listen@mjh.name>

On Thu, 4 Jan 2007 16:03:22 +0100
Florent Daigni?re (NextGen$) <nextgens at freenetproject.org> wrote:

> * bbackde at googlemail.com <bbackde at googlemail.com> [2007-01-04
> 15:48:09]:
>
> > [...]
> >
> > But the current behavior often leads to downloads that have more
> > than 100% progress *g*
>
> Is it *that* important to have an accurate progress bar ? some blocks
> can take *ages* to be retrived whereas some will come up
> instantaneously... Imho we can't provide a reliable
> percentage/ETA/whatever for small files even if we had detailled stats
> on what's going on.

That's probably very true, but I always wonder whether > 100% means
that there is a bug in fproxy / freenet and it will never finish
because it didn't notice that it was finished already or something.
It's probably a bad idea to say that the download is > 100%, from a
user point of view.

That an ETA will not be that reliable is a different topic if you ask me.

> Btw, haven't you ever experienced the same "bug" on the queue toadlet
> on fproxy ?
> [...]

One of my downloads on fproxy's queue page says it's at 101.2%.  I
think that this is this bug?

Regards,
Milan


--
Milan Holzaepfel <mail(a)mjh(d)name>             <URL:http://mjh.name/>
pub  4096R/C790FC23  EB8E 5E81 81E3 53A9 9B74  B895 5179 54C0 C790 FC23

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 827 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070106/a015ffe4/attachment.pgp 

From freenetwork at web.de  Sat Jan  6 09:56:25 2007
From: freenetwork at web.de (freenetwork at web.de)
Date: Sat, 06 Jan 2007 10:56:25 +0100
Subject: [Tech] Node gets more FEC blocks than needed
In-Reply-To: <20070106085646.bdd50735.listen@mjh.name>
Message-ID: <E1H38Hs-00032G-00@smtp06.web.de>

>> Btw, haven't you ever experienced the same "bug" on the queue toadlet
>> on fproxy ?
>> [...]

>One of my downloads on fproxy's queue page says it's at 101.2%.  I
>think that this is this bug?

Depends.
If it's written in italic then the estimation is not 'finished' and the values vary very much (even being >100%)
If it's written in bold non-italic letters, then the percentages should be accurate (and never >100%, I second that)





From nextgens at freenetproject.org  Sat Jan  6 11:56:13 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re_=28NextGen$=29?=)
Date: Sat, 6 Jan 2007 12:56:13 +0100
Subject: [Tech] Node gets more FEC blocks than needed
In-Reply-To: <20070106085646.bdd50735.listen@mjh.name>
References: <d6149a790612291304s416823e3k9a287164bcad014f@mail.gmail.com>
	<20070104143150.GC4525@freenetproject.org>
	<d6149a790701040640o4dde48afm3f5e3f4891d33b8f@mail.gmail.com>
	<20070104144254.GE4525@freenetproject.org>
	<d6149a790701040648g2869aafdtb4fc970ceb4c19b9@mail.gmail.com>
	<20070104150321.GG4525@freenetproject.org>
	<20070106085646.bdd50735.listen@mjh.name>
Message-ID: <20070106115613.GA4674@freenetproject.org>

* Milan Holz?pfel <listen at mjh.name> [2007-01-06 08:56:46]:

> On Thu, 4 Jan 2007 16:03:22 +0100
> Florent Daigni?re (NextGen$) <nextgens at freenetproject.org> wrote:
> 
> > * bbackde at googlemail.com <bbackde at googlemail.com> [2007-01-04
> > 15:48:09]:
> >
> > > [...]
> > >
> > > But the current behavior often leads to downloads that have more
> > > than 100% progress *g*
> >
> > Is it *that* important to have an accurate progress bar ? some blocks
> > can take *ages* to be retrived whereas some will come up
> > instantaneously... Imho we can't provide a reliable
> > percentage/ETA/whatever for small files even if we had detailled stats
> > on what's going on.
> 
> That's probably very true, but I always wonder whether > 100% means
> that there is a bug in fproxy / freenet and it will never finish
> because it didn't notice that it was finished already or something.

It's not a bug.

> It's probably a bad idea to say that the download is > 100%, from a
> user point of view.

Well, it won't say such a thing if you aren't running the
"AdvancedDarknetMode" :)

See how it's implemented on
http://emu.freenetproject.org/cgi-bin/viewcvs.cgi/trunk/freenet/src/freenet/clients/http/QueueToadlet.java?rev=11427&view=markup

look for "private HTMLNode createProgressCell(" ...

Depending on if advanced mode is on or off, we consider that the total
amount of blocks is either the "required" or the "total number of
inserted blocks".

Both approaches do have problems :
 in non advanced mode, a download can finish at 2/3
 in avanced mode a download can finish (worst case assuming it finishes) at 3/2 !

ATM it's up to FCP client authors to choose one of the above "strategy"
to show the progress.

> 
> That an ETA will not be that reliable is a different topic if you ask me.

it's related :)

> 
> > Btw, haven't you ever experienced the same "bug" on the queue toadlet
> > on fproxy ?
> > [...]
> 
> One of my downloads on fproxy's queue page says it's at 101.2%.  I
> think that this is this bug?

Yes, that one :) As said previously it's not a bug :) but actually a
feature.

> 
> Regards,
> Milan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070106/6c7270bc/attachment.pgp 

From toad at amphibian.dyndns.org  Tue Jan  9 04:44:06 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Tue, 9 Jan 2007 04:44:06 +0000
Subject: [Tech] Build 1010 - MANDATORY SECURITY FIX
Message-ID: <20070109044406.GA28018@amphibian.dyndns.org>

Freenet 0.7 build 1010 is now available. This build fixes a serious
security bug in our encryption code, affecting both link encryption and
encryption of keys. Please upgrade immediately; it is a mandatory build
as of now (meaning it will not connect to older builds). You will
probably have to use the update script manually to do this; sorry for
that, we *will* build an update-over-mandatory mechanism...
Caveats:
- All old KSKs are no longer retrievable.
- You will need to generate a new SSK for your inserts to be encrypted
  properly.
Otherwise the new build is backwards compatible with existing content.
So there is no need for a content reset. PLEASE UPGRADE!
If you are inclined to forward this to slashdot, please wait 24 hours or
so in case I missed any catastrophic bugs. And please listen out for new
builds.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070109/dad6ddde/attachment.pgp 

From toad at amphibian.dyndns.org  Mon Jan 15 18:01:10 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 15 Jan 2007 18:01:10 +0000
Subject: [Tech] Node gets more FEC blocks than needed
In-Reply-To: <20070104143150.GC4525@freenetproject.org>
References: <d6149a790612291304s416823e3k9a287164bcad014f@mail.gmail.com>
	<20070104143150.GC4525@freenetproject.org>
Message-ID: <20070115180110.GL29706@amphibian.dyndns.org>

On Thu, Jan 04, 2007 at 03:31:51PM +0100, Florent Daigni?re (NextGen$) wrote:
> * bbackde at googlemail.com <bbackde at googlemail.com> [2006-12-29 22:04:07]:
> 
> > Following SimpleProgress shows it: some threads in freenet node seems
> > to be started and do not finish if we have all the needed blocks:
> > 
> > SimpleProgress {Total=9606, FatallyFailed=2, FinalizedTotal=true, Failed=287, Su
> > cceeded=6411, Identifier=get-11673399350836773, Required=6404} EndMessage
> > 
> > Its a request with DDA enabled, not in global queue.
> > 
> > Reasons? Fixes? Ideas?
> 
> Well, it's a wholly erroneous interpretation of what "required" means.
> It does mean that the download *can't* complete without that "amount" of
> blocks... but it doesn't mean it will.

It is possible to decode if we have the required number of blocks.
However this is a per segment question, not an overall one. So in
conclusion... one or more segments has got more blocks than it needed
to (maybe because some were already in the store). Unless this goes way
above required, I don't see a problem.
> 
> Ever heard of ForwardErrorCorrection and Hamming distance ?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070115/21211d13/attachment.pgp 

From toad at amphibian.dyndns.org  Mon Jan 15 18:01:45 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 15 Jan 2007 18:01:45 +0000
Subject: [Tech] Node gets more FEC blocks than needed
In-Reply-To: <d6149a790701040648g2869aafdtb4fc970ceb4c19b9@mail.gmail.com>
References: <d6149a790612291304s416823e3k9a287164bcad014f@mail.gmail.com>
	<20070104143150.GC4525@freenetproject.org>
	<d6149a790701040640o4dde48afm3f5e3f4891d33b8f@mail.gmail.com>
	<20070104144254.GE4525@freenetproject.org>
	<d6149a790701040648g2869aafdtb4fc970ceb4c19b9@mail.gmail.com>
Message-ID: <20070115180145.GM29706@amphibian.dyndns.org>

On Thu, Jan 04, 2007 at 03:48:09PM +0100, bbackde at googlemail.com wrote:
> Ok, so I will open a feature request for this issue. Because with the
> informations provided in SimpleProgress a freenet client cannot
> display a correct progress to the user. Either we need more
> informations about the progress of each segment, or the node
> normalizes this by itself and provides a percentage :)
> 
> But the current behavior often leads to downloads that have more than
> 100% progress *g*

Why is that a problem? It's only slightly over 100%.
> 
> On 1/4/07, Florent Daigni?re (NextGen$) <nextgens at freenetproject.org> wrote:
> > * bbackde at googlemail.com <bbackde at googlemail.com> [2007-01-04 15:40:24]:
> >
> > > Of course I heard about FEC, in fact I implemented all this stuff in
> > > the 0.5 part of Frost by myself ;)
> > >
> > > And thats why I thought if the node says X blocks are required, then
> > > this amount of blocks is required to decode the file (no matter if
> > > data or check blocks).
> > > So please tell me what Required means in the SimpleProgress message
> > > then? Or do you mean that there were not enough blocks for one
> > > segment, but more than enough for another segment, is it this?
> >
> > That's probably it ... it's hard to tell without logs.
> >
> > >
> > > On 1/4/07, Florent Daigni?re (NextGen$) <nextgens at freenetproject.org> wrote:
> > > > * bbackde at googlemail.com <bbackde at googlemail.com> [2006-12-29 22:04:07]:
> > > >
> > > > > Following SimpleProgress shows it: some threads in freenet node seems
> > > > > to be started and do not finish if we have all the needed blocks:
> > > > >
> > > > > SimpleProgress {Total=9606, FatallyFailed=2, FinalizedTotal=true, Failed=287, Su
> > > > > cceeded=6411, Identifier=get-11673399350836773, Required=6404} EndMessage
> > > > >
> > > > > Its a request with DDA enabled, not in global queue.
> > > > >
> > > > > Reasons? Fixes? Ideas?
> > > >
> > > > Well, it's a wholly erroneous interpretation of what "required" means.
> > > > It does mean that the download *can't* complete without that "amount" of
> > > > blocks... but it doesn't mean it will.
> > > >
> > > > Ever heard of ForwardErrorCorrection and Hamming distance ?
> >
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.6 (GNU/Linux)
> >
> > iD8DBQFFnRJuU/Z/dHFfxtcRAtxsAJ0b1s3hkAc5U/MVnz5iVBMsI6O2AgCfRwOW
> > En/iX2M3+WV8QIpGwJw2tpk=
> > =B4sk
> > -----END PGP SIGNATURE-----
> >
> >
> > _______________________________________________
> > Tech mailing list
> > Tech at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> >
> >
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070115/9d11ba2a/attachment.pgp 

From toad at amphibian.dyndns.org  Mon Jan 15 18:04:23 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 15 Jan 2007 18:04:23 +0000
Subject: [Tech] Node gets more FEC blocks than needed
In-Reply-To: <20070106115613.GA4674@freenetproject.org>
References: <d6149a790612291304s416823e3k9a287164bcad014f@mail.gmail.com>
	<20070104143150.GC4525@freenetproject.org>
	<d6149a790701040640o4dde48afm3f5e3f4891d33b8f@mail.gmail.com>
	<20070104144254.GE4525@freenetproject.org>
	<d6149a790701040648g2869aafdtb4fc970ceb4c19b9@mail.gmail.com>
	<20070104150321.GG4525@freenetproject.org>
	<20070106085646.bdd50735.listen@mjh.name>
	<20070106115613.GA4674@freenetproject.org>
Message-ID: <20070115180423.GN29706@amphibian.dyndns.org>

On Sat, Jan 06, 2007 at 12:56:13PM +0100, Florent Daigni?re (NextGen$) wrote:
> * Milan Holz?pfel <listen at mjh.name> [2007-01-06 08:56:46]:
> 
> > On Thu, 4 Jan 2007 16:03:22 +0100
> > Florent Daigni?re (NextGen$) <nextgens at freenetproject.org> wrote:
> > 
> > > * bbackde at googlemail.com <bbackde at googlemail.com> [2007-01-04
> > > 15:48:09]:
> > >
> > > > [...]
> > > >
> > > > But the current behavior often leads to downloads that have more
> > > > than 100% progress *g*
> > >
> > > Is it *that* important to have an accurate progress bar ? some blocks
> > > can take *ages* to be retrived whereas some will come up
> > > instantaneously... Imho we can't provide a reliable
> > > percentage/ETA/whatever for small files even if we had detailled stats
> > > on what's going on.
> > 
> > That's probably very true, but I always wonder whether > 100% means
> > that there is a bug in fproxy / freenet and it will never finish
> > because it didn't notice that it was finished already or something.
> 
> It's not a bug.
> 
> > It's probably a bad idea to say that the download is > 100%, from a
> > user point of view.
> 
> Well, it won't say such a thing if you aren't running the
> "AdvancedDarknetMode" :)
> 
> See how it's implemented on
> http://emu.freenetproject.org/cgi-bin/viewcvs.cgi/trunk/freenet/src/freenet/clients/http/QueueToadlet.java?rev=11427&view=markup
> 
> look for "private HTMLNode createProgressCell(" ...
> 
> Depending on if advanced mode is on or off, we consider that the total
> amount of blocks is either the "required" or the "total number of
> inserted blocks".
> 
> Both approaches do have problems :
>  in non advanced mode, a download can finish at 2/3
>  in avanced mode a download can finish (worst case assuming it finishes) at 3/2 !

That's bizarre. I don't see why we shouldn't use # required in both
cases. Having said that we may want to put some logic in to not count
blocks which are unnecessary (within a given segment) towards the
overall total?
> 
> ATM it's up to FCP client authors to choose one of the above "strategy"
> to show the progress.
> 
> > 
> > That an ETA will not be that reliable is a different topic if you ask me.
> 
> it's related :)
> 
> > 
> > > Btw, haven't you ever experienced the same "bug" on the queue toadlet
> > > on fproxy ?
> > > [...]
> > 
> > One of my downloads on fproxy's queue page says it's at 101.2%.  I
> > think that this is this bug?
> 
> Yes, that one :) As said previously it's not a bug :) but actually a
> feature.
> 
> > 
> > Regards,
> > Milan



> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070115/c5a51c13/attachment.pgp 

From toad at amphibian.dyndns.org  Mon Jan 15 18:07:37 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 15 Jan 2007 18:07:37 +0000
Subject: [Tech] EvilDude, Round 2
In-Reply-To: <200701031630.52885.dbkr@freenetproject.org>
References: <200701021554.50318.bo-le@web.de>
	<d6149a790701030619i66d19abaxf4f57e6db63be8c1@mail.gmail.com>
	<200701031630.52885.dbkr@freenetproject.org>
Message-ID: <20070115180737.GO29706@amphibian.dyndns.org>

On Wed, Jan 03, 2007 at 04:30:52PM +0000, Dave Baker wrote:
> I don't really see as that's relevant, since it would be spoofable anyway. 
> Surely the point is setting a limit on the size of a key when it's requested 
> (frosy must do this, surely..?) and limiting the number of redirects (which 
> the ClientGet command doesn't seem to have an option for, but I assume the 
> node has a sensible limit and detects redirect loops).

It doesn't? It should... hmm, it doesn't. Filed bug #1065.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070115/fcc8dd3d/attachment.pgp 

From nextgens at freenetproject.org  Mon Jan 15 18:08:25 2007
From: nextgens at freenetproject.org (Florent =?iso-8859-1?Q?Daigni=E8re_=28NextGen$=29?=)
Date: Mon, 15 Jan 2007 19:08:25 +0100
Subject: [Tech] Node gets more FEC blocks than needed
In-Reply-To: <20070115180423.GN29706@amphibian.dyndns.org>
References: <d6149a790612291304s416823e3k9a287164bcad014f@mail.gmail.com>
	<20070104143150.GC4525@freenetproject.org>
	<d6149a790701040640o4dde48afm3f5e3f4891d33b8f@mail.gmail.com>
	<20070104144254.GE4525@freenetproject.org>
	<d6149a790701040648g2869aafdtb4fc970ceb4c19b9@mail.gmail.com>
	<20070104150321.GG4525@freenetproject.org>
	<20070106085646.bdd50735.listen@mjh.name>
	<20070106115613.GA4674@freenetproject.org>
	<20070115180423.GN29706@amphibian.dyndns.org>
Message-ID: <20070115180803.GC4263@freenetproject.org>

* Matthew Toseland <toad at amphibian.dyndns.org> [2007-01-15 18:04:23]:

> On Sat, Jan 06, 2007 at 12:56:13PM +0100, Florent Daigni?re (NextGen$) wrote:
> > * Milan Holz?pfel <listen at mjh.name> [2007-01-06 08:56:46]:
> > 
> > > On Thu, 4 Jan 2007 16:03:22 +0100
> > > Florent Daigni?re (NextGen$) <nextgens at freenetproject.org> wrote:
> > > 
> > > > * bbackde at googlemail.com <bbackde at googlemail.com> [2007-01-04
> > > > 15:48:09]:
> > > >
> > > > > [...]
> > > > >
> > > > > But the current behavior often leads to downloads that have more
> > > > > than 100% progress *g*
> > > >
> > > > Is it *that* important to have an accurate progress bar ? some blocks
> > > > can take *ages* to be retrived whereas some will come up
> > > > instantaneously... Imho we can't provide a reliable
> > > > percentage/ETA/whatever for small files even if we had detailled stats
> > > > on what's going on.
> > > 
> > > That's probably very true, but I always wonder whether > 100% means
> > > that there is a bug in fproxy / freenet and it will never finish
> > > because it didn't notice that it was finished already or something.
> > 
> > It's not a bug.
> > 
> > > It's probably a bad idea to say that the download is > 100%, from a
> > > user point of view.
> > 
> > Well, it won't say such a thing if you aren't running the
> > "AdvancedDarknetMode" :)
> > 
> > See how it's implemented on
> > http://emu.freenetproject.org/cgi-bin/viewcvs.cgi/trunk/freenet/src/freenet/clients/http/QueueToadlet.java?rev=11427&view=markup
> > 
> > look for "private HTMLNode createProgressCell(" ...
> > 
> > Depending on if advanced mode is on or off, we consider that the total
> > amount of blocks is either the "required" or the "total number of
> > inserted blocks".
> > 
> > Both approaches do have problems :
> >  in non advanced mode, a download can finish at 2/3
> >  in avanced mode a download can finish (worst case assuming it finishes) at 3/2 !
> 
> That's bizarre. I don't see why we shouldn't use # required in both
> cases. Having said that we may want to put some logic in to not count
> blocks which are unnecessary (within a given segment) towards the
> overall total?

I don't regard that as an important matter. I suggest you fill in a
ticket on mantis under the category wishlist ;)

NextGen$
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070115/a170eb87/attachment.pgp 

From toad at amphibian.dyndns.org  Mon Jan 15 18:08:29 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 15 Jan 2007 18:08:29 +0000
Subject: [Tech] EvilDude, Round 2
In-Reply-To: <d6149a790701030905g3ff470c0je195b987dc0de3bf@mail.gmail.com>
References: <200701021554.50318.bo-le@web.de>
	<d6149a790701030619i66d19abaxf4f57e6db63be8c1@mail.gmail.com>
	<200701031630.52885.dbkr@freenetproject.org>
	<d6149a790701030905g3ff470c0je195b987dc0de3bf@mail.gmail.com>
Message-ID: <20070115180829.GP29706@amphibian.dyndns.org>

On Wed, Jan 03, 2007 at 06:05:51PM +0100, bbackde at googlemail.com wrote:
> It would help to prevent the usage of specific keys that were inserted
> as a redirect target of a KSK. If a client could determine that the
> inserted KSK was inserted as redirect to a specific key, we could
> easily drop and ignore this KSK. We would only allow KSKs that were
> inserted as data. This prevents attackers from inserting alot of KSK
> redirects (which is very fast, compared to the insert of a KSK with
> the data) in a short time, and it prevents spreading of specific CHK
> keys over KSKs.
> So I see some advantages there. It should'nt not too hard IF the node
> knows when a KSK was inserted as redirect to a specific CHK. If not I
> think we will have some more problems with KSK attacks than under 0.5.
> On 0.5 this kind of attack is not possible such easily...

"Inserted as data" ? Meaning what exactly, from a node's point of view?
Any data which does not fit in a 1kB KSK will be inserted as data and
redirected to.
> 
> On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > I don't really see as that's relevant, since it would be spoofable anyway.
> > Surely the point is setting a limit on the size of a key when it's requested
> > (frosy must do this, surely..?) and limiting the number of redirects (which
> > the ClientGet command doesn't seem to have an option for, but I assume the
> > node has a sensible limit and detects redirect loops).
> >
> > This would prevent the second attack. I don't really see that the first is a
> > problem anyway, since the risk of downloading arbitrary content is implicit
> > when using frost. You could potentially only allow keys that are non-redirect
> > KSKs (thus limiting the size of a frost post to 1k, which may or may not be
> > desirable). That would prevent it, but as I say, I don't really see it as a
> > problem assuming that there's a filesize limit.
> >
> >
> > Dave
> >
> >
> > On Wednesday 03 January 2007 14:19, bbackde at googlemail.com wrote:
> > > I think he means that a client should be able to determine if a
> > > KSK/SSK was inserted as a manual redirect to another key, or if the
> > > KSK/SSK was inserted with data and a redirect to the content was
> > > automatically created by the node.
> > >
> > > Is it possible to separate this in the node? This would be a great and
> > > simple solution.
> > >
> > > On 1/2/07, bo-le at web.de <bo-le at web.de> wrote:
> > > > hi,
> > > >
> > > > just in the frost-board 'frost'
> > > > ----- EvilDude ----- 2007.01.02 - 07:22:29GMT -----
> > > >
> > > > If I wanted to download a key but it was rare, could I increase my
> > > > chances of getting it if I inserted a KSK redirecting to it, forcing
> > > > every frost user to download that file?
> > > >
> > > > Similarly, could I create a denial of service attack by inserting a bunch
> > > > of redirects to all the large files I can find? I'm not going to attempt
> > > > it this time around, I'm just wondering.
> > > >
> > > > -----
> > > >
> > > > The fcp client should be able to detect the redirects if wanted.
> > > >
> > > > My suggestion: a new progress, that show the redirects.
> > > >
> > > >    static final int showRedirects = 128;
> > > >
> > > >    verbose=verbose | showRedirects
> > > >
> > > >
> > > > Now the ClientGet returns a new progress message
> > > >
> > > >    FollowingRedirect
> > > >    Identifier=hello
> > > >    RequestedURI=freenet:KSK at something-1-2
> > > >    RedirectTargetURI=freenet:CHK at jdncdj,hdcbd,a8
> > > >    EndMessage
> > > >
> > > > Now the client app can decide how to handle it.
> > > >
> > > > thanks.
> > > >
> > > > --
> > > > Mfg
> > > > saces
> > >
> > > _______________________________________________
> > > Tech mailing list
> > > Tech at freenetproject.org
> > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > _______________________________________________
> > Tech mailing list
> > Tech at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> >
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070115/80977509/attachment.pgp 

From toad at amphibian.dyndns.org  Mon Jan 15 18:12:05 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 15 Jan 2007 18:12:05 +0000
Subject: [Tech] EvilDude, Round 2
In-Reply-To: <d6149a790701031142r4d747171p6599b31c50f1c9a2@mail.gmail.com>
References: <200701021554.50318.bo-le@web.de>
	<200701031630.52885.dbkr@freenetproject.org>
	<d6149a790701030905g3ff470c0je195b987dc0de3bf@mail.gmail.com>
	<200701031732.31334.dbkr@freenetproject.org>
	<d6149a790701031142r4d747171p6599b31c50f1c9a2@mail.gmail.com>
Message-ID: <20070115181205.GQ29706@amphibian.dyndns.org>

On Wed, Jan 03, 2007 at 08:42:18PM +0100, bbackde at googlemail.com wrote:
> What is wrong with the described advantages?
> 
> Scenario:
> 
> board_1 and board_2 exists. An attacker tries to spam them to make the
> retrieval of messages nearly impossible.
> 
> On board_1 he inserts 100 KSK keys with redirects to CHK keys that
> were randomly created and point to no existing content. The insert
> completes _very_ quickly.
> 
> On board_2 he inserts 100 KSKs with random data behind it. The insert
> takes a _much_ longer time to complete.
> 
> Now a Frost client updates this board.
> 
> On board_1 the client requests the 1st KSK. The node redirects
> transparently to the CHK in the KSK redirect, and needs a _long_ time
> until he gives up and reports code=28 to the client. The client does
> not know if the request failed because the CHK does'nt exist at all,
> or if the CHK could not be retrieved this time, but later. The client
> steps over all 100 KSK keys (each one failing after a long search
> time) and after this the client have to request all keys again during
> next update, because some CHK might be retrievable now.

The client will get code 28 errors, and can limit the number of retries
it makes for these messages.
> 
> On board_2 the client also requests the 1st KSK key. The node
> redirects to the CHK key, and after a _shorter_ time the request is
> successful or not. If it _is_ successful the client marks the slot
> used (tried never again) and inspects the retrieved data, which are
> random and therefore invalid. If the request was _not_ successful, the
> client gets code=28, but this time he knows that the redirect points
> to valid CHK key! So a retry will be successful later, and its worth
> to retry.
> 
> So why does the node even allow to insert a redirect to not-existing
> data? He should check if the data is retrievable from the inserting
> node. I can't understand why someone should insert a redirect to
> content that cannot be retrieved by himself.

An attacker can hack the node easily enough. Also there are lots of
people on Frost who demand that we allow the node to produce a CHK way
before the data has actually been inserted, i.e. to insert the top bit
first. This is exactly the same as the above.
> 
> I think an indication if the CHK is valid would be better than
> without. 

The node cannot know whether the CHK is valid until it has fetched it.

> Even with patched nodes, each insert node on the htl chain
> could check if the CHK is known, and it is known if at least e.g. 2
> nodes (on the chain or neighbors) say it is valid.

Not possible, because the data - even metadata - is encrypted.
> 
> I don't know if my points are valid, but I want some explanations that
> help me to understand this. A simple 'its not worth to think about',
> without to provide a working circumvention for this, its not enough
> for me.
> 
> And why is there no type of key in 0.7 that is designed for
> applications like Frost that provide a global messaging feature? There
> are many improvements for freesites (e.g. USK) and easy filetransfer
> (although to less detailed for clients, currently).

What would you suggest? Start a new thread please.
> 
> On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > > If a client could determine that the inserted KSK was inserted
> > > as redirect to a specific key,
> >
> > That's the problem though - I don't think that's possible. Even if it were
> > marked somehow on insertion, it would only require some minor hackery of the
> > node's source to make a user-created redirect look like a node-created
> > redirect.
> >
> > Again though, I'm still not convinced it's worth worrying about.
> >
> >
> > Dave
> >
> > On Wednesday 03 January 2007 17:05, bbackde at googlemail.com wrote:
> > > It would help to prevent the usage of specific keys that were inserted
> > > as a redirect target of a KSK. If a client could determine that the
> > > inserted KSK was inserted as redirect to a specific key, we could
> > > easily drop and ignore this KSK. We would only allow KSKs that were
> > > inserted as data. This prevents attackers from inserting alot of KSK
> > > redirects (which is very fast, compared to the insert of a KSK with
> > > the data) in a short time, and it prevents spreading of specific CHK
> > > keys over KSKs.
> > > So I see some advantages there. It should'nt not too hard IF the node
> > > knows when a KSK was inserted as redirect to a specific CHK. If not I
> > > think we will have some more problems with KSK attacks than under 0.5.
> > > On 0.5 this kind of attack is not possible such easily...
> > >
> > > On 1/3/07, Dave Baker <dbkr at freenetproject.org> wrote:
> > > > I don't really see as that's relevant, since it would be spoofable
> > > > anyway. Surely the point is setting a limit on the size of a key when
> > > > it's requested (frosy must do this, surely..?) and limiting the number of
> > > > redirects (which the ClientGet command doesn't seem to have an option
> > > > for, but I assume the node has a sensible limit and detects redirect
> > > > loops).
> > > >
> > > > This would prevent the second attack. I don't really see that the first
> > > > is a problem anyway, since the risk of downloading arbitrary content is
> > > > implicit when using frost. You could potentially only allow keys that are
> > > > non-redirect KSKs (thus limiting the size of a frost post to 1k, which
> > > > may or may not be desirable). That would prevent it, but as I say, I
> > > > don't really see it as a problem assuming that there's a filesize limit.
> > > >
> > > >
> > > > Dave
> > > >
> > > > On Wednesday 03 January 2007 14:19, bbackde at googlemail.com wrote:
> > > > > I think he means that a client should be able to determine if a
> > > > > KSK/SSK was inserted as a manual redirect to another key, or if the
> > > > > KSK/SSK was inserted with data and a redirect to the content was
> > > > > automatically created by the node.
> > > > >
> > > > > Is it possible to separate this in the node? This would be a great and
> > > > > simple solution.
> > > > >
> > > > > On 1/2/07, bo-le at web.de <bo-le at web.de> wrote:
> > > > > > hi,
> > > > > >
> > > > > > just in the frost-board 'frost'
> > > > > > ----- EvilDude ----- 2007.01.02 - 07:22:29GMT -----
> > > > > >
> > > > > > If I wanted to download a key but it was rare, could I increase my
> > > > > > chances of getting it if I inserted a KSK redirecting to it, forcing
> > > > > > every frost user to download that file?
> > > > > >
> > > > > > Similarly, could I create a denial of service attack by inserting a
> > > > > > bunch of redirects to all the large files I can find? I'm not going
> > > > > > to attempt it this time around, I'm just wondering.
> > > > > >
> > > > > > -----
> > > > > >
> > > > > > The fcp client should be able to detect the redirects if wanted.
> > > > > >
> > > > > > My suggestion: a new progress, that show the redirects.
> > > > > >
> > > > > >    static final int showRedirects = 128;
> > > > > >
> > > > > >    verbose=verbose | showRedirects
> > > > > >
> > > > > >
> > > > > > Now the ClientGet returns a new progress message
> > > > > >
> > > > > >    FollowingRedirect
> > > > > >    Identifier=hello
> > > > > >    RequestedURI=freenet:KSK at something-1-2
> > > > > >    RedirectTargetURI=freenet:CHK at jdncdj,hdcbd,a8
> > > > > >    EndMessage
> > > > > >
> > > > > > Now the client app can decide how to handle it.
> > > > > >
> > > > > > thanks.
> > > > > >
> > > > > > --
> > > > > > Mfg
> > > > > > saces
> > > > >
> > > > > _______________________________________________
> > > > > Tech mailing list
> > > > > Tech at freenetproject.org
> > > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > > >
> > > > _______________________________________________
> > > > Tech mailing list
> > > > Tech at freenetproject.org
> > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > >
> > > _______________________________________________
> > > Tech mailing list
> > > Tech at freenetproject.org
> > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> > _______________________________________________
> > Tech mailing list
> > Tech at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> >
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070115/814cd1f0/attachment.pgp 

From toad at amphibian.dyndns.org  Mon Jan 15 18:14:48 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 15 Jan 2007 18:14:48 +0000
Subject: [Tech] Global messaging (was: EvilDude, Round 2)
In-Reply-To: <d6149a790701040121l7ad0c9bbja3532a699dca887d@mail.gmail.com>
References: <200701021554.50318.bo-le@web.de>
	<200701032112.14882.dbkr@freenetproject.org>
	<d6149a790701031332u5284e85ft1373a3014e033949@mail.gmail.com>
	<200701040858.57178.dbkr@freenetproject.org>
	<d6149a790701040121l7ad0c9bbja3532a699dca887d@mail.gmail.com>
Message-ID: <20070115181448.GR29706@amphibian.dyndns.org>

On Thu, Jan 04, 2007 at 10:21:54AM +0100, bbackde at googlemail.com wrote:
> > Sorry - I still don't understand. A key that's readable and writable to
> > everyone - is that not a KSK?
> 
> Yes, thats KSK. But you see the problems we have with the 'new' KSK
> keys which are 1KB in size and provide transparent redirects to other
> keys? KSKs on 0.5 were great for Frosts messaging (32KB size, no
> transparent redirects), but the 'new' KSKs introduced the problems we
> talk about here (new kind of attacks, ...).

Inserting random garbage has always been a viable attack against Frost.
All you have to do is set an appropriate maximum data size.
> 
> The devs tried to make it easier for clients, but now some clients
> have serious problems.
> It could help to introduce a new type of freenet key, like KSK but
> without redirects and 32KB maximum size (like CHK). This would be
> perfect for Frost like clients.

How does that differ to just setting max size = 32kB in Frost? Sure you
have the code=28 issues, but so what? You can handle them - if you get a
code 28, there's a possibility that it's an attack, so you don't retry
it quite so aggressively.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070115/92ce3710/attachment.pgp 

From bbackde at googlemail.com  Mon Jan 15 18:32:30 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Mon, 15 Jan 2007 19:32:30 +0100
Subject: [Tech] EvilDude, Round 2
In-Reply-To: <20070115180737.GO29706@amphibian.dyndns.org>
References: <200701021554.50318.bo-le@web.de>
	<d6149a790701030619i66d19abaxf4f57e6db63be8c1@mail.gmail.com>
	<200701031630.52885.dbkr@freenetproject.org>
	<20070115180737.GO29706@amphibian.dyndns.org>
Message-ID: <d6149a790701151032p6cf48f9fuc321fdd1a458c235@mail.gmail.com>

A MaxRedirects limit setting would be great and a solution for what I
meant. You often wrote about this, and I assumed you know that there
is no such setting :)

On 1/15/07, Matthew Toseland <toad at amphibian.dyndns.org> wrote:
> On Wed, Jan 03, 2007 at 04:30:52PM +0000, Dave Baker wrote:
> > I don't really see as that's relevant, since it would be spoofable anyway.
> > Surely the point is setting a limit on the size of a key when it's requested
> > (frosy must do this, surely..?) and limiting the number of redirects (which
> > the ClientGet command doesn't seem to have an option for, but I assume the
> > node has a sensible limit and detects redirect loops).
>
> It doesn't? It should... hmm, it doesn't. Filed bug #1065.
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFFq8LpA9rUluQ9pFARAisOAKChxYgzyfOlu4TqyTJ6yxQ2383HswCgrFR1
> t5T3ztO6eLyUdDqKkFp55QE=
> =Gqwf
> -----END PGP SIGNATURE-----
>
>
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
>
>


From toad at amphibian.dyndns.org  Mon Jan 15 18:51:52 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Mon, 15 Jan 2007 18:51:52 +0000
Subject: [Tech] EvilDude, Round 2
In-Reply-To: <d6149a790701151032p6cf48f9fuc321fdd1a458c235@mail.gmail.com>
References: <200701021554.50318.bo-le@web.de>
	<d6149a790701030619i66d19abaxf4f57e6db63be8c1@mail.gmail.com>
	<200701031630.52885.dbkr@freenetproject.org>
	<20070115180737.GO29706@amphibian.dyndns.org>
	<d6149a790701151032p6cf48f9fuc321fdd1a458c235@mail.gmail.com>
Message-ID: <20070115185152.GA24500@amphibian.dyndns.org>

On Mon, Jan 15, 2007 at 07:32:30PM +0100, bbackde at googlemail.com wrote:
> A MaxRedirects limit setting would be great and a solution for what I
> meant. You often wrote about this, and I assumed you know that there
> is no such setting :)

Hmmm ok. But I'm not sure it would solve the problem. It's quite
possible to insert a big file as a KSK, and "forget" to insert anything
except for the top level (the KSK itself, which would be a splitfile
header pointing to a bigger splitfile).
> 
> On 1/15/07, Matthew Toseland <toad at amphibian.dyndns.org> wrote:
> > On Wed, Jan 03, 2007 at 04:30:52PM +0000, Dave Baker wrote:
> > > I don't really see as that's relevant, since it would be spoofable anyway.
> > > Surely the point is setting a limit on the size of a key when it's requested
> > > (frosy must do this, surely..?) and limiting the number of redirects (which
> > > the ClientGet command doesn't seem to have an option for, but I assume the
> > > node has a sensible limit and detects redirect loops).
> >
> > It doesn't? It should... hmm, it doesn't. Filed bug #1065.
> >
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.6 (GNU/Linux)
> >
> > iD8DBQFFq8LpA9rUluQ9pFARAisOAKChxYgzyfOlu4TqyTJ6yxQ2383HswCgrFR1
> > t5T3ztO6eLyUdDqKkFp55QE=
> > =Gqwf
> > -----END PGP SIGNATURE-----
> >
> >
> > _______________________________________________
> > Tech mailing list
> > Tech at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> >
> >
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070115/a5604e1d/attachment.pgp 

From bbackde at googlemail.com  Mon Jan 15 19:52:32 2007
From: bbackde at googlemail.com (bbackde at googlemail.com)
Date: Mon, 15 Jan 2007 20:52:32 +0100
Subject: [Tech] Global messaging (was: EvilDude, Round 2)
In-Reply-To: <20070115181448.GR29706@amphibian.dyndns.org>
References: <200701021554.50318.bo-le@web.de>
	<200701032112.14882.dbkr@freenetproject.org>
	<d6149a790701031332u5284e85ft1373a3014e033949@mail.gmail.com>
	<200701040858.57178.dbkr@freenetproject.org>
	<d6149a790701040121l7ad0c9bbja3532a699dca887d@mail.gmail.com>
	<20070115181448.GR29706@amphibian.dyndns.org>
Message-ID: <d6149a790701151152g425f30b9u65ca419cab338e29@mail.gmail.com>

> so you don't retry it quite so aggressively.

No problem! But it is harder than on 0.5 and more complicated for the
clients. I only meant 32KB would be great.

On 1/15/07, Matthew Toseland <toad at amphibian.dyndns.org> wrote:
> On Thu, Jan 04, 2007 at 10:21:54AM +0100, bbackde at googlemail.com wrote:
> > > Sorry - I still don't understand. A key that's readable and writable to
> > > everyone - is that not a KSK?
> >
> > Yes, thats KSK. But you see the problems we have with the 'new' KSK
> > keys which are 1KB in size and provide transparent redirects to other
> > keys? KSKs on 0.5 were great for Frosts messaging (32KB size, no
> > transparent redirects), but the 'new' KSKs introduced the problems we
> > talk about here (new kind of attacks, ...).
>
> Inserting random garbage has always been a viable attack against Frost.
> All you have to do is set an appropriate maximum data size.
> >
> > The devs tried to make it easier for clients, but now some clients
> > have serious problems.
> > It could help to introduce a new type of freenet key, like KSK but
> > without redirects and 32KB maximum size (like CHK). This would be
> > perfect for Frost like clients.
>
> How does that differ to just setting max size = 32kB in Frost? Sure you
> have the code=28 issues, but so what? You can handle them - if you get a
> code 28, there's a possibility that it's an attack, so you don't retry
> it quite so aggressively.
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFFq8SYA9rUluQ9pFARAlVgAJ9ow4ZDEBQ61gbg78pxl7Q0TCaF6ACgu5hI
> TsAlHclMI/TYtzpjtN9NGVk=
> =qUO1
> -----END PGP SIGNATURE-----
>
>
> _______________________________________________
> Tech mailing list
> Tech at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech
>
>


From toad at amphibian.dyndns.org  Tue Jan 30 16:54:41 2007
From: toad at amphibian.dyndns.org (Matthew Toseland)
Date: Tue, 30 Jan 2007 16:54:41 +0000
Subject: [Tech] BDB uses far more memory than it's supposed to; ideas?
Message-ID: <20070130165441.GA24058@amphibian.dyndns.org>

After various reports of OOMs, and after high CPU usage which might have
been related to excessive GCing, I did some profiling.

Having analysed the first 70% of memory on the dump, with a 100MB
overall limit, the Berkeley DB Java Edition that we use for the database
uses at least 54.5MB of RAM. databaseMaxMemory is not overridden in the
config, and therefore should be at the default value of 20MB.

The documentation says:
"Note that the cache does not include transient objects created by the JE
library, such as cursors, locks and transactions."

It is possible that the cache only refers to the first stack trace,
which is 17MB. However there are many other traces which appear to be
dealing with the database log (BDB is a log-structured database), many
of them rather large - for example the next 3 traces of 10MB, 5MB and
5MB respectively.

Any ideas? We can:
- Just ignore it. We may have to increase the default memory limit. But
  anecdotally OOMs happen when you have lots of queued requests.
- Use another database. BDB has been fairly unreliable, hence the code
  in the store to reconstruct the database (store index) from the store
  file by deleting the database and parsing each key.
- Given the second item, if we had a reliable database we could store
  queued requests in it, thus limiting the overall memory usage
  regardless of the size of the request queue. But that would be a
  significant amount of work even with a database.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://emu.freenetproject.org/pipermail/tech/attachments/20070130/c7977b15/attachment.pgp