[freenet-dev] Status update

[Matthew Toseland]

STATE OF THE FREENET 21/09/07
=====================
=====================

GSOC
====

The new spider and librarian are more or less working, however there appears 
to be a problem with inserting the resulting indexes. Once we are sure that 
this works, and our anonymous volunteer index maintainers have successfully 
inserted some indexes, we will bundle the XMLLibrarian so that users have a 
freesite-searching capability out of the box. Swati said she'd be interested 
in continuing to work with Freenet but so far hasn't done so.

Vive's work has been very helpful, see clustering.

I am told that the Echo blogging plugin is almost functional, but it has some 
major issues to resolve relating to external dependancies. It is being 
actively worked on by Mc2`/fred.

The new link layer encryption, JFK, is not ready yet, but it is being actively 
worked on between nextgens and kryptos.

The new C++ FCP library is not ready yet. I haven't seen any trace of its 
author mkolar recently.

Many unit tests have been written by sback and they have revealed a number of 
low-level bugs.

CLUSTERING ETC
===========

The periodically resetting nodes' locations appears to have worked, or some 
external factor has changed; either way it looks like the excessive 
clustering of node locations previously observed is not a problem at the 
moment.

NEW UNINSTALL SURVEY
===============

We have a new post-uninstall survey set up on emu. So far the results have 
been rather odd, with more people saying they can't get Freenet working at 
all than saying they can't be bothered to exchange noderefs. It is likely 
that there are serious bugs in the installer behind this, this is being 
worked on.

OPENNET
=======

Obviously everyone wants to know how opennet is going, when they can stop 
exchanging noderefs...

Path folding is implemented and more or less working. Universal Plug and Play 
automatic port forwarding for the opennet port as well as the darknet port, 
telling the user to forward both ports etc are also done. There are several 
major challenges remaining:

1) How to restore service as quickly as possible after downtime, preferably 
without reannouncing.

Some progress has been made here. As of recent trunk SVN, the node will 
remember the last 50 opennet nodes it has dropped, and if they reconnect 
(assuming they manage to get through the NAT, which is possible with UP&P or 
manual port forwarding on maybe 40% of nodes), if there is a free slot in the 
opennet connection table, it will accept the connection. Any other ideas, 
apart from reannouncement, which IMHO is a last resort (see below), are 
welcome.

2) Security issues with the opennet destination noderef messages.

Currently the destination node reference is sent as a single unpadded message. 
It's big enough that it will always be a packet on its own. Therefore it can 
conceivably be used to trace traffic to its origin or destination, if you 
have global traffic analysis capabilities (anyone who does probably already 
knows this so don't panic!). Anyway, this needs to be fixed in the not too 
distant future. The current plan is to pad the noderef to a fixed size and 
use a block data transfer same as we do for CHKs.

3) Identifying potential seednodes.

We need to be able to identify that a node really is directly connected to the 
internet and can receive connections from anywhere. The obvious way to do 
this is to send a request out a few hops random routed on opennet, which will 
then send a single packet to our IP address. If we receive it, we're "open". 
This can of course be abused for a denial of service attack, or used to 
harvest the network by broadcasting too many such messages. So we will 
throttle it to a maximum of 1 message per connection per < some arbitrary 
time period >. Once we know that the node *could* be a seednode, we can ask 
the user whether to register with the central seednodes list.

4) Anonymous connect.

For opennet announcement to work, it must be possible to connect to a node 
which doesn't know you. At the moment our connection setup protocol doesn't 
support this. We will need a new variant on it: this would have to have outer 
wrapper encryption based on the identity of the responder rather than of 
both, and it would have to exchange crypto information early on. I had hoped 
that JFK would be finished by the time we needed this; this now seems 
unlikely, but it shouldn't be a lot of work. Once we have this we can set up 
alternative connection setup mechanisms (for open nodes) such as one-time 
invites.

5) Opennet announcements.

Once we have the prerequisites above, we can implement the actual opennet 
announcement protocol. It is vital that this works (it never did on 0.5), and 
its goals are somewhat different to those of the 0.5 version. In particular, 
it is possible for the announcee to pick his eventual location, but in 
exchange he gets a good set of specialised peers close to the target (and 
some a bit further away too).

6) Securing announcements.

I would like to introduce a CAPTCHA for announcements, because they are 
potentially a very powerful tool for harvesting noderefs.

PREMIX ROUTING
==============

It is very troubling to me, and no doubt to many other Freenet contributors, 
that Freenet does not provide anywhere near the level of request security 
that people initially assume it does. Over the years a lot of effort has been 
exerted looking for a quick fix, but the only practical and effective 
solution remains some form of onion routing as an initial anonymising layer. 
However, IMHO we have enough information to implement this in the not too 
distant future, possibly even before 0.7.0:
- Public topology: each node should maintain its own view of the local network 
topology out to N hops. It will know the connections and pubkeys of the nodes 
within that range. Any changes will be broadcast in real time, signed and 
dated by the keys of the nodes involved, and when a node connects it will ask 
each of its peers for the topology out to N-1 hops.
- Trust ranking: We can use a very simple trust ranking algorithm since we 
have a single start point: our own node. We give each node we are connected 
to 1/n trust, where n is the number of nodes we are connected to. For each of 
these nodes, we give 1/n/p, where p is the number of nodes that node is 
connected to, and so on. We can therefore assign a trust value to each node. 
- Nodes with a trust value above some arbitrary or adaptive threshold are 
considered worthy for premix routing.
- We will take some measures to ensure that we cannot be traced by the number 
of nodes we choose (and therefore the proportion of a splitfile each node 
receives). We will probably have a fixed maximum number of nodes to use for 
premix routing (MAX_PREMIX_NODES); we can either take the top 
MAX_PREMIX_NODES nodes by trust rank, or choose randomly among the nodes 
which meet a specific threshold.
- We can now choose a random node on which to start each request (each valid 
node must have equal probability), and a path to reach that node. We can wrap 
the request in a mixmaster-style onion so that only the destination node can 
read the requested key.
- Note that the above will be somewhat ineffective on opennet; opennet can use 
a different, broader peer selection strategy (later on!).

FROST
=====

The spammer has moved on to posting his ALICE-generated drivel as Anonymous as 
well as random identities. This continues to annoy everyone, and probably to 
cause a significant number of newbies to not use Frost. There is no sign that 
I can see of the drastic changes needed to make Frost safe in a hostile 
environment, but then I'm not a Frost dev... and the node side changes which 
will be needed to support some of the new mechanisms (Ultra-Lightweight 
Passive Requests) will not be implemented until after opennet is ready.

PROBE REQUESTS
===========

Probe requests have been broken recently, 1064 should fix this. The purpose of 
probe requests is to give us an idea of the network topology and size, and to 
test out new routing/backtracking algorithms.

EVERYTHING ELSE
===========

For most of the remaining issues, have a look at the bug tracker:
https://bugs.freenetproject.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://emu.freenetproject.org/pipermail/devl/attachments/20070922/20eef9b2/attachment.pgp