projects/en/hidserv.wml (3c1c18594) - tor-webwml.git

hidserv.wml

## translation metadata
# Revision: $Revision$
# Translation-Priority: 3-low

#include "head.wmi" TITLE="NLnet Project: Speed Up Tor Hidden Services"

<h2>NLnet Project: Speed Up Tor Hidden Services</h2>
<hr />

Tor Hidden Services allow users to set up anonymous information services,
like websites, that can only be accessed through the Tor network and are
protected against identification of the host that runs the services.
The most critical limitations of Tor Hidden Services are the time it takes
until a Hidden Service is registered in the network and the latency of
contact establishment when accessed by a user.
Due to design issues in the original Tor protocol,
the connection to a new Hidden Service can take several minutes, which
leads most users to give up before the connection has been established.
Using Tor Hidden Services for direct interactive user-to-user
communication (e.g. messaging) is nearly impossible due to the high latency
of Hidden Service circuit setup.

This project aims at speeding up Tor Hidden Services by improving the way
Tor circuits are set up between the user and the Hidden Service as well as
the way a Hidden Service is registered in the Tor network.
In a first step precise diagnostics of the behavior of the Hidden Services
in lab setups and real world situations will be conducted to find the
root causes of the bad timing effects.
Based on these diagnostics, optimization strategies will be designed and
verified for unwanted implications for the security and anonymity of the
Tor network.
The most promising optimizations will then be implemented to achieve a
notable improvement for the users. Precise success metrics will be
developed in the diagnostics phase, after it becomes clear where the time
is lost and what improvements are realistic.
The ultimate goal is to have the Hidden Services protocol change production
ready and propagated to the Tor users within a timeframe of less than
12 months.

This project is generously funded by:

<table width="100%" border="0" cellspacing="0" cellpadding="3">
<thead>
<tr>
<th><big>Project</big></th>
<th><big>Due Date</big></th>
</tr>
</thead>

<tr bgcolor="#e5e5e5">
 <td>
 Deliverable A: Analysis, measurements and problem
 clarification 
 As Tor Hidden Services have not been actively developed
 further in the last year or so of Tor development, certain aspects of
 the problems are under-diagnosed. To identify the precise sources of
 latency and time loss, an extensive analysis of the deeper reasons for
 them needs to be conducted. Deliverable A will require about one month
 of work. The results of the analysis will influence the design
 decisions to be taken in Deliverable B.
 </td>
 <td>
 June 15, 2008
 </td>
</tr>

<tr>
 <td>
 Deliverable B: Design and evaluation of the necessary
 changes 
 The changes to Tor Hidden Services will affect core
 functionality of the protocol and therefore require a careful
 evaluation of possible repercussions for the security and anonymity. A
 two-month period is planned for the design and evaluation phase, which
 concludes with an extensive peer review.
 </td>
 <td>
 August 15, 2008
 </td>
</tr>

<tr bgcolor="#e5e5e5">
 <td>
 Deliverable C: Implementation 
 After design, evaluation and peer review the modifications
 need to be implemented and integrated with the current Tor code base.
 The actual implementation of the necessary changes will take
 approximately two months.
 </td>
 <td>
 October 15, 2008
 </td>
</tr>

<tr>
 <td>
 Deliverable D: Implementation and test of the change up to
 release state 
 The modification is highly critical to the security and
 anonymity of the Tor network, it requires extensive testing and
 debugging in laboratory and real life conditions. A period of three
 months is projected for testing and debugging, where the responsible
 developer is committed to the testing effort with 1/3 of its time. Part
 of the testing phase will be a public beta period.
 </td>
 <td>
 January 15, 2009
 </td>
</tr>

<tr bgcolor="#e5e5e5">
 <td>
 Deliverable E: Rollout 
 The actual rollout to the Tor server network will be
 conducted in sync with the regular Tor release schedule. As this
 schedule is dependent on a number of external factors, like the
 completion of other software projects that should go into the same
 release, the actual release time and the time until this release has
 been accepted and installed by most Tor server operators can vary. From
 experience a period of three to four months can be
 expected.
 </td>
 <td>
 May 15, 2009
 </td>
</tr>
</table>

<a id="Reports"></a>
<h2><a class="anchor" href="#Reports">Monthly Status Reports</a></h2>

There will be in total eight monthly status reports beginning with the
first deliverable on June 15, 2008 and ending with completion of
implementation and testing work on January 15, 2009.

<table width="100%" border="0" cellspacing="0" cellpadding="3">
<thead>
<tr>
<th><big>Month,</big></th>
<th><big>Status Report</big></th>
</tr>
</thead>

<tr bgcolor="#e5e5e5">
 <td>
 <a id="Jun08"></a>
 <a class="anchor" href="#Jun08">Jun 08</a>
 </td>
 <td>
 The original goal of analyzing the problems that lead to
 slowdown of Tor Hidden Services has been accomplished. Part of this
 analysis was measuring the delay that a user experiences when
 setting up or accessing a hidden service. Furthermore, measurement data
 from April 2008 could be leveraged to explore timings of internal
 substeps of establishing a connection to a hidden service. The results
 of this analysis are contained in a 22-page
 <a href="http://freehaven.net/~karsten/hidserv/perfanalysis-2008-06-15.pdf">report</a>
 that has been made public on the Tor
 <a href="http://archives.seul.org/or/dev/Jun-2008/msg00019.html">developer
 mailing list</a>.
 
 The analysis also unveiled a few bugs which were responsible
 for part of the delay in making a hidden service available for
 clients. Some bugs have been fixed subsequent to the analysis, others
 will be fixed soon. The evaluation has further brought up several
 possible approaches to improve Tor Hidden Service performance. Some of
 these ideas can be applied immediately, while others require deeper
 analysis and new measurements. Finally, in the course of the analysis,
 we discovered that some improvements require more in-depth
 changes to Tor which are not directly related to hidden services. These
 changes cannot be achieved in the time frame of this
 project.
 </td>
</tr>

<tr>
 <td>
 <a id="Jul08"></a>
 <a class="anchor" href="#Jul08">Jul 08</a>
 </td>
 <td>
 All bugs that have been found in the analysis have been
 fixed. This includes the 2 bugs that have already been fixed during the
 analysis and 4 more bugs that were fixed within the past 30 days. While
 the bugfixes remove unintended performance bottlenecks due to
 programming errors, some of the design changes that have been spotted
 in the previous analysis have side-effects on anonymity or overall
 network load which need to be evaluated against individual performance
 gains. A <a href="http://freehaven.net/~karsten/hidserv/discussion-2008-07-15.pdf">report</a>
 has been published to the
 <a href="http://archives.seul.org/or/dev/Jul-2008/msg00034.html">developer
 mailing list</a> including 7 possible design changes that need to be
 discussed. Some evaluations (namely Low-Bandwidth Measurements and the
 Grand Scaling Plan) have turned out to require more time than expected
 and had to be scheduled for a later time in the project than
 deliverable B. The current plan is to perform these evaluations within
 the timeframe until January 15 and work with assumptions until final
 results are available.
 </td>
</tr>

<tr bgcolor="#e5e5e5">
 <td>
 <a id="Aug08"></a>
 <a class="anchor" href="#Aug08">Aug 08</a>
 </td>
 <td>
 During the past 30 days the 7 proposed designs have been
 further evaluated and discussed. Four of them have proven to be
 applicable in terms of the required changes to the code and possible
 anonymity implications. One has been classified as bug rather than
 design change. Two had to be excluded for either unforeseeable security
 problems, or uncertainty of actual performance
 improvements.
 
 Together with the results from July 15, the design phase has
 been concluded. The tasks for the upcoming implementation phase are now
 quite clear: One bug needs to be fixed and four design changes need to
 be implemented. Further, evaluations of the changed design need to be
 performed in order to verify their usefulness. A
 <a href="http://freehaven.net/~karsten/hidserv/design-2008-08-15.pdf">report</a>
 with the results of the design phase has been published to the
 <a href="http://archives.seul.org/or/dev/Aug-2008/msg00025.html">developer
 mailing list</a>.
 </td>
</tr>

<tr>
 <td>
 <a id="Sep08"></a>
 <a class="anchor" href="#Sep08">Sep 08</a>
 </td>
 <td>
 During the first half of the implementation phase two bugs
 could be fixed that were related to hidden services: the
 <a href="http://bugs.noreply.org/flyspray/index.php?do=details&amp;id=767">first
 bug</a> has already been identified in the design phase and was
 responsible for an unusual high failure rate when making a hidden
 service available in the system; the
 <a href="http://bugs.noreply.org/flyspray/index.php?id=814&amp;do=details">second
 bug</a> was found during the implementation phase and was responsible
 for failure to connect to a working hidden service. Both bugfixes will
 be included in the next unstable version and likely be backported to
 one of the next stable releases.
 
 The four design changes that were proposed as result of the
 design phase have been implemented in an
 <a href="https://tor-svn.freehaven.net/svn/tor/branches/hidserv-design-changes/">experimental
 branch</a> of the unstable development tree. Early function tests have
 shown that these changes work and provide better (perceived)
 performance. This needs to be confirmed throughout the next four weeks
 in internal tests. The next goal is to prepare a release of this
 experimental branch that can be given out to beta testers at the
 beginning of the upcoming testing phase.
 </td>
</tr>

<tr bgcolor="#e5e5e5">
 <td>
 <a id="Oct08"></a>
 <a class="anchor" href="#Oct08">Oct 08</a>
 </td>
 <td>
 The implementation phase has been concluded. The bugfixes
 that were found in the past 30 days have been released in developer
 version <a href="http://archives.seul.org/or/talk/Oct-2008/msg00093.html">0.2.1.6-alpha</a>.
 The four design changes that were identified in the design phase have
 been specified in
 <a href="https://svn.torproject.org/svn/tor/trunk/doc/spec/proposals/155-four-hidden-service-improvements.txt">proposal 155</a>.
 Three design changes have been included in the development codebase and
 will automatically be included in the next development version. The
 first two design changes improve connection establishment to a hidden
 service by reducing a timeout from 60 to 30 seconds and by making a
 second attempt in parallel after a delay of 15 seconds. The third
 design change affects publication of a hidden service in the network by
 advertising the service at 5 rather than 3 points in the network in
 parallel and succeeding as soon as 3 points are established. The fourth
 design change has turned out to be rather ineffective, but would add
 considerable code complexity and was therefore dismissed. By now there
 are no more open bugfixes or new designs. All changes are in the
 development codebase and can be tested in the next phase.
 </td>
</tr>

<tr>
 <td>
 <a id="Nov08"></a>
 <a class="anchor" href="#Nov08">Nov 08</a>
 </td>
 <td>
 The performance improvements that were implemented in the
 last phase have been released in Tor version 0.2.1.7-alpha. Users can
 download this development version from the Tor homepage and test the
 improvements with minimal effort. Further, two bugfixes (<a
 href="http://bugs.noreply.org/flyspray/index.php?id=767&amp;do=details">1</a>,
 <a href="http://bugs.noreply.org/flyspray/index.php?id=814&amp;do=details">2</a>)
 that were found in the course of this project have been backported to
 the stable branch and will be included with the next stable version
 0.2.0.32.
 
 The main focus of the past 31 days was to perform new
 measurements to see whether the improvements are effective or not.
 Measurements were conducted for two days in the time of November 6th to
 8th. Unfortunately, the Tor network suffered a serious problem in this
 time: An expired directory authority certificate produced huge amounts
 of traffic within the Tor network which <a
 href="http://archives.seul.org/or/talk/Nov-2008/msg00053.html">forced
 many operators to shut down their relays</a>. A second measurement was
 performed between 13th and 15th. The raw data are available <a
 href="http://freehaven.net/~karsten/hidserv/perfdata-2008-11-13.tar.gz">here</a>
 (40 MB). But results show that the overall
 network performance is still worse than in June 2008 when the first
 hidden service measurements have been performed. This becomes visible
 when comparing requests to the Tor directories which have not been
 affected by the performance improvements and which exhibit
 significantly worse performance than before. The effects of performance
 improvements are visible, but absolute values are not comparable at
 this time. New measurements will be conducted in December in the hope
 that the effects of this problem have mitigated.
 
 Further, there might be a <a
 href="http://bugs.noreply.org/flyspray/index.php?id=847&amp;do=details">bug</a>
 in the way how Tor downloads directory information during
 bootstrapping. Even though this is not related to hidden services, an
 improvement would benefit hidden service publication, too. Part of the
 work during the upcoming 30 days will be to investigate this bug.
 
 </td>
</tr>

<tr bgcolor="#e5e5e5">
 <td>
 <a id="Dec08"></a>
 <a class="anchor" href="#Dec08">Dec 08</a>
 </td>
 <td>
 Part of the last 30 days has been used to fix bugs that have
 influenced the previous hidden service measurements. The first <a
 href="http://archives.seul.org/or/cvs/Nov-2008/msg00100.html">bugfix</a>
 corrects a possible segmentation fault that was very likely responsible
 for a number of failed measurement runs. Another <a
 href="https://bugs.torproject.org/flyspray/index.php?id=847&amp;do=details">bug</a>
 could be explained that lead to significant delays in bootstrapping:
 Very slow directory authorities occupied bootstrapping clients for
 a long time before clients finally gave up and bootstrapped using
 another authority. As a result, the slowest two directory authorities
 have dedicated more bandwidth to their nodes, so that the effect is
 mitigated. A third <a
 href="https://bugs.torproject.org/flyspray/index.php?id=874&amp;do=details">bug</a>
 has been introduced with the hidden service performance improvements in
 November; the effect was that Tor processes running hidden services
 would stop advertising their service upon reloading their
 configuration. Further, this bug has uncovered that Tor has
 re-established its introduction points upon reloading, which might have
 affected hidden service stability. This bug has been fixed and will be
 included in the upcoming version 0.2.1.9-alpha.
 
 Apart from fixing bugs, new measurements have been performed
 between December 8 and 10. These will very likely be the final
 measurements to compare hidden service performance now with the
 beginning of the project. The data have not been completely evaluated,
 so it is difficult to make a statement about improvements at this
 point. However, a <a
 href="http://freehaven.net/~karsten/hidserv/prelimreport-2008-12-15.pdf">preliminary
 evaluation</a> shows that service publication times have improved
 significantly. This is a result of Tor clients bootstrapping faster and
 of the performance improvements added in November. In contrast to this,
 the results for establishing a connection to a hidden service are less
 promising. While the improvements added in November seem to have a
 positive effect on performance, some substeps exhibit significantly
 worse performance. One example is fetching hidden service descriptors
 in order to contact a hidden service. A possible explanation is that
 the sudden increase in the number of hidden service directory nodes in
 September has had a negative effect on performance. Part of the work in
 the final 31 days will be to evaluate these data in more detail and
 make a final conclusion on the achievements of this
 project.
 </td>
</tr>

<tr>
 <td>
 <a id="Jan09"></a>
 <a class="anchor" href="#Jan09">Jan 09</a>
 </td>
 <td>
 The testing phase has been concluded. Testing was performed
 in a public beta phase with all changes to hidden services being part
 of the 0.2.1.x-alpha series. The result of the public beta phase is a
 couple of identified bugs that could already be fixed.
 
 Another part of testing was a second set of measurements
 that was performed in December. A
 <a href="http://freehaven.net/~karsten/hidserv/comparison-2009-01-15.pdf">comparison</a>
 of measurements performed in June and December has revealed that the
 changes of this project are effective. Service publication times could
 be more than halved from 2:12 minutes to 58 seconds in the mean. This
 improvement is far better than expected. With this improvement it might
 even be worthwile to think about reducing stabilization time from 30
 seconds to a lower value in the future. However, connection
 establishment remains at approximately 56 seconds between requesting a
 hidden service to having established a connection to the hidden server.
 The main reason for missing improvements is the switch from the
 centralized to a decentralized storage for hidden service descriptors.
 This deteriorating effect of distributing the hidden service directory
 has not been expected before. Future work should focus on improving
 downloads from the distributed hidden service directory, for example by
 parallelizing requests.
 
 This report concludes the series of monthly status updates.
 The rollout of the 0.2.1.x series including the hidden service
 performance improvements is going to take place within the next weeks
 to months.
 </td>
</tr>
</table>

<a id="Links"></a>
<h2><a class="anchor" href="#Links">Links</a></h2>
<ul>
<li>Research paper on Performance Measurements and Statistics of Tor
Hidden Services
(<a href="http://www.uni-bamberg.de/fileadmin/uni/fakultaeten/wiai_lehrstuehle/praktische_informatik/Dateien/Publikationen/loesing2008performance.pdf">PDF</a>)
by Karsten Loesing, Werner Sandmann, Christian Wilms, and Guido Wirtz. In
the Proceedings of the 2008 International Symposium on Applications and the
Internet (SAINT), Turku, Finland, July 2008.

</ul>

</div>

#include <foot.wmi>

tor-webwml.git

Monthly status report for January.