tor-webwml.git

1) ## translation metadata

2) # Revision: $Revision$

3) # Translation-Priority: 4-optional

4) 
5) #include "head.wmi" TITLE="NLnet Project: Speed Up Tor Hidden Services"
6) 
7) <div class="main-column">
8) 
9) <!-- PUT CONTENT AFTER THIS TAG -->
10) 
11) <h2>NLnet Project: Speed Up Tor Hidden Services</h2>
12) <hr />
13) 
14) <p>
15) Tor Hidden Services allow users to set up anonymous information services,
16) like websites, that can only be accessed through the Tor network and are
17) protected against identification of the host that runs the services.

18) The most critical limitations of Tor Hidden Services are the time it takes

19) until a Hidden Service is registered in the network and the latency of
20) contact establishment when accessed by a user.
21) Due to design issues in the original Tor protocol,
22) the connection to a new Hidden Service can take several minutes, which
23) leads most users to give up before the connection has been established.
24) Using Tor Hidden Services for direct interactive user-to-user
25) communication (e.g. messaging) is nearly impossible due to the high latency
26) of Hidden Service circuit setup.
27) </p>
28) 
29) <p>
30) This project aims at speeding up Tor Hidden Services by improving the way
31) Tor circuits are set up between the user and the Hidden Service as well as
32) the way a Hidden Service is registered in the Tor network.
33) In a first step precise diagnostics of the behavior of the Hidden Services
34) in lab setups and real world situations will be conducted to find the
35) root causes of the bad timing effects.
36) Based on these diagnostics, optimization strategies will be designed and
37) verified for unwanted implications for the security and anonymity of the
38) Tor network.
39) The most promising optimizations will then be implemented to achieve a
40) notable improvement for the users. Precise success metrics will be
41) developed in the diagnostics phase, after it becomes clear where the time
42) is lost and what improvements are realistic.
43) The ultimate goal is to have the Hidden Services protocol change production
44) ready and propagated to the Tor users within a timeframe of less than
45) 12 months.
46) </p>
47) 
48) <p>

49) This project is generously funded by:
50) </p>
51) 
52) <p>
53) <a href="http://www.nlnet.nl/news/2008/20080514-awards.html">
54) <img src="$(IMGROOT)/nlnet-160x60.png" alt="The NLnet foundation" /></a>

55) </p>
56) 
57) <table width="100%" border="0" cellspacing="0" cellpadding="3">

58) <thead>
59) <tr>
60) <th><big>Project</big></th>
61) <th><big>Due Date</big></th>

62) </tr>

63) </thead>

64) 
65) <tr bgcolor="#e5e5e5">
66)   <td>
67)     <b>Deliverable A:</b> Analysis, measurements and problem
68)     clarification<br />
69)     <small><em>As Tor Hidden Services have not been actively developed
70)     further in the last year or so of Tor development, certain aspects of
71)     the problems are under-diagnosed. To identify the precise sources of
72)     latency and time loss, an extensive analysis of the deeper reasons for
73)     them needs to be conducted. Deliverable A will require about one month
74)     of work. The results of the analysis will influence the design
75)     decisions to be taken in Deliverable B.</em></small>
76)   </td>
77)   <td>
78)     June 15, 2008
79)   </td>
80) </tr>
81) 
82) <tr>
83)   <td>
84)     <b>Deliverable B:</b> Design and evaluation of the necessary
85)     changes<br />
86)     <small><em>The changes to Tor Hidden Services will affect core
87)     functionality of the protocol and therefore require a careful
88)     evaluation of possible repercussions for the security and anonymity. A
89)     two-month period is planned for the design and evaluation phase, which
90)     concludes with an extensive peer review.</em></small>
91)   </td>
92)   <td>
93)     August 15, 2008
94)   </td>
95) </tr>
96) 
97) <tr bgcolor="#e5e5e5">
98)   <td>
99)     <b>Deliverable C:</b> Implementation<br />
100)     <small><em>After design, evaluation and peer review the modifications
101)     need to be implemented and integrated with the current Tor code base.
102)     The actual implementation of the necessary changes will take
103)     approximately two months.</em></small>
104)   </td>
105)   <td>
106)     October 15, 2008
107)   </td>
108) </tr>
109) 
110) <tr>
111)   <td>
112)     <b>Deliverable D:</b> Implementation and test of the change up to
113)     release state<br />
114)     <small><em>The modification is highly critical to the security and
115)     anonymity of the Tor network, it requires extensive testing and
116)     debugging in laboratory and real life conditions. A period of three
117)     months is projected for testing and debugging, where the responsible
118)     developer is committed to the testing effort with 1/3 of its time. Part
119)     of the testing phase will be a public beta period.</em></small>
120)   </td>
121)   <td>
122)     January 15, 2009
123)   </td>
124) </tr>
125) 
126) <tr bgcolor="#e5e5e5">
127)   <td>
128)     <b>Deliverable E:</b> Rollout<br />
129)     <small><em>The actual rollout to the Tor server network will be
130)     conducted in sync with the regular Tor release schedule. As this
131)     schedule is dependent on a number of external factors, like the
132)     completion of other software projects that should go into the same
133)     release, the actual release time and the time until this release has
134)     been accepted and installed by most Tor server operators can vary. From
135)     experience a period of three to four months can be
136)     expected.</em></small>
137)   </td>
138)   <td>
139)     May 15, 2009
140)   </td>
141) </tr>
142) </table>
143) 
144) <br />
145) 
146) <a id="Reports"></a>
147) <h2><a class="anchor" href="#Reports">Monthly Status Reports</a></h2>
148) <p>
149) There will be in total eight monthly status reports beginning with the
150) first deliverable on June 15, 2008 and ending with completion of
151) implementation and testing work on January 15, 2009.
152) </p>
153) 

154) <table width="100%" border="0" cellspacing="0" cellpadding="3">
155) <thead>
156) <tr>
157) <th><big>Month,</big></th>
158) <th><big>Status Report</big></th>
159) </tr>
160) </thead>
161) 
162) <tr bgcolor="#e5e5e5">
163)   <td>

164)     <a id="Jun08"></a>
165)     <a class="anchor" href="#Jun08">Jun 08</a>

166)   </td>
167)   <td>
168)     <small><em>The original goal of analyzing the problems that lead to
169)     slowdown of Tor Hidden Services has been accomplished. Part of this

170)     analysis was measuring the delay that a user experiences when

171)     setting up or accessing a hidden service. Furthermore, measurement data
172)     from April 2008 could be leveraged to explore timings of internal
173)     substeps of establishing a connection to a hidden service. The results
174)     of this analysis are contained in a 22-page
175)     <a href="http://freehaven.net/~karsten/hidserv/perfanalysis-2008-06-15.pdf">report</a>
176)     that has been made public on the Tor
177)     <a href="http://archives.seul.org/or/dev/Jun-2008/msg00019.html">developer

178)     mailing list</a>.</em></small>

179)     <br/>

180)     <small><em>The analysis also unveiled a few bugs which were responsible
181)     for part of the delay in making a hidden service available for

182)     clients. Some bugs have been fixed subsequent to the analysis, others
183)     will be fixed soon. The evaluation has further brought up several
184)     possible approaches to improve Tor Hidden Service performance. Some of
185)     these ideas can be applied immediately, while others require deeper
186)     analysis and new measurements. Finally, in the course of the analysis,

187)     we discovered that some improvements require more in-depth

188)     changes to Tor which are not directly related to hidden services. These
189)     changes cannot be achieved in the time frame of this

190)     project.</em></small>

191)   </td>
192) </tr>
193) 
194) <tr>
195)   <td>

196)     <a id="Jul08"></a>
197)     <a class="anchor" href="#Jul08">Jul 08</a>

198)   </td>
199)   <td>

200)     <small><em>All bugs that have been found in the analysis have been
201)     fixed. This includes the 2 bugs that have already been fixed during the

202)     analysis and 4 more bugs that were fixed within the past 30 days. While

203)     the bugfixes remove unintended performance bottlenecks due to
204)     programming errors, some of the design changes that have been spotted
205)     in the previous analysis have side-effects on anonymity or overall

206)     network load which need to be evaluated against individual performance

207)     gains. A <a href="http://freehaven.net/~karsten/hidserv/discussion-2008-07-15.pdf">report</a>
208)     has been published to the
209)     <a href="http://archives.seul.org/or/dev/Jul-2008/msg00034.html">developer
210)     mailing list</a> including 7 possible design changes that need to be
211)     discussed. Some evaluations (namely Low-Bandwidth Measurements and the

212)     Grand Scaling Plan) have turned out to require more time than expected
213)     and had to be scheduled for a later time in the project than
214)     deliverable B. The current plan is to perform these evaluations within
215)     the timeframe until January 15 and work with assumptions until final
216)     results are available.</em></small>

217)   </td>
218) </tr>
219) 
220) <tr bgcolor="#e5e5e5">
221)   <td>

222)     <a id="Aug08"></a>
223)     <a class="anchor" href="#Aug08">Aug 08</a>

224)   </td>
225)   <td>

226)     <small><em>During the past 30 days the 7 proposed designs have been
227)     further evaluated and discussed. Four of them have proven to be
228)     applicable in terms of the required changes to the code and possible
229)     anonymity implications. One has been classified as bug rather than
230)     design change. Two had to be excluded for either unforeseeable security
231)     problems, or uncertainty of actual performance
232)     improvements.</em></small>
233)     <br/>
234)     <small><em>Together with the results from July 15, the design phase has
235)     been concluded. The tasks for the upcoming implementation phase are now
236)     quite clear: One bug needs to be fixed and four design changes need to
237)     be implemented. Further, evaluations of the changed design need to be
238)     performed in order to verify their usefulness. A
239)     <a href="http://freehaven.net/~karsten/hidserv/design-2008-08-15.pdf">report</a>
240)     with the results of the design phase has been published to the
241)     <a href="http://archives.seul.org/or/dev/Aug-2008/msg00025.html">developer
242)     mailing list</a>.</em></small>

243)   </td>
244) </tr>
245) 
246) <tr>
247)   <td>

248)     <a id="Sep08"></a>
249)     <a class="anchor" href="#Sep08">Sep 08</a>

250)   </td>
251)   <td>

252)     <small><em>During the first half of the implementation phase two bugs
253)     could be fixed that were related to hidden services: the

254)     <a href="http://bugs.noreply.org/flyspray/index.php?do=details&amp;id=767">first

255)     bug</a> has already been identified in the design phase and was
256)     responsible for an unusual high failure rate when making a hidden
257)     service available in the system; the

258)     <a href="http://bugs.noreply.org/flyspray/index.php?id=814&amp;do=details">second

259)     bug</a> was found during the implementation phase and was responsible
260)     for failure to connect to a working hidden service. Both bugfixes will
261)     be included in the next unstable version and likely be backported to
262)     one of the next stable releases.</em></small>
263)     <br/>
264)     <small><em>The four design changes that were proposed as result of the
265)     design phase have been implemented in an
266)     <a href="https://tor-svn.freehaven.net/svn/tor/branches/hidserv-design-changes/">experimental
267)     branch</a> of the unstable development tree. Early function tests have
268)     shown that these changes work and provide better (perceived)
269)     performance. This needs to be confirmed throughout the next four weeks
270)     in internal tests. The next goal is to prepare a release of this
271)     experimental branch that can be given out to beta testers at the
272)     beginning of the upcoming testing phase.</em></small>

273)   </td>
274) </tr>
275) 
276) <tr bgcolor="#e5e5e5">
277)   <td>

278)     <a id="Oct08"></a>

279)     <a class="anchor" href="#Oct08">Oct 08</a>

280)   </td>
281)   <td>

282)     <small><em>The implementation phase has been concluded. The bugfixes
283)     that were found in the past 30 days have been released in developer
284)     version <a href="http://archives.seul.org/or/talk/Oct-2008/msg00093.html">0.2.1.6-alpha</a>.
285)     The four design changes that were identified in the design phase have
286)     been specified in

287)     <a href="https://svn.torproject.org/svn/tor/trunk/doc/spec/proposals/155-four-hidden-service-improvements.txt">proposal 155</a>.

288)     Three design changes have been included in the development codebase and
289)     will automatically be included in the next development version. The
290)     first two design changes improve connection establishment to a hidden
291)     service by reducing a timeout from 60 to 30 seconds and by making a
292)     second attempt in parallel after a delay of 15 seconds. The third
293)     design change affects publication of a hidden service in the network by
294)     advertising the service at 5 rather than 3 points in the network in

295)     parallel and succeeding as soon as 3 points are established. The fourth

296)     design change has turned out to be rather ineffective, but would add
297)     considerable code complexity and was therefore dismissed. By now there
298)     are no more open bugfixes or new designs. All changes are in the
299)     development codebase and can be tested in the next phase.</em></small>

300)   </td>
301) </tr>
302) 
303) <tr>
304)   <td>

305)     <a id="Nov08"></a>
306)     <a class="anchor" href="#Nov08">Nov 08</a>

307)   </td>
308)   <td>

309)     <small><em>The performance improvements that were implemented in the
310)     last phase have been released in Tor version 0.2.1.7-alpha. Users can
311)     download this development version from the Tor homepage and test the
312)     improvements with minimal effort. Further, two bugfixes (<a

313)     href="http://bugs.noreply.org/flyspray/index.php?id=767&amp;do=details">1</a>,
314)     <a href="http://bugs.noreply.org/flyspray/index.php?id=814&amp;do=details">2</a>)

315)     that were found in the course of this project have been backported to
316)     the stable branch and will be included with the next stable version
317)     0.2.0.32.</em></small>
318)     <br/>
319)     <small><em>The main focus of the past 31 days was to perform new
320)     measurements to see whether the improvements are effective or not.
321)     Measurements were conducted for two days in the time of November 6th to
322)     8th. Unfortunately, the Tor network suffered a serious problem in this
323)     time: An expired directory authority certificate produced huge amounts
324)     of traffic within the Tor network which <a
325)     href="http://archives.seul.org/or/talk/Nov-2008/msg00053.html">forced
326)     many operators to shut down their relays</a>. A second measurement was

327)     performed between 13th and 15th. The raw data are available <a
328)     href="http://freehaven.net/~karsten/hidserv/perfdata-2008-11-13.tar.gz">here</a>
329)     (40 MB). But results show that the overall

330)     network performance is still worse than in June 2008 when the first
331)     hidden service measurements have been performed. This becomes visible
332)     when comparing requests to the Tor directories which have not been
333)     affected by the performance improvements and which exhibit
334)     significantly worse performance than before. The effects of performance
335)     improvements are visible, but absolute values are not comparable at
336)     this time. New measurements will be conducted in December in the hope
337)     that the effects of this problem have mitigated.</em></small>
338)     <br/>
339)     <small><em>Further, there might be a <a

340)     href="http://bugs.noreply.org/flyspray/index.php?id=847&amp;do=details">bug</a>

341)     in the way how Tor downloads directory information during
342)     bootstrapping. Even though this is not related to hidden services, an
343)     improvement would benefit hidden service publication, too. Part of the
344)     work during the upcoming 30 days will be to investigate this bug.
345)     </em></small>

346)   </td>
347) </tr>
348) 
349) <tr bgcolor="#e5e5e5">
350)   <td>

351)     <a id="Dec08"></a>
352)     <a class="anchor" href="#Dec08">Dec 08</a>

353)   </td>
354)   <td>

355)     <small><em>Part of the last 30 days has been used to fix bugs that have
356)     influenced the previous hidden service measurements. The first <a
357)     href="http://archives.seul.org/or/cvs/Nov-2008/msg00100.html">bugfix</a>
358)     corrects a possible segmentation fault that was very likely responsible
359)     for a number of failed measurement runs. Another <a

360)     href="https://bugs.torproject.org/flyspray/index.php?id=847&amp;do=details">bug</a>

361)     could be explained that lead to significant delays in bootstrapping:
362)     Very slow directory authorities occupied bootstrapping clients for
363)     a long time before clients finally gave up and bootstrapped using
364)     another authority. As a result, the slowest two directory authorities
365)     have dedicated more bandwidth to their nodes, so that the effect is
366)     mitigated. A third <a

367)     href="https://bugs.torproject.org/flyspray/index.php?id=874&amp;do=details">bug</a>

368)     has been introduced with the hidden service performance improvements in
369)     November; the effect was that Tor processes running hidden services
370)     would stop advertising their service upon reloading their
371)     configuration. Further, this bug has uncovered that Tor has
372)     re-established its introduction points upon reloading, which might have
373)     affected hidden service stability. This bug has been fixed and will be
374)     included in the upcoming version 0.2.1.9-alpha.</em></small>
375)     <br/>
376)     <small><em>Apart from fixing bugs, new measurements have been performed
377)     between December 8 and 10. These will very likely be the final
378)     measurements to compare hidden service performance now with the
379)     beginning of the project. The data have not been completely evaluated,
380)     so it is difficult to make a statement about improvements at this
381)     point. However, a <a
382)     href="http://freehaven.net/~karsten/hidserv/prelimreport-2008-12-15.pdf">preliminary
383)     evaluation</a> shows that service publication times have improved
384)     significantly. This is a result of Tor clients bootstrapping faster and
385)     of the performance improvements added in November. In contrast to this,
386)     the results for establishing a connection to a hidden service are less
387)     promising. While the improvements added in November seem to have a
388)     positive effect on performance, some substeps exhibit significantly
389)     worse performance. One example is fetching hidden service descriptors
390)     in order to contact a hidden service. A possible explanation is that
391)     the sudden increase in the number of hidden service directory nodes in
392)     September has had a negative effect on performance. Part of the work in
393)     the final 31 days will be to evaluate these data in more detail and
394)     make a final conclusion on the achievements of this
395)     project.</em></small>

396)   </td>
397) </tr>
398) 
399) <tr>
400)   <td>

401)     <a id="Jan09"></a>
402)     <a class="anchor" href="#Jan09">Jan 09</a>

403)   </td>
404)   <td>

405)     <small><em>The testing phase has been concluded. Testing was performed
406)     in a public beta phase with all changes to hidden services being part
407)     of the 0.2.1.x-alpha series. The result of the public beta phase is a
408)     couple of identified bugs that could already be fixed.</em></small>
409)     <br/>
410)     <small><em>Another part of testing was a second set of measurements
411)     that was performed in December. A
412)     <a href="http://freehaven.net/~karsten/hidserv/comparison-2009-01-15.pdf">comparison</a>
413)     of measurements performed in June and December has revealed that the
414)     changes of this project are effective. Service publication times could
415)     be more than halved from 2:12 minutes to 58 seconds in the mean. This
416)     improvement is far better than expected. With this improvement it might
417)     even be worthwile to think about reducing stabilization time from 30
418)     seconds to a lower value in the future. However, connection
419)     establishment remains at approximately 56 seconds between requesting a
420)     hidden service to having established a connection to the hidden server.
421)     The main reason for missing improvements is the switch from the
422)     centralized to a decentralized storage for hidden service descriptors.
423)     This deteriorating effect of distributing the hidden service directory
424)     has not been expected before. Future work should focus on improving
425)     downloads from the distributed hidden service directory, for example by
426)     parallelizing requests.</em></small>
427)     <br/>
428)     <small><em>This report concludes the series of monthly status updates.
429)     The rollout of the 0.2.1.x series including the hidden service
430)     performance improvements is going to take place within the next weeks
431)     to months.</em></small>

432)   </td>
433) </tr>
434) </table>
435) 
436) <br />

437) 
438) <!-- Do we want a people section? If so, would it make sense to write what

439) these people will be doing? And what exactly are these people going to