The Birth of DHT, May 2005
When BitTorrent started in 2002, decentralization was one of its main innovations. The central structure of services like Napster ultimately led to their downfall, and while decentralized systems such as eDonkey/eMule and Gnutella existed, they were often cumbersome and filled with fakes and spam.
BitTorrent was also somewhat individualized. Clients only dealt with clients on the swarms they were interested in, and all conducted business through a tracker.
This led to problems though, when trackers went down, as the trackers were the only way for peers to get information about others in the swarm. There was no fallback, except trying to add more trackers and hope everyone else adds the same. However, with the launch of Distributed Hash Tables (DHT) these problems were all but over.
That two similar but incompatible DHT systems were launched within weeks of each other is quite surprising, given the history behind both. To this day, in fact, the systems are still incompatible, although there are plug-ins that allow the use of both to act as a bridge between the two swarms (one Vuze, one Mainline).
When you factor in that both were released just months after eXeem had tried and failed to do a similar thing (earning significant criticism while doing so) the success and longevity of both look even more impressive. But how did they come about?
The Vuze DHT debuted first, with version 2.3.0 of the Azureus client on May 2, 2005. In its announcements back then, they were keen to stress the difference from eXeem, stating it was a decentralized layer on top of BitTorrent, rather than a decentralised BitTorrent system itself. Within 24 hours there were more than 200,000 peers, and there are currently around 1.1 million peers on the network.
According to Paul Gardner, the main developer of the Azureus DHT system, tracker redundancy wasn’t the main reason behind its development. Instead, decentralization for search was driving it.
“That was one of my pet aims when I joined the Azureus development team,” Gardner told TF earlier this month. “But the others in the team weren’t sure if search was a priority, so I found a way of working on some decentralization that perhaps one day could evolve into/be adapted for search. Of course decentralized tracking was a good aim in itself.”
Paul Gardner, Azureus/Vuze Developer
“I started from scratch,” Gardner recalls, “there weren’t any libraries out there I could use, so had to figure out which kind of DHT to use (Kademlia) etc. [It took] a few months I guess.”
Three weeks later, Bittorrent Inc. released their own version of DHT with the release of version 4.1. This was then adopted by the then popular client BitComet in early June, and by other clients soon after.
While the timing may suggest otherwise, BitTorrent’s DHT wasn’t a response to Vuze’s release at all, as the person responsible – Drue Loewenstern – had been working on it since 2002.
“I started working on the DHT in the summer of 2002 after making the first Mac BitTorrent clients, a year before Azureus was established on Sourceforge. Finishing it off and integration into BitTorrent started in 05 when BT became a company. I was in testing and about to release it when Azureus launched theirs,” Loewenstern says.
The inspiration for the BitTorrent mainline DHT came from an unlikely and famous source: Aaron Swartz.
“Distributed hash tables were an inspiring area of research. I was really into P2P, having just worked on MojoNation and BitTorrent, and wanted to do all sorts of cool decentralized things like trust metrics. Aaron Swartz, 15 at the time, circulated a one page implementation of the Chord algorithm and I was struck by its simplicity, Loewenstern notes.
“I started looking into DHTs specifically and Kademlia was the first DHT paper that really clicked with me and seemed like it might work in the real world So I decided to start implementing it without really knowing what I was going to do with it.”
Contrary to Vuze, redundancy was one of the main motivations driving the development of the mainline DHT.
In the case of BitTorrent, the goal of the DHT has always been to make BT more robust, to improve performance by finding more peers, and to simplify publishing by making a tracker optional,” Loewenstern says.
DHT ‘Haters’
Of course, not everyone was thrilled to see the introduction of DHT. Private trackers were opposed to DHT as it enabled people to use the site’s torrents without being under the strict control of the tracker admins.
The solution to this was a form of access control called the private flag, which disabled DHT, along with Peer Exchange (PEX) and restricted peer access to trackers – locking things into the way of 2005.
The flag works by being inside the data used to generate the hash, so if disabled, it would change the overall torrent hash, meaning a torrent with the flag enabled would be a completely separate swarm from one with the flag disabled. It also gave these sites a new way to market themselves, by taking the term “private flagged torrent trackers” and condensing it to “private trackers,” implying some form of privacy.
This move though, was not by choice.
“There’s always been tensions between clients and private trackers,” Vuze’s Gardner says. “In particular they like to ban Vuze because it is ‘open source and people have hacked it to report incorrect stats’ or other such ‘reasons’. I’ve never been a fan of [the private flag] as a solution.”
Loewenstern agrees
“It came to be because some index site operators enforce upload/download ratios in an effort to keep seeders around for torrents that nobody wants to be left holding the bag for by seeding. They thought DHT (and PEX) might let users bypass the ratio system so they made a lot of noise about banning clients that implemented DHT,” he says.
“Azureus didn’t want to get banned so they came up with the private flag and added it to their client. It wasn’t my decision to add it to BitTorrent. Without PEX, torrents take longer to ramp up so it annoys me when people upload private torrents to public index sites.”
NEXT, The BitComet Incident
The BitComet Incident, December 2005
BitComet, one of the dominant BitTorrent clients at the time, came out in support of the mainline DHT standard in early June of 2005. It set the tempo for other clients and showed that DHT in general was a really good idea. Yet BitComet was also going to provide the first major client dispute later that year with the release of v0.60.
At the core, the same old argument of wanting absolute control over peers (and thus the private flag) made into a big deal by those desperate to prevent any loss of control.
In short, BitComet decided to implement a system where if the tracker on a private-flagged torrent went offline for a certain period of time, the client would enable DHT, and when the tracker came back up, DHT would be disabled again. This common sense approach provoked outrage based on deliberate misrepresentations, but it generated enough noise that BitComet withdrew the 0.60 client, eventually replacing it with v0.61 and the now ‘standard’ DHT functionality.
More DHT Push-back
This was only the first major push-back against DHT by those desperate to remain in control, and perhaps the most overt. It’s become common for private trackers to ‘demand’ that DHT be disabled in the client and for people to become disorientated as a result, when often those making the claims were the most confused. There have even been claims made that certain clients leak peer info to DHT, a myth we debunked back in 2009.
There is one thing that does ‘encourage’ people to disable DHT though. Especially in 2005 (but still occasionally now) network hardware had problems with DHT. The problem was caused by the significant use of UDP, rather than the more traditional TCP. Routers with limited ram would have it filled by the UDP traffic and lock up or go very slowly.
Thanks to Gigabit LAN and Wireless-N and AC, however, this is less of an issue now than it once was, even with multiple computers running DHT connected. It’s now mostly the province of ISP-provided hardware, which tends to hover on the low-end of the equipment spectrum for cost purposes.
The Pirate Bay Boost, September 2009
Despite all its benefits, DHT adoption was slow until 2009 when it got a huge boost from one of the biggest torrent sites around, The Pirate Bay. In November 2009, they announced they were going to be closing their tracker, and moving towards magnet links, to make the site lighter and quicker.
Magnets were not new, they were part of the DHT spec, but hadn’t gained much use until TPB’s change. Suddenly, lots of users were trying to work out how to get magnets to work, both with clients and browsers, and because of the requirement for DHT in order to make magnets work, DHT itself started becoming more popular. Of course, that meant that a lot of the old rumours about DHT also resurfaced, some of which still persist to this day.
Search at last, March 2011
As mentioned earlier, search was one of Gardner’s main motivations for creating DHT, but it wasn’t until 2011 that a DHT search engine emerged. BTDigg started at the end of March 2011, and was an invaluable resource for many.
The ability to tap into the DHT network as a whole with a decent interface meant it was not only the biggest search engine (since it had pretty much any torrent that ran on a DHT-enabled client without the private flag set) but also the most like mainstream search engines, in that it actively searched and indexed entries, rather than waiting for them to be manually added like traditional indexes.
“The main reason to make BTDigg,” their spokesperson ‘NE’ told TorrentFreak, “was an absence of independent search engines in the BitTorrent network. It was the same situation as the Web in pre-Google era, the era of Web Catalogues.”
Alas, the site closed its web front end earlier this year, remaining available via both TOR and I2P, a situation they put down to their host withdrawing one of their servers, with no reason given. In the meantime, successors like Strike carry on their work
DHT today, 2015
Which brings us to now, ten years after DHT provided the last functional change to the BitTorrent protocol (utp was more of a technological one). DHT has enabled Bittorrent to remain the top dog, and while a dozen-or-so competing P2P systems jockeyed for dominance in the six years after Napster’s launch, nothing has even come close to challenging BitTorrent.
“I researched the question ‘Why is BitTorrent still popular?’ and concluded that the main reason is because the BitTorrent protocol inherits good features from previous protocols (like eMule & Kazaa) and eliminates their weak sides. So the appearance of BitTorrent was logical evolution of previous people’s efforts in file sharing,” NE told us earlier this year.
DHT was clearly the last step, enabling an underlying central mesh network to support the decentralized nature of the individual torrent swarms. It gave it the resilience and longevity to last.
Ten years on though, is there anything that the two architects of DHT would do differently if they could go back to 2005? Loewenstern doesn’t seem to think so.
“It’s quite humbling to have academics who really know their stuff go over your work with a fine toothed comb and find all the problems. There are a couple of BEPs (Bittorrent Enhancement Proposal) for addressing some design decisions but as [BitTorrent creator] Bram Cohen says, ‘the DHT works,’ so I’d rather go make something else cool.”
Gardner agrees saying, “Nothing major to be honest, after 10 years most things have been ironed out. I mean everything needs rewriting once it is complete, that’s the way software works.”
Ten years on and going strong, DHT is clearly ready for the next ten. With over 25 Million users between the two networks, it’s not going anywhere.
