After much arguing, cogitating, arguing, and cogitating, the bt2 protocol
designs are now further along than they were before.
Features planned for bt2 -
merkle hash trees - this is by far the most compelling reason to break
compatibility. Files in a multi-file will each have their own hash root.
udp-based tracker protocol (with http-based alternative for those who care
less about bandwidth than convenience)
tracker redirection - when talking to a tracker, that tracker will tell
you where your primary tracker is (usually the one you just hit) and your
backup tracker is. This will enable (1) dynamic tracker load
redistribution, and (2) tracker backups. Of course, some smarts on the
tracker are necessary. When trackers are distributed they either have to
send some peers to each other or make sure a seed is on each tracker, and
when using a backup tracker it has to redirect everyone back to the main
tracker when it comes back up again. Since those smarts don't involve
changing the protocol, I'm completely punting on it until later.
a beefed up peer protocol - there are some subtle changes to the state
machine planned, which are strict improvements but kind of involved so
I'll skip over them now. Also peers will announce which files they want to
enable cross-torrent trading. Finally, announcements of having parts of
pieces will be added, since that's enabled by hash trees. Doing that last
part well requires some smarts, but again the smarts don't change the
protocol so I'm punting.
The above are the things I'm convinced are a good idea. Notably missing is
gossip. Too dangerous, may be added later via an extension mechanism.
The main sticking point left is how to deal with piece sizes - ideally all
peers should be using the same piece size, but whether that's a good thing
to require and if so (or even if not) how to set it I'm not sure of yet.
Also I'm not sure how to make trackers support scrape functionality (or
more to the point, I'm not sure how much scrape functionality to carry
over, and in what way). Other than those issues all that's left is a whole
mess of details.
Rest assured that bt2 protocol will have the same ease of implementation
and reliable compatibilty of bt1. Also rest assured I won't push the damn
thing out until I'm certain it will get feature requests down to a dull
roar for at least a few months.
After much arguing, cogitating, arguing, and cogitating, the bt2 protocol designs are now further along than they were before. Features planned for bt2 - ...
... Sweet! ... The peers could provide a list of root hashes for other files they have (all of? Parts of?) ? And the primary tracker URL for each hash? Or am...
... When peers connect to each other they'll both all announce the individual files that they have. Tracker announces will probably give a unique identifier...
... Related thing, how about peers refering to pieces *strictly* by hash? This adds to anonimity by allowing for lack of a better term, better "zombie" clients...
Elliott Mitchell
ehem@...
Jun 5, 2004 5:42 pm
... I'm not sure that just a hash with no further information is sufficient to uniquely identify a chunk. There are only so many hashes possible - and although...
... True, it is a valid concern; but you have to look at the statistics. Figure, that maybe, perhaps there might be a petabyte (2^50) of data floating around...
Elliott Mitchell
ehem@...
Jun 7, 2004 8:18 pm
... It turns out that the size of those hashes can be fairly significant, and the program logic winds up doing a lookup to translate back from hash to index on...
... Let me know if any of this is wrong... If you were to eliminate the "offset" and "length" fields when requesting a chunk, and make the chunksize...
... Yes, a piece size of between 16 kb and 64 kb would be required and that would significantly increase the protocol overhead compared with 2 mb pieces. And...
... Really? The mainline client by default already requests chunks in mere 32KB chunks and will ignore requests larger than 256KB. Even if this size has to be...
Elliott Mitchell
ehem@...
Jun 7, 2004 8:45 pm
... You'd have to send more HAVEs and you need more hashes. ... That depends on the actual implementation of merkle trees. ... True, but that'd introduce the...
... Yes, but how large is that? 8-12 additional bytes is not much additional data unless you're sending out *lots* of them (either a tiny block size or a...
Elliott Mitchell
ehem@...
Jun 8, 2004 2:38 pm
... Personally I think the idea of retrieving blocks by their hash codes is quite beautiful (I was throwing this idea around in here a month or two ago, but my...
... I'd have to disagree; the overhead in requesting by offset would be 4 bytes (2^32*16384 = 64 terabyte max torrent size), while requesting by hash would be...
John Hoffman
theshadow@...
Jun 9, 2004 4:40 pm
... Assuming you use the full hash value during the request (you need all of it to check the block, but not all of it is needed to make an unambiguous...
Elliott Mitchell
ehem@...
Jun 9, 2004 7:07 pm
... I wouldnt drop part of the hash. It might be nice if there was an option to use different hashes in bt2, but dropping it is a bad thing, as constructing...
... True. Notice though that this is a separate issue. Over the wire you just need to identify which piece you're looking for. It seems unlikely someone will...
Elliott Mitchell
ehem@...
Jun 9, 2004 10:46 pm
... You need to avoid clients that lie to try to get more bandwidth from you. I wasnt thinking of accidents. I have heard that there are ssh man in the middle...
... That's true, provided you do your requests one at a time. But if you request several blocks at once from a given peer (the normal case, I would think, for...
... fwiw, its been done in the mojonation project (a large source of inspiraton for Bram to have bittorrent to lack complexity ;) and its successors mnet &...
... As far as I can tell from a cursory scan of the mnet docs, mnet doesn't use a tree of hashes at all, and does a lot of fooling around with redundancy...
... I was unaware of those projects, but will certainly look at them now. Thanks. ... It seems to me that if the system is based on content addressability, ...
... So if you edited for example a configuration file, it's name would change and you'd have to edit the symlink to link to the new name? That doesn't sound...
... This would indeed not be a terribly convenient file system to do *everything* with; its strength would be in storing archival content. However, let's look...
... Or both. What's your definition of a tiny block size? And of a massive number of peers? ... Because it's the other peer that decides on the requests, you'd...
... Well, I've already pointed out that for 256K blocks (current default for the mainstream) and 50 peers (towards the upper end for the mainstream), the...
Elliott Mitchell
ehem@...
Jun 11, 2004 7:52 pm
... The piece/chunk distinction is done because if pieces are set to the size of chunks the size of have bitfields becomes unreasonably large. With hash trees,...
... That translation will need to be done at the final destination, but helper clients need not do that, and can thus remain oblivious. For bandwidth savings...
Elliott Mitchell
ehem@...
Jun 7, 2004 8:43 pm
... via ... with a ... sneakernet), and ... "zombie" ... reassemble ... Unless you're talking about setting up a zombie that just finds random pieces and...
Having problems with message search?
Fill out this form to ensure your group is one of the first to be migrated to the new message search system.
Offline 
Send Email