BEP-56: Data compression extension

beps/bep_0056.rst

@@ -0,0 +1,145 @@
+:BEP: 56
+:Title: Data compression extension
+:Version: $Revision$
+:Last-Modified: $Date$
+:Author:  Alexander Ivanov <[email protected]>
+:Status:  Draft
+:Type:    Standards Track
+:Created: 31-Sep-2021
+:Post-History: 
+
+Abstract
+========
+This extension adds a capability for clients to negotiate and use
+compression methods for data streams or torrent pieces, effectively
+improving bandwidth for supporting clients.
+
+Rationale
+=========
+This extension would allow clients to download files faster, without
+using file archivers. Since large files are often pre-compressed before
+torrent creation, downloaders needs to keep both the archives
+(for seeding) and uncompressed files (for own usage).
+
+Most users prefer to remove such torrents, thus harming proper file
+distribution. For example: Organizations using Bittorrent for software
+distribution needs to have centralized storage for new customers, no
+matter how many customers have the same software already.
+
+Extension header
+================
+
+This extension uses the extension protocol (specified in `BEP 0010`_)
+to advertise client capability of using chunk compression. It defines
+following items in the extension protocol handshake message:
++-------+-----------------------------------------------------------+
+| name  | description                                               |
++=======+===========================================================+
+| c     | Dictionary of supported compression algorithms which maps |
+|       | its identifiers to its priority (unsigned 8-bit integer), |
+|       | clients can adjust it based on compression speed/ratio,   |
+|       | hardware support, performance, and power mode et cetera.  |
+|       | Priority set to zero means that the compression algorithm |
+|       | is not supported or disabled by user, the client must     |
+|       | ignore unknown algorithms.                                |
++-------+-----------------------------------------------------------+
+
+
+
+The compression algorithm is selected by taking the dictionary item with
+highest priority from intersection of items supported by both peers,
+if there isn't any suitable compression algorithm - compression will be disabled.
+
+Example of extension handshake message:
+
+::
+
+  {
+    'c': {
+      'p_zstd': 255,
+      's_zstd': 153,
+      'p_lz4': 106,
+      'p_density': 70,
+      's_lz4': 41,
+      's_density': 37
+    }
+  }
+
+
+Compression methods
+===================
+Extension provides two approaches (methods) to compression, which have
+their own trade-offs, so choice between these should be made by clients
+on per-torrent basis, using its metadata (properties like piece size).
+
+With **by-piece compression** method, client must compress each piece
+individually, which lowers overall compression ratio but result can
+be stored in cache and reused, probably providing more efficiency.
+If the client is caching compressed pieces in memory, then it can be
+decompressed when saving to disk or sending to peer, which not supports
+compression. To reduce piece re-compression, client should raise
+current algorithm's priority during handshake. This method has low
+efficiency with pieces smaller than 4 MB.
+
+Clients using **stream compression** method instead compresses whole
+data stream, so compression ratio should be higher. During handshake,
+clients should lower or raise algorithm's priority depending on expected
+factors that could impact compression efficiency and performance. This
+method can introduce performance issues if used on thousands of
+simultaneous connections.
+
+Allowed compression algorithms
+------------------------------
+
+Compression algorithms must satisfy the following requirements:
+
+1. Decompression speed must not be lower than 500 MB/s.
+
+2. It must not produce a larger piece than the original by 1%.
+
+For consistency, identifiers are prefixed by ``p_`` or ``s_``
+for "piece" and "stream" compression methods accordingly.
+
++-------------+-----------------------------+
+| identifier  | compression algorithm       |
++=============+=============================+
+| p_lz4       | LZ4                         |
++-------------+-----------------------------+
+| s_lz4       | LZ4                         |
++-------------+-----------------------------+
+| p_density   | Chameleon (DENSITY library) |
++-------------+-----------------------------+
+| s_density   | Chameleon (DENSITY library) |
++-------------+-----------------------------+
+| p_zstd      | ZStandard                   |
++-------------+-----------------------------+
+| s_zstd      | ZStandard                   |
++-------------+-----------------------------+
+
+This specification deliberately doesn't provide negotiation
+for configuration options, default ones must be used unless
+specified otherwise.
+
+**NOTE**: Currently, only ``p_zstd`` and ``s_zstd`` algorithms
+are required for implementation.
+
+References
+==========
+
+.. _`BEP 0010`: http://www.bittorrent.org/beps/bep_0010.html
+
+
+Copyright
+=========
+
+This document has been placed in the public domain.
+
+
+..
+   Local Variables:
+   mode: indented-text
+   indent-tabs-mode: nil
+   sentence-end-double-space: t
+   fill-column: 70
+   coding: utf-8
+   End: