-
Notifications
You must be signed in to change notification settings - Fork 337
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Core] implement synchronous and v2 metafile creation #430
Conversation
deluge/metafile.py
Outdated
import logging | ||
import os.path | ||
import time | ||
from hashlib import sha1 as sha | ||
from hashlib import sha256 | ||
|
||
import libtorrent as lt |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cannot import libtorrent here, this burdens UI clients with libtorrent requirement
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I imported libtorrent in the corresponding function only
deluge/core/core.py
Outdated
include_v1=True, | ||
include_v2=False, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't feel it is a good idea to have these libtorrent implementation details leaking into the functions and API. It would be better to use something like a TorrentFormat Enum and have a format
param that can accept str or Enum.
A tentative implementation attempt:
class TorrentFormat(str, Enum):
V1 = 'v1_only'
V2 = 'v2_only'
HYBRID = 'hybrid'
@classmethod
def _missing_(cls, value):
if not value:
return cls.V1
value = value.lower()
for member in cls:
if member.value.startswith(value):
return member
def to_lt_flag(self):
if self.value == 'v1_only':
return 64
if self.value == 'v2_only':
return 32
return 0
This can be better represent available options to users.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did it a bit differently. The called it torrent_format everywhere to avoid clash with the builtin format
function. The format names are v1
, v2
and hybrid
(without the _only suffix). And enforce a valid format string in missing. Instead of returning v1
on invalid format.
be2894a
to
b18dbec
Compare
If target=None and add_to_session is True, the torrent will be directly added to the session without writing the torrent file to disk. This will allow to programmatically control deluge via RPC without creating .torrent files all over the place. Also, make most create_torrent parameters optional.
This allows to create a torrent file on the remote server and get its content in one call.
Add support for v2 torrents in create_torrent, but keep the old default of only adding the v1 metadata. Unify the single-file and directory cases to avoid code duplication. V2 torrents require files to be piece-aligned. The same for hybrid v1/v2 ones. To handle both cases of piece-aligned and non-aligned files, always read the files in piece-aligned chunks. Re-slice the buffer if needed (for v1-only multi-file torrents). Also, had to adapt to progress event. It now depends on the number of bytes hashed rather than the number of pieces. To avoid sending and excessive amount of event when handling a directory with many small files, add a mechanism to limit event period at 1 per piece_length.
Sorry for taking so long. Was overwhelmed by a couple of more urgent tasks. |
@cas-- , would it be possible to create a release ? (could be an -dev or an -alpha release). It would be very useful to be able to use this new features by just installing deluge from pip. |
Add support for v2 torrents in create_torrent, but keep the old default of only adding the v1 metadata. Unify the single-file and directory cases to avoid code duplication. V2 torrents require files to be piece-aligned. The same for hybrid v1/v2 ones. To handle both cases of piece-aligned and non-aligned files, always read the files in piece-aligned chunks. Re-slice the buffer if needed (for v1-only multi-file torrents). Also, had to adapt to progress event. It now depends on the number of bytes hashed rather than the number of pieces. To avoid sending and excessive amount of event when handling a directory with many small files, add a mechanism to limit event period at 1 per piece_length. Closes: deluge-torrent#430
Add support for v2 torrents in create_torrent, but keep the old default of only adding the v1 metadata. Unify the single-file and directory cases to avoid code duplication. V2 torrents require files to be piece-aligned. The same for hybrid v1/v2 ones. To handle both cases of piece-aligned and non-aligned files, always read the files in piece-aligned chunks. Re-slice the buffer if needed (for v1-only multi-file torrents). Also, had to adapt to progress event. It now depends on the number of bytes hashed rather than the number of pieces. To avoid sending and excessive amount of event when handling a directory with many small files, add a mechanism to limit event period at 1 per piece_length. Closes: deluge-torrent#430
Assuming an entity which has access to deluge daemons on multiple storage servers,
this will allow to generate a torrent file on a source server, followed by triggering
the transfer from multiple source locations towards the destination storage server.
In particular, I'm interested to add support for deluge/bittorent as a wire transfer protocol in rucio
The individual commit messages contain additional information about the changes.