Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] implement synchronous and v2 metafile creation #430

Closed
wants to merge 3 commits into from

Conversation

rcarpa
Copy link
Contributor

@rcarpa rcarpa commented Aug 21, 2023

Assuming an entity which has access to deluge daemons on multiple storage servers,
this will allow to generate a torrent file on a source server, followed by triggering
the transfer from multiple source locations towards the destination storage server.

In particular, I'm interested to add support for deluge/bittorent as a wire transfer protocol in rucio

The individual commit messages contain additional information about the changes.

deluge/core/core.py Outdated Show resolved Hide resolved
import logging
import os.path
import time
from hashlib import sha1 as sha
from hashlib import sha256

import libtorrent as lt
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cannot import libtorrent here, this burdens UI clients with libtorrent requirement

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I imported libtorrent in the corresponding function only

Comment on lines 1002 to 1003
include_v1=True,
include_v2=False,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't feel it is a good idea to have these libtorrent implementation details leaking into the functions and API. It would be better to use something like a TorrentFormat Enum and have a format param that can accept str or Enum.

A tentative implementation attempt:

class TorrentFormat(str, Enum):
     V1 = 'v1_only'
     V2 = 'v2_only'
     HYBRID = 'hybrid'

     @classmethod
     def _missing_(cls, value):
         if not value:
             return cls.V1

         value = value.lower()
         for member in cls:
             if member.value.startswith(value):
                return member

     def to_lt_flag(self):
         if self.value == 'v1_only':
             return 64
         if self.value == 'v2_only':
             return 32
         return 0

This can be better represent available options to users.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did it a bit differently. The called it torrent_format everywhere to avoid clash with the builtin format function. The format names are v1, v2 and hybrid (without the _only suffix). And enforce a valid format string in missing. Instead of returning v1 on invalid format.

deluge/metafile.py Outdated Show resolved Hide resolved
deluge/tests/test_core.py Outdated Show resolved Hide resolved
deluge/tests/test_metafile.py Outdated Show resolved Hide resolved
deluge/tests/test_metafile.py Outdated Show resolved Hide resolved
deluge/tests/test_core.py Outdated Show resolved Hide resolved
If target=None and add_to_session is True, the torrent will be
directly added to the session without writing the torrent file
to disk. This will allow to programmatically control deluge via
RPC without creating .torrent files all over the place.

Also, make most create_torrent parameters optional.
This allows to create a torrent file on the remote server
and get its content in one call.
Add support for v2 torrents in create_torrent, but keep the old
default of only adding the v1 metadata.

Unify the single-file and directory cases to avoid code
duplication.

V2 torrents require files to be piece-aligned. The same for
hybrid v1/v2 ones. To handle both cases of piece-aligned and
non-aligned files, always read the files in piece-aligned
chunks. Re-slice the buffer if needed (for v1-only multi-file
torrents).

Also, had to adapt to progress event. It now depends on the
number of bytes hashed rather than the number of pieces. To
avoid sending and excessive amount of event when handling a
directory with many small files, add a mechanism to limit
event period at 1 per piece_length.
@rcarpa
Copy link
Contributor Author

rcarpa commented Oct 31, 2023

Sorry for taking so long. Was overwhelmed by a couple of more urgent tasks.

@rcarpa
Copy link
Contributor Author

rcarpa commented Dec 18, 2023

@cas-- , would it be possible to create a release ? (could be an -dev or an -alpha release). It would be very useful to be able to use this new features by just installing deluge from pip.

doadin pushed a commit to doadin/deluge that referenced this pull request Jun 4, 2024
Add support for v2 torrents in create_torrent, but keep the old
default of only adding the v1 metadata.

Unify the single-file and directory cases to avoid code
duplication.

V2 torrents require files to be piece-aligned. The same for
hybrid v1/v2 ones. To handle both cases of piece-aligned and
non-aligned files, always read the files in piece-aligned
chunks. Re-slice the buffer if needed (for v1-only multi-file
torrents).

Also, had to adapt to progress event. It now depends on the
number of bytes hashed rather than the number of pieces. To
avoid sending and excessive amount of event when handling a
directory with many small files, add a mechanism to limit
event period at 1 per piece_length.

Closes: deluge-torrent#430
doadin pushed a commit to doadin/deluge that referenced this pull request Sep 21, 2024
Add support for v2 torrents in create_torrent, but keep the old
default of only adding the v1 metadata.

Unify the single-file and directory cases to avoid code
duplication.

V2 torrents require files to be piece-aligned. The same for
hybrid v1/v2 ones. To handle both cases of piece-aligned and
non-aligned files, always read the files in piece-aligned
chunks. Re-slice the buffer if needed (for v1-only multi-file
torrents).

Also, had to adapt to progress event. It now depends on the
number of bytes hashed rather than the number of pieces. To
avoid sending and excessive amount of event when handling a
directory with many small files, add a mechanism to limit
event period at 1 per piece_length.

Closes: deluge-torrent#430
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants