Add slice_with_offset and dry_run Support for Tar Dataset Creation; New Script for Partial Conversion #10511
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Sasha Meister <[email protected]>
Signed-off-by: ssh-meister <[email protected]>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Signed-off-by: Sasha Meister <[email protected]>
Signed-off-by: ssh-meister <[email protected]>
pzelasko
reviewed
Sep 19, 2024
| encoded_audio = BytesIO() | ||
| if codec == "opus": | ||
| kwargs = {"format": "ogg", "subtype": "opus"} | ||
| if codec is not None: |
There was a problem hiding this comment.
Actually we could do a fast-path here: if codec=None and offset=0 and duration=None, instead of reading+writing with soundfile (triggering encoding and decoding) just read the raw audio file bytes and write them to tar file directly. Will likely speed up data tarring by a non-trivial factor. Up to you.
| def _write_to_tar( | ||
| self, tar, audio_filepath: str, squashed_filename: str, duration: float = None, offset: float = 0 | ||
| ) -> None: | ||
| if ((codec := self.config.force_codec) is None or audio_filepath.endswith(f".{codec}")) and not duration: |
There was a problem hiding this comment.
There are so many if/else here now that I think we should have some unit test coverage for this script (or just the function).
| ), | ||
| ) | ||
| parser.add_argument( | ||
| "--dry_run", |
There was a problem hiding this comment.
dry_run is not an accurate name since the script will create some output: the manifests without tar files. I suggest renaming it to --only_manifest instead.
BTW dry_run would also be a nice option, telling you very quickly how many shards with how much data per shard are going to be created without actually reading any audio or writing anything.
| """ | ||
| # Partial Tarred Audio Dataset Creator | ||
|
|
||
| ## Overview |
There was a problem hiding this comment.
I'm not sure I understand the purpose of this script. Why would you want to only tar specific shards? Can you add some comments / docs about the intended use cases?
Signed-off-by: ssh-meister <[email protected]>
Signed-off-by: Sasha Meister <[email protected]>
Signed-off-by: ssh-meister <[email protected]>
Signed-off-by: Sasha Meister <[email protected]>
|
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
|
This PR was closed because it has been inactive for 7 days since being marked as stale. |
karpnv
changed the title
|
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
|
This PR was closed because it has been inactive for 7 days since being marked as stale. |
|
beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base. Your code was analyzed with PyLint. The following annotations have been identified: Mitigation guide:
By applying these rules, we reduce the occurance of this message in future. Thank you for improving NeMo's documentation! |
|
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
|
This PR was closed because it has been inactive for 7 days since being marked as stale. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
slice_with_offsetFeature:dry_runFlag:partial_conversion_to_tarred_audio_dataset.py:Changelog
--
create_shards: Updated to support theslice_by_offset flagfor segment-based tar creation anddry_runfunctionality.-- Added new functionality to handle sharded manifests when
dry_runis enabled.--
partial_conversion_to_tarred_audio_dataset.py:Processes dry-run manifests to generate individual shards.PR Type:
Additional Information