Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fixing librispeech dataset setup script #464

Closed
wants to merge 6 commits into from
Closed

fixing librispeech dataset setup script #464

wants to merge 6 commits into from

Conversation

sourabh2k15
Copy link
Contributor

@sourabh2k15 sourabh2k15 commented Jul 27, 2023 •
edited
Loading

  1. Fixed tar extraction paths for librispeech split tar.gz files that are downloaded.
  2. Fixed tokenizer write path so it's written to the same directory that's provided.
  3. Fixed final output directory path in preprocessing script so records are written to the data directory provided.
  4. Tokenizer will be trained if there's no existing tokenizer thus eliminating the need for train_tokenizer flag

With these changes one should be able to run librispeech setup e2e by running :

python3 datasets/dataset_setup.py --data_dir=/data/ --temp_dir=/data/ --librispeech

This will result in final data being written in /data/librispeech_preprocessed/* and raw extracted data being written in /data/librispeech/Librispeech, vocab file will be written to /data/librispeech/Librispeech/spm_model.vocab

@sourabh2k15 sourabh2k15 requested a review from a team as a code owner July 27, 2023 22:52
@github-actions
Copy link

MLCommons CLA bot:
Thank you very much for your submission, we really appreciate it. Before we can accept your contribution, we ask that you sign the MLCommons CLA (Apache 2). Please use this [Google form] (https://forms.gle/Ew1KkBVpyeJDuRw67) to initiate authorization. If you are from an MLCommons member organization, we will request that you be added to the CLA. If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact [email protected].
1 out of 2 committers have signed the MLCommons CLA.
✅ @sourabh2k15
❌ @root
root seems not to be a GitHub user. You need a GitHub account after you become MLCommons member. If you have already a GitHub account, please add the email address used for this commit to your account.
You can retrigger this bot by commenting recheck in this Pull Request

@github-actions github-actions bot locked and limited conversation to collaborators Jul 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@sourabh2k15