kathir-ks / indictrans2-tokenization Public

Notifications You must be signed in to change notification settings
Fork 0
Star 0

A codebase to split text into sentences and tokenize them in batches for inference using IndicTrans2

0 stars 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 32 Commits
README.md		README.md
hf_login.sh		hf_login.sh
setup_env.sh		setup_env.sh
tokenization.py		tokenization.py
tokenize_sentences.py		tokenize_sentences.py

Repository files navigation

indictrans2-tokenization

A codebase to split text into sentences and tokenize them in batches for inference using IndicTrans2

About

A codebase to split text into sentences and tokenize them in batches for inference using IndicTrans2

Report repository

Releases

No releases published

Packages

No packages published

Languages