fineweb-translation

code to translate fineweb-edu dataset released by huggingface to indian languages using indictrans2 model.

Name		Name	Last commit message	Last commit date
Latest commit History 76 Commits
Dockerfile		Dockerfile
README.md		README.md
_tokenization.py		_tokenization.py
configuration_indictrans.py		configuration_indictrans.py
custom_infra.sh		custom_infra.sh
decode.py		decode.py
inference.py		inference.py
inference_distributed.py		inference_distributed.py
modeling_flax_indictrans.py		modeling_flax_indictrans.py
parallel_infra.sh		parallel_infra.sh
setup_inference_env.sh		setup_inference_env.sh
setup_tokenization_env.sh		setup_tokenization_env.sh
tokenization.py		tokenization.py
tokenization_infra.sh		tokenization_infra.sh
tokenization_parallel.py		tokenization_parallel.py
upload_to_hub.py		upload_to_hub.py
v4_256_infra.sh		v4_256_infra.sh

Provide feedback