Releasing the extended package to PyPi
because my PR was not accepted by original repo https://github.com/alvations/sacremoses/pull/103
And I have used this version of tokenizer in my models.
Releasing the extended package to PyPi
because my PR was not accepted by original repo https://github.com/alvations/sacremoses/pull/103
And I have used this version of tokenizer in my models.