A suggestion to modify the updateSegmentation.sh

[akshayc11]

If we make the assumption that the input text is going to be UTF-8 file format, I think that The following needs to be modified from

zeroth/s5/local/updateSegmentation.sh

Line 30 in 3cbb45c

morfessor -l $lmDir/zeroth_morfessor.seg -T - -o - \

morfessor -l $lmDir/zeroth_morfessor.seg -T - -o - \
to
morfessor -e 'utf-8' -l $lmDir/zeroth_morfessor.seg -T - -o - \

This is because morfessor defaults to the assumption that the text is ASCII.

[akshayc11]

Can confirm that the ascii issue may actually be there...

[jty016]

Thank you, @akshayc11. yes, I think this is good point.

According to the morfessor's help, they said.

-e <encoding>, --encoding <encoding>
    encoding of input and output files (if none is given, both the local encoding and UTF-8 are tried)

so that's why I provide locale setup at https://github.com/goodatlas/zeroth/wiki/Requirements

Anyway, I'll do a test for sure.
And If it is possible to you, plz make an official PR.