Normalisation Unification

[kaldan007]

Description:
We are able to find two different places where we able to find normalisation script. We need unify them under botok normalisation.

such as this one:
https://github.com/buda-base/tibetan-etext-tools/blob/main/DKCC/normalization.py#L31

[eroux]

yes, thanks for opening an issue about this. Here are the differences between the two:

• botok's normalization is Tibetan-specific, the normalization file you point to has functions for non-Tibetan too
• there's some subtle differences in the Tibetan normalization

I'll merge the scripts this morning

[eroux]

ok, from now on you can just use

https://github.com/OpenPecha/Botok/blob/master/botok/utils/corpus_normalization.py#L73

it does all the normalization necessary (spaces, non-Tibetan Unicode and Tibetan)

[kaldan007]

Thank you so much