Add option for fallback model when stanza doesn't exist

LibreTranslate API
,
1

@christopherpickering christopherpickering

The stanza models don’t exist for all languages, and in this case Locomotive failes with a Cannot download stanza model … It would be nice to be able to specify a fallback local (currently I hardcode it) to a similar language.

See train.py:

try:
            os.makedirs(stanza_dir, exist_ok=True)
            stanza.download(stanza_lang_code, dir=stanza_dir, processors="tokenize")
            break
2

There’s a mechanism in train.py for a fallback mechanism (it’s hard-coded), one simply needs to add to the mapping variable source lang => stanza lang to use instead, perhaps it could be useful to make it a command line parameter instead.

2 Likes
3

Also, adding

allow_unknown_language=True

to stanza.download skips the error, but probably gives a poorer output. I’m not sure what to do for languages where there is nothing very similar? Like Amharic?

Have you tried to make new stanza models? GitHub - stanfordnlp/stanza-train: Model training tutorials for the Stanza Python NLP Library

1 Like
4

No. The current goal is to move away from Stanza. Primarily because Stanza requires us to depend on PyTorch.

3 Likes