Stanza fails for some inputs

argosopentech · August 16, 2022, 10:07pm

github.com/argosopentech/argos-translate

Script freezes when certain input given

opened 07:03AM - 03 Aug 22 UTC

bug help wanted

I have been troubleshooting with the following script, and it appears that certa…in strings freeze the script. For example, if I run the script as it is now, `translation.translate` will never return and therefore stall the script. I have given a few examples in an effort to troubleshoot what is going on (and what input is causing it) I would expect that the translator does its best with strings that it doesn't know, but always returns! FWIW, from the lack of CPU consumption, it doesn't appear to be stuck in a loop 🤷‍♂️ ```python import argostranslate.package, argostranslate.translate from_code = "en" to_code = "de" # Translate installed_languages = argostranslate.translate.get_installed_languages() from_lang = list(filter( lambda x: x.code == from_code, installed_languages))[0] to_lang = list(filter( lambda x: x.code == to_code, installed_languages))[0] translation = from_lang.get_translation(to_lang) # translatedText = translation.translate('this test try and test the game not the player they are nearly dead. | tetser \n|') # not working # translatedText = translation.translate('this is a test try and test the game not the player they are dead.| tetser \n|') # not working # translatedText = translation.translate('this is a test try it out and let me know how it goes, works?| dlfkgh \n|') # not working # translatedText = translation.translate('this is a test try it out and let me know how it goes, works?| dlfkgh7 \n|') # WORKING # translatedText = translation.translate('this is a test try it out and let me know how it goes, works?| dlfkgh \n|') # not working # translatedText = translation.translate('this is a test try it out and let me know how it goes, works?| aaa\n|') # not working translatedText = translation.translate('try it out and let me know how it goes, works?| aaa\n|') # not working # translatedText = translation.translate('try it out let me know how it goes, works?| aaa\n|') # WORKING # translatedText = translation.translate('try it out let me know how it goes, work?| aaa\n|') # WORKING print(translatedText) ```

import argostranslate.package, argostranslate.translate

from_code = "en"
to_code = "de"

# Translate
installed_languages = argostranslate.translate.get_installed_languages()
from_lang = list(filter(
	lambda x: x.code == from_code,
	installed_languages))[0]
to_lang = list(filter(
	lambda x: x.code == to_code,
	installed_languages))[0]
translation = from_lang.get_translation(to_lang)
# translatedText = translation.translate('this test try and test the game not the player they are nearly dead. | tetser \n|') # not working
# translatedText = translation.translate('this is a test try and test the game not the player they are dead.| tetser \n|') # not working
# translatedText = translation.translate('this is a test try it out and let me know how it goes, works?| dlfkgh \n|') # not working
# translatedText = translation.translate('this is a test try it out and let me know how it goes, works?| dlfkgh7 \n|') # WORKING
# translatedText = translation.translate('this is a test try it out and let me know how it goes, works?| dlfkgh \n|') # not working
# translatedText = translation.translate('this is a test try it out and let me know how it goes, works?| aaa\n|') # not working
translatedText = translation.translate('try it out and let me know how it goes, works?| aaa\n|') # not working
# translatedText = translation.translate('try it out let me know how it goes, works?| aaa\n|') # WORKING
# translatedText = translation.translate('try it out let me know how it goes, work?| aaa\n|') # WORKING
print(translatedText)

It seems to get lost here: stanza/models/tokenize/utils.py
line 90: pred = np.argmax(trainer.predict(batch), axis=2)