The problem of translating symbols OOV (out of the vocabulary)

I think there may be a problem with this change. It looks correct to me but I’m getting weird behavior from the detokenizer.

This is the behavior I see using the current master branch:
Argos Translate screenshot with the detokenizer working incorrectly

The underscores aren’t correctly being detokenized into spaces.

This pull request has been merged but hasn’t been released to PyPI yet so fixing this issue isn’t urgent.

1 Like