Help Wanted: Improve en-de translation

[lynxpda]

@NicoLe

Did you update OpenNMT-py and CTranslate2 to the last available versions in the course of your modifications (maybe to turn RoPE on)?
OpenNMT has implemented flash attention by default in the last months, that speeds up cycles and optimizes memory quite a lot.
They also implemented cuda12 if you have it installed (I do).
This is readily operable, only requires a change in the requirements.txt and pip upgrade.

No, haven’t updated it, but thanks, will definitely try it now.

@NicoLe

Also, the byte fallback option is for translating as is anything that comes OOV, isn’t it?

That’s right, I described in more detail here on the byte_fallback option:

The problem of translating symbols OOV (out of the vocabulary)

When using Libretranslate, I noticed this problem: if some characters are not in the vocabulary, they are simply absent in the translated text. This often spoils the translation. For example: Source text: Origami (折り紙, Japanese pronunciation: [oɾiɡami] or [oɾiꜜɡami], from ori meaning “folding” , and kami meaning “paper” (kami changes to gami due to rendaku)) is the Japanese art of paper folding. In modern usage, the word “origami” is often used as an inclusive term for all folding practices, …

@NicoLe

On the issue of hyperparameters, once implemented correctly, silu/swiglu and geglu have pretty much the same effects as deepening on the COMET score. I still have to see what they actually yield in practice.

Very interesting! I look forward to seeing your results.

@NicoLe

I am more skeptical about RPE. Upon second review, I was told it had shown no overall improvement. Since RPE 20 or 32 only improved COMET significantly at an early stage of training, and trained more slowly, I think of either doing without it or using RoPE, but for that, I need to have a configuration that’s as stable as possible.

Interesting, thanks for sharing. Yes, I noticed that the speed of RPE is worse than PE.
In my experience RoPE is about the same speed as PE.