Firefox Translation Models

pierotofy · November 28, 2024, 5:10am

Wonder if anyone has tried to run these and has any feedback on speed/quality?

Btw, was quite delighted to see argos in the benchmark numbers at firefox-translations-models/evaluation/bleu-results.md at main · mozilla/firefox-translations-models · GitHub

pierotofy · November 28, 2024, 11:13pm

I managed to get these running.

Requirement: Python <= 3.10 (wheels are not available for newer versions)

pip install bergamot

Translation code:

import bergamot

config = bergamot.ServiceConfig(numWorkers=4)
service = bergamot.Service(config)
model = service.modelFromConfigPath("bergamot.config.yml")
options = bergamot.ResponseOptions(
    alignment=False, qualityScores=False, HTML=False
)
response = service.translate(model, bergamot.VectorString([
    "Ovechkin’s first assist of the night was on the game-winning goal by rookie Nicklas Backstrom", 
    "In the last 3 months, over 80 arrestees were released from the Central Booking facility without being formally charged.",
    "Since its inception, The Onion has become a veritable news parody empire, with a print edition, a website that drew 5,000,000 unique visitors in the month of October, personal ads, a 24 hour news network, podcasts, and a recently launched world atlas called Our Dumb World.",
    "The hostel’s guests were mostly citizens of the United Arab Emirates.",
    "It was developed by John Smith in the 1970s to help inexperienced folders or those with limited motor skills.",
    "When people don’t see moose as potentially dangerous, they may approach too closely and put themselves at risk.",

]), options)

for r in response:
    print(r.target.text)

bergamot.config.yml:


# These Marian options are set according to
# https://github.com/mozilla/firefox-translations/blob/main/extension/controller/translation/translationWorker.js
# to imitate production setting

# For reference, see https://github.com/mozilla/firefox-translations-models/blob/main/evals/translators/bergamot.sh

bergamot-mode: wasm
models:
  - ./model.enit.intgemm.alphas.bin
vocabs:
  - ./vocab.enit.spm
  - ./vocab.enit.spm
shortlist:
    - ./lex.50.50.enit.s2t.bin
    - false
beam-size: 1
normalize: 1.0
word-penalty: 0
max-length-break: 128
mini-batch-words: 1024
workspace: 128
max-length-factor: 2.0
skip-cost: true
cpu-threads: 4
quiet: false
quiet-translation: false
gemm-precision: int8shiftAlphaAll
alignment: soft

Model files downloaded and extracted from firefox-translations-models/models/prod/enit at main · mozilla/firefox-translations-models · GitHub

Il primo assist di Ovechkin della notte è stato sul gol vincente del rookie Nicklas Backstrom
Negli ultimi 3 mesi, oltre 80 arrestati sono stati rilasciati dalla struttura di prenotazione centrale senza essere formalmente accusati.
Fin dalla sua nascita, The Onion è diventato un vero e proprio impero di parodia di notizie, con un'edizione cartacea, un sito web che ha attirato 5.000.000 di visitatori unici nel mese di ottobre, annunci personali, una rete di notizie 24 ore, podcast e un atlante mondiale lanciato di recente chiamato Our Dumb World.
Gli ospiti dell'ostello erano per lo più cittadini degli Emirati Arabi Uniti.
È stato sviluppato da John Smith negli anni '70 per aiutare le cartelle inesperte o quelle con capacità motorie limitate.
Quando le persone non vedono il alce come potenzialmente pericoloso, possono avvicinarsi troppo da vicino e mettersi a rischio.

argosopentech · November 29, 2024, 12:03pm

Neat! Argos Translate’s BLEU and COMET scores seems to be competitive with but generally lower than Google and Microsoft’s.

COMET Scores

Translator/Dataset	en-af	cs-en	tr-en	ar-en	sk-en	en-cy	en-et	en-fi	en-it	en-bn	en-gl	en-hy	en-hr	fr-en	en-pt	ro-en	hu-en	th-en	en-zh	vi-en	en-ja	ru-en	mt-en	en-sr	en-nb	en-hu	en-el	el-en	fi-en	en-da	en-nl	en-mt	en-sv	en-am	lv-en	en-ar	en-th	en-ru	en-fa	cy-en	bs-en	et-en	nb-en	en-ca	bn-en	nl-en	en-lt	en-tr	bg-en	ko-en	uk-en	hy-en	en-es	fa-en	ca-en	en-uk	en-vi	is-en	en-bg	en-lv	gl-en	zh-en	en-hi	en-ig	en-bs	en-cs	hr-en	en-ro	sr-en	sv-en	en-my	en-ko	hi-en	af-en	en-is	id-en	de-en	da-en	it-en	pl-en	am-en	sl-en	en-fr	en-pl	en-id	my-en	pt-en	en-sk	ig-en	es-en	en-sl	ja-en	lt-en	en-de
bergamot	N/A	0.83	0.85	N/A	0.86	N/A	0.87	0.88	0.86	N/A	N/A	N/A	0.86	0.84	0.89	0.87	0.84	N/A	N/A	0.85	N/A	0.83	0.76	N/A	N/A	0.84	0.87	0.85	0.86	0.89	0.86	N/A	0.89	N/A	0.84	N/A	N/A	0.87	0.79	N/A	0.82	0.85	0.85	0.86	N/A	0.86	0.84	0.87	0.87	N/A	0.83	N/A	0.83	0.83	0.85	0.84	N/A	0.73	0.89	0.85	N/A	0.80	N/A	N/A	N/A	0.85	0.86	0.88	0.85	0.89	N/A	N/A	N/A	N/A	N/A	0.87	0.83	0.89	0.85	0.83	N/A	0.85	0.82	0.86	0.90	N/A	0.87	0.86	N/A	0.83	0.86	N/A	0.84	0.84
google	0.87	0.85 (+0.02, +2.97%)	0.89 (+0.04, +4.71%)	0.88	0.89 (+0.03, +3.46%)	0.88	0.91 (+0.04, +4.95%)	0.91 (+0.03, +3.46%)	0.88 (+0.02, +2.33%)	0.87	0.87	0.90	0.90 (+0.05, +5.37%)	0.86 (+0.02, +1.87%)	0.90 (+0.02, +1.73%)	0.89 (+0.02, +1.89%)	0.86 (+0.03, +3.01%)	0.88	0.88	0.88 (+0.03, +3.67%)	0.91	0.85 (+0.02, +2.65%)	0.84 (+0.08, +10.71%)	0.90	0.90	0.88 (+0.05, +5.48%)	0.90 (+0.02, +2.39%)	0.87 (+0.02, +2.71%)	0.89 (+0.03, +3.42%)	0.91 (+0.02, +2.65%)	0.88 (+0.02, +2.56%)	0.73	0.91 (+0.02, +2.13%)	0.88	0.87 (+0.03, +3.55%)	0.88	0.89	0.90 (+0.03, +3.09%)	0.88 (+0.09, +11.26%)	0.90	0.89 (+0.08, +9.59%)	0.89 (+0.04, +5.14%)	0.89 (+0.04, +5.23%)	0.88 (+0.01, +1.71%)	0.89	0.88 (+0.02, +2.11%)	0.89 (+0.05, +6.46%)	0.90 (+0.04, +4.17%)	0.89 (+0.02, +2.29%)	0.87	0.87 (+0.04, +4.24%)	0.89	0.85 (+0.02, +2.40%)	0.89 (+0.06, +7.07%)	0.89 (+0.05, +5.32%)	0.90 (+0.06, +6.58%)	0.90	0.87 (+0.14, +18.68%)	0.92 (+0.02, +2.53%)	0.90 (+0.05, +5.58%)	0.89	0.84 (+0.04, +5.15%)	0.83	0.72	0.92	0.89 (+0.04, +4.62%)	0.88 (+0.02, +2.71%)	0.90 (+0.02, +2.32%)	0.88 (+0.03, +4.10%)	0.90 (+0.01, +1.19%)	0.87	0.89	0.90	0.90	0.86	0.90 (+0.02, +2.66%)	0.86 (+0.03, +3.37%)	0.91 (+0.01, +1.54%)	0.86 (+0.02, +1.85%)	0.86 (+0.02, +2.90%)	0.88	0.88 (+0.03, +3.80%)	0.85 (+0.03, +3.51%)	0.90 (+0.04, +4.88%)	0.92 (+0.02, +2.26%)	0.87	0.89 (+0.01, +1.53%)	0.92 (+0.06, +6.76%)	0.74	0.84 (+0.01, +1.75%)	0.91 (+0.05, +5.74%)	0.85	0.87 (+0.03, +3.44%)	0.87 (+0.03, +3.42%)
microsoft	0.87	0.85 (+0.03, +3.35%)	0.89 (+0.04, +4.50%)	0.87	0.89 (+0.03, +3.22%)	0.88	0.91 (+0.04, +4.67%)	0.92 (+0.04, +4.41%)	0.88 (+0.02, +2.42%)	0.87	0.86	0.88	0.89 (+0.03, +3.89%)	0.86 (+0.02, +2.43%)	0.90 (+0.01, +1.28%)	0.89 (+0.02, +2.12%)	0.87 (+0.03, +3.29%)	0.88	0.86	0.88 (+0.03, +3.36%)	0.90	0.85 (+0.03, +3.05%)	0.83 (+0.07, +9.25%)	0.75	0.90	0.89 (+0.05, +5.92%)	0.90 (+0.02, +2.52%)	0.87 (+0.02, +2.61%)	0.89 (+0.03, +3.83%)	0.91 (+0.02, +2.54%)	0.88 (+0.02, +2.12%)	0.74	0.92 (+0.03, +3.01%)	0.87	0.86 (+0.02, +2.64%)	0.86	0.88	0.89 (+0.02, +2.00%)	0.82 (+0.02, +3.03%)	0.89	0.89 (+0.07, +8.61%)	0.88 (+0.04, +4.25%)	0.89 (+0.04, +5.13%)	0.88 (+0.02, +2.31%)	0.89	0.87 (+0.02, +1.81%)	0.89 (+0.05, +5.97%)	0.90 (+0.03, +3.82%)	0.88 (+0.01, +1.63%)	0.87	0.86 (+0.03, +3.54%)	0.85	0.86 (+0.02, +2.65%)	0.87 (+0.04, +4.79%)	0.89 (+0.04, +4.92%)	0.89 (+0.05, +5.48%)	0.89	0.86 (+0.13, +17.94%)	0.91 (+0.02, +1.79%)	0.89 (+0.04, +5.23%)	0.87	0.84 (+0.04, +4.61%)	0.81	0.71	0.91	0.90 (+0.04, +5.07%)	0.88 (+0.02, +2.46%)	0.90 (+0.02, +2.76%)	0.68 (-0.17, -19.68%)	0.91 (+0.02, +2.06%)	0.86	0.89	0.89	0.89	0.85	0.90 (+0.02, +2.63%)	0.86 (+0.03, +4.16%)	0.91 (+0.01, +1.62%)	0.86 (+0.02, +1.97%)	0.86 (+0.03, +3.13%)	0.87	0.88 (+0.03, +3.71%)	0.86 (+0.03, +4.15%)	0.89 (+0.04, +4.14%)	0.92 (+0.01, +1.47%)	0.84	0.89 (+0.01, +1.43%)	0.91 (+0.05, +6.25%)	0.67	0.85 (+0.02, +2.34%)	0.89 (+0.03, +3.98%)	0.84	0.86 (+0.02, +2.32%)	0.87 (+0.03, +3.72%)
argos	N/A	0.71 (-0.12, -14.20%)	0.76 (-0.09, -10.40%)	0.81	0.78 (-0.08, -9.22%)	N/A	0.86 (-0.02, -1.73%)	0.85 (-0.03, -3.43%)	0.84 (-0.02, -2.57%)	N/A	N/A	N/A	N/A	0.83 (-0.01, -1.00%)	0.86 (-0.02, -2.62%)	N/A	0.66 (-0.18, -21.30%)	N/A	0.80	N/A	0.75	0.78 (-0.05, -6.14%)	N/A	N/A	N/A	0.78 (-0.06, -7.20%)	0.87 (-0.00, -0.08%)	0.85 (+0.00, +0.24%)	0.79 (-0.07, -7.61%)	0.83 (-0.06, -6.75%)	0.85 (-0.01, -1.73%)	N/A	0.87 (-0.02, -2.68%)	N/A	N/A	0.78	N/A	0.84 (-0.03, -3.61%)	0.80 (+0.01, +1.43%)	N/A	N/A	0.86 (+0.01, +1.63%)	0.87 (+0.02, +2.10%)	0.77 (-0.09, -10.72%)	N/A	0.86 (+0.00, +0.37%)	N/A	0.78 (-0.09, -10.09%)	0.85 (-0.01, -1.37%)	0.72	0.76 (-0.07, -8.54%)	N/A	0.82 (-0.01, -1.52%)	0.82 (-0.01, -0.64%)	0.87 (+0.02, +2.23%)	0.70 (-0.15, -17.61%)	N/A	N/A	0.88 (-0.01, -1.47%)	N/A	N/A	0.79 (-0.01, -1.66%)	0.73	N/A	N/A	0.68 (-0.18, -20.82%)	N/A	N/A	N/A	0.88 (-0.01, -1.05%)	N/A	0.71	0.82	N/A	N/A	0.81 (-0.07, -7.90%)	0.82 (-0.01, -1.29%)	0.84 (-0.05, -5.24%)	0.83 (-0.01, -1.44%)	0.84 (+0.00, +0.36%)	N/A	N/A	0.81 (-0.01, -1.47%)	0.84 (-0.02, -1.84%)	0.83 (-0.07, -8.26%)	N/A	0.87 (-0.01, -0.79%)	0.72 (-0.14, -16.08%)	N/A	0.82 (-0.01, -0.93%)	N/A	0.72	N/A	0.79 (-0.05, -5.97%)
nllb	0.86	0.73 (-0.10, -11.82%)	0.77 (-0.08, -9.17%)	0.85	0.76 (-0.10, -11.31%)	0.80	0.83 (-0.04, -4.59%)	0.85 (-0.03, -3.84%)	0.86 (+0.00, +0.21%)	0.85	0.86	0.87	0.84 (-0.01, -1.70%)	0.83 (-0.01, -0.77%)	0.88 (-0.01, -0.61%)	0.80 (-0.07, -8.21%)	0.70 (-0.14, -16.93%)	0.85	0.80	0.84 (-0.01, -1.19%)	0.85	0.83 (+0.01, +0.74%)	0.62 (-0.14, -17.90%)	0.83	0.87	0.83 (-0.01, -1.36%)	0.87 (-0.01, -0.67%)	0.85 (+0.00, +0.48%)	0.65 (-0.20, -23.75%)	0.88 (-0.01, -0.68%)	0.85 (-0.01, -0.76%)	0.69	0.88 (-0.01, -1.47%)	0.87	0.67 (-0.17, -20.28%)	0.83	0.80	0.86 (-0.01, -1.63%)	0.84 (+0.05, +5.81%)	0.57	0.75 (-0.06, -7.89%)	0.59 (-0.25, -30.00%)	0.76 (-0.09, -10.49%)	0.85 (-0.01, -1.49%)	0.87	0.81 (-0.05, -5.79%)	0.81 (-0.03, -3.68%)	0.86 (-0.01, -0.97%)	0.86 (-0.01, -1.17%)	0.84	0.82 (-0.02, -2.03%)	0.85	0.84 (+0.01, +0.91%)	0.83 (+0.00, +0.16%)	0.83 (-0.02, -2.68%)	0.83 (-0.02, -1.94%)	0.86	0.63 (-0.10, -13.68%)	0.88 (-0.01, -1.54%)	0.73 (-0.12, -13.82%)	0.83	0.79 (-0.01, -1.63%)	0.79	0.71	0.87	0.84 (-0.01, -1.18%)	0.73 (-0.13, -14.74%)	0.87 (-0.01, -0.72%)	0.81 (-0.03, -4.08%)	0.77 (-0.11, -12.76%)	0.85	0.86	0.88	0.76	0.79	0.86 (-0.01, -1.38%)	0.75 (-0.08, -9.91%)	0.78 (-0.11, -12.31%)	0.82 (-0.03, -3.55%)	0.77 (-0.06, -7.70%)	0.85	0.68 (-0.17, -19.51%)	0.84 (+0.01, +1.81%)	0.84 (-0.01, -1.50%)	0.90 (-0.01, -0.67%)	0.84	0.84 (-0.04, -4.16%)	0.86 (-0.00, -0.16%)	0.66	0.83 (-0.00, -0.10%)	0.84 (-0.02, -2.19%)	0.77	0.67 (-0.17, -20.10%)	0.83 (-0.01, -1.09%)
opusmt	0.82	0.82 (-0.00, -0.55%)	0.85 (-0.00, -0.05%)	0.84	0.86 (+0.00, +0.33%)	0.51	0.86 (-0.01, -1.30%)	0.91 (+0.03, +2.96%)	0.85 (-0.01, -0.77%)	N/A	0.66	0.68	N/A	0.85 (+0.01, +0.72%)	N/A	N/A	0.84 (+0.00, +0.58%)	0.82	0.79	0.82 (-0.03, -3.83%)	N/A	0.82 (-0.01, -1.36%)	N/A	N/A	N/A	0.84 (+0.00, +0.51%)	0.86 (-0.01, -1.17%)	N/A	0.86 (+0.01, +0.63%)	0.88 (-0.01, -1.02%)	0.85 (-0.01, -1.22%)	N/A	0.89 (-0.00, -0.21%)	N/A	0.84 (-0.00, -0.02%)	0.83	N/A	0.84 (-0.03, -3.32%)	N/A	0.52	N/A	0.86 (+0.02, +2.05%)	N/A	0.77 (-0.10, -11.04%)	0.79	0.86 (+0.00, +0.43%)	N/A	N/A	0.86 (-0.00, -0.52%)	0.82	0.80 (-0.04, -4.21%)	0.66	0.84 (+0.01, +1.12%)	N/A	0.78 (-0.06, -7.63%)	0.79 (-0.05, -5.86%)	0.76	0.78 (+0.05, +7.09%)	0.82 (-0.07, -7.86%)	N/A	0.72	0.79 (-0.01, -1.20%)	0.60	0.68	N/A	0.84 (-0.01, -1.23%)	N/A	0.86 (-0.01, -1.57%)	N/A	0.89 (+0.00, +0.50%)	N/A	N/A	0.75	0.81	0.74	0.86 (-0.02, -1.73%)	0.84 (+0.02, +1.83%)	0.89 (+0.00, +0.02%)	0.85 (+0.00, +0.27%)	0.83 (-0.00, -0.23%)	N/A	N/A	0.83 (+0.01, +1.43%)	N/A	0.88 (-0.03, -2.95%)	N/A	N/A	0.86 (-0.00, -0.26%)	0.52	0.84 (+0.01, +0.97%)	N/A	0.72	N/A	0.83 (-0.01, -1.31%)

BLEU Scores

Translator/Dataset	en-af	cs-en	tr-en	ar-en	sk-en	en-cy	en-et	en-fi	en-it	en-bn	en-gl	en-hy	en-hr	fr-en	en-pt	ro-en	hu-en	th-en	en-zh	vi-en	en-ja	ru-en	mt-en	en-sr	en-nb	en-hu	en-el	el-en	fi-en	en-da	en-nl	en-mt	en-sv	en-am	lv-en	en-ar	en-th	en-ru	en-fa	cy-en	bs-en	et-en	nb-en	en-ca	bn-en	nl-en	en-lt	en-tr	bg-en	ko-en	uk-en	hy-en	en-es	fa-en	ca-en	en-uk	en-vi	is-en	en-bg	en-lv	gl-en	zh-en	en-hi	en-ig	en-bs	en-cs	hr-en	en-ro	sr-en	sv-en	en-my	en-ko	hi-en	af-en	en-is	id-en	de-en	da-en	it-en	pl-en	am-en	sl-en	en-fr	en-pl	en-id	my-en	pt-en	en-sk	ig-en	es-en	en-sl	ja-en	lt-en	en-de
bergamot	N/A	31.56	30.08	N/A	34.50	N/A	25.50	24.01	29.77	N/A	N/A	N/A	28.80	35.43	49.85	40.03	26.53	N/A	N/A	33.00	N/A	33.21	42.70	N/A	N/A	21.57	26.90	32.47	29.44	45.55	27.30	N/A	45.45	N/A	29.10	N/A	N/A	31.93	17.30	N/A	31.40	32.37	37.60	42.20	N/A	29.65	22.83	25.42	38.50	N/A	35.57	N/A	32.41	28.70	38.00	27.23	N/A	23.40	42.10	27.07	N/A	26.60	N/A	N/A	N/A	25.85	34.10	37.97	37.25	46.40	N/A	N/A	N/A	N/A	N/A	39.05	33.16	45.80	32.60	27.87	N/A	30.65	36.01	22.27	46.65	N/A	44.87	32.25	N/A	32.38	29.15	N/A	30.87	32.01
google	38.95	33.36 (+1.79, +5.68%)	39.58 (+9.50, +31.58%)	43.03	43.05 (+8.55, +24.78%)	43.20	28.60 (+3.10, +12.16%)	27.43 (+3.41, +14.22%)	28.97 (-0.80, -2.69%)	20.80	37.60	25.50	33.90 (+5.10, +17.71%)	37.81 (+2.38, +6.70%)	53.75 (+3.90, +7.82%)	44.53 (+4.50, +11.24%)	31.30 (+4.77, +18.00%)	30.95	45.31	40.55 (+7.55, +22.88%)	33.00	38.20 (+4.99, +15.03%)	59.15 (+16.45, +38.52%)	40.00	35.05	24.80 (+3.23, +14.95%)	29.55 (+2.65, +9.85%)	37.13 (+4.67, +14.37%)	33.20 (+3.76, +12.76%)	49.70 (+4.15, +9.11%)	29.30 (+2.00, +7.33%)	40.60	48.15 (+2.70, +5.94%)	15.05	36.80 (+7.70, +26.46%)	28.70	15.60	34.49 (+2.57, +8.04%)	27.80 (+10.50, +60.69%)	62.55	45.65 (+14.25, +45.38%)	35.80 (+3.43, +10.61%)	42.05 (+4.45, +11.84%)	35.70 (-6.50, -15.40%)	39.45	33.05 (+3.40, +11.47%)	28.57 (+5.73, +25.11%)	25.52 (+0.10, +0.39%)	41.30 (+2.80, +7.27%)	29.47	42.43 (+6.87, +19.31%)	43.50	34.74 (+2.32, +7.17%)	40.85 (+12.15, +42.33%)	48.95 (+10.95, +28.82%)	32.63 (+5.40, +19.83%)	45.55	38.90 (+15.50, +66.24%)	44.60 (+2.50, +5.94%)	33.77 (+6.70, +24.75%)	43.50	32.29 (+5.69, +21.39%)	33.43	15.30	36.45	27.72 (+1.87, +7.24%)	40.60 (+6.50, +19.06%)	40.60 (+2.63, +6.94%)	47.55 (+10.30, +27.65%)	50.50 (+4.10, +8.84%)	3.75	29.20	42.13	58.85	30.70	47.75 (+8.70, +22.28%)	35.65 (+2.49, +7.52%)	50.40 (+4.60, +10.04%)	34.50 (+1.90, +5.83%)	31.23 (+3.37, +12.08%)	40.50	38.75 (+8.10, +26.43%)	29.47 (-6.54, -18.15%)	25.50 (+3.23, +14.52%)	49.25 (+2.60, +5.57%)	31.05	46.60 (+1.73, +3.86%)	40.10 (+7.85, +24.34%)	31.25	33.64 (+1.27, +3.91%)	35.40 (+6.25, +21.44%)	28.08	36.87 (+6.00, +19.44%)	33.16 (+1.14, +3.58%)
microsoft	40.70	34.67 (+3.11, +9.86%)	39.16 (+9.08, +30.19%)	42.07	41.95 (+7.45, +21.59%)	54.90	28.47 (+2.97, +11.63%)	29.27 (+5.26, +21.89%)	32.30 (+2.53, +8.51%)	20.65	35.10	18.85	32.43 (+3.63, +12.62%)	39.13 (+3.70, +10.44%)	50.15 (+0.30, +0.60%)	45.53 (+5.50, +13.74%)	31.03 (+4.50, +16.97%)	31.25	43.42	39.30 (+6.30, +19.09%)	30.72	38.38 (+5.17, +15.57%)	56.75 (+14.05, +32.90%)	1.00	34.95	26.05 (+4.48, +20.74%)	29.95 (+3.05, +11.34%)	37.33 (+4.87, +14.99%)	35.09 (+5.64, +19.17%)	49.65 (+4.10, +9.00%)	28.80 (+1.50, +5.49%)	44.20	50.00 (+4.55, +10.01%)	9.05	32.67 (+3.57, +12.26%)	26.83	15.05	33.62 (+1.70, +5.32%)	20.50 (+3.20, +18.50%)	61.65	42.65 (+11.25, +35.83%)	36.17 (+3.80, +11.74%)	42.90 (+5.30, +14.10%)	45.25 (+3.05, +7.23%)	37.95	32.60 (+2.95, +9.95%)	26.10 (+3.27, +14.31%)	32.18 (+6.76, +26.59%)	41.20 (+2.70, +7.01%)	29.50	42.30 (+6.73, +18.93%)	32.30	33.76 (+1.35, +4.17%)	36.15 (+7.45, +25.96%)	46.50 (+8.50, +22.37%)	32.03 (+4.80, +17.63%)	44.95	38.17 (+14.77, +63.11%)	38.55 (-3.55, -8.43%)	33.20 (+6.13, +22.66%)	40.20	32.09 (+5.49, +20.63%)	33.73	17.65	32.65	28.26 (+2.41, +9.33%)	38.50 (+4.40, +12.90%)	42.07 (+4.10, +10.80%)	1.60 (-35.65, -95.70%)	51.90 (+5.50, +11.85%)	1.55	29.87	40.57	57.95	30.83	47.00 (+7.95, +20.36%)	37.73 (+4.57, +13.79%)	51.05 (+5.25, +11.46%)	34.55 (+1.95, +5.98%)	31.83 (+3.97, +14.23%)	38.00	37.95 (+7.30, +23.82%)	36.48 (+0.47, +1.31%)	24.83 (+2.57, +11.53%)	50.15 (+3.50, +7.50%)	24.40	46.47 (+1.60, +3.57%)	39.20 (+6.95, +21.55%)	24.80	32.93 (+0.56, +1.72%)	33.60 (+4.45, +15.27%)	26.32	32.50 (+1.63, +5.29%)	33.54 (+1.53, +4.79%)
argos	N/A	20.55 (-11.01, -34.89%)	20.42 (-9.66, -32.11%)	30.37	24.80 (-9.70, -28.12%)	N/A	23.17 (-2.33, -9.15%)	19.33 (-4.69, -19.51%)	27.17 (-2.60, -8.73%)	N/A	N/A	N/A	N/A	34.04 (-1.39, -3.93%)	43.80 (-6.05, -12.14%)	N/A	12.97 (-13.55, -51.08%)	N/A	29.00	N/A	15.70	25.76 (-7.45, -22.42%)	N/A	N/A	N/A	17.40 (-4.18, -19.35%)	26.25 (-0.65, -2.42%)	31.83 (-0.63, -1.95%)	22.14 (-7.30, -24.79%)	36.80 (-8.75, -19.21%)	24.90 (-2.40, -8.79%)	N/A	42.05 (-3.40, -7.48%)	N/A	N/A	15.07	N/A	27.89 (-4.03, -12.63%)	21.30 (+4.00, +23.12%)	N/A	N/A	31.33 (-1.03, -3.19%)	38.85 (+1.25, +3.32%)	30.40 (-11.80, -27.96%)	N/A	29.90 (+0.25, +0.84%)	N/A	16.18 (-9.24, -36.35%)	36.10 (-2.40, -6.23%)	12.93	26.27 (-9.30, -26.15%)	N/A	29.93 (-2.49, -7.67%)	27.90 (-0.80, -2.79%)	42.30 (+4.30, +11.32%)	15.73 (-11.50, -42.23%)	N/A	N/A	39.45 (-2.65, -6.29%)	N/A	N/A	20.41 (-6.19, -23.27%)	25.47	N/A	N/A	15.65 (-10.20, -39.46%)	N/A	N/A	N/A	44.35 (-2.05, -4.42%)	N/A	14.10	25.70	N/A	N/A	29.00 (-10.05, -25.74%)	30.97 (-2.18, -6.59%)	38.20 (-7.60, -16.59%)	31.07 (-1.53, -4.68%)	27.67 (-0.20, -0.72%)	N/A	N/A	33.44 (-2.57, -7.14%)	19.67 (-2.60, -11.68%)	34.45 (-12.20, -26.15%)	N/A	42.77 (-2.10, -4.68%)	20.35 (-11.90, -36.90%)	N/A	30.70 (-1.68, -5.18%)	N/A	12.43	N/A	26.56 (-5.46, -17.04%)
nllb	34.80	22.66 (-8.90, -28.20%)	22.90 (-7.18, -23.87%)	36.23	26.95 (-7.55, -21.88%)	30.90	17.57 (-7.93, -31.11%)	17.41 (-6.60, -27.48%)	28.30 (-1.47, -4.93%)	14.45	31.40	16.10	22.53 (-6.27, -21.76%)	34.95 (-0.48, -1.36%)	45.65 (-4.20, -8.43%)	31.57 (-8.47, -21.15%)	15.85 (-10.68, -40.25%)	25.45	27.04	31.70 (-1.30, -3.94%)	15.58	33.42 (+0.21, +0.63%)	26.95 (-15.75, -36.89%)	25.60	28.95	16.60 (-4.97, -23.06%)	23.30 (-3.60, -13.38%)	32.80 (+0.33, +1.03%)	13.96 (-15.49, -52.60%)	40.30 (-5.25, -11.53%)	24.35 (-2.95, -10.81%)	28.60	38.20 (-7.25, -15.95%)	10.35	15.40 (-13.70, -47.08%)	19.10	7.45	27.47 (-4.46, -13.97%)	18.80 (+1.50, +8.67%)	17.20	26.30 (-5.10, -16.24%)	11.40 (-20.97, -64.78%)	27.60 (-10.00, -26.60%)	35.45 (-6.75, -16.00%)	32.65	25.55 (-4.10, -13.83%)	16.30 (-6.53, -28.61%)	19.00 (-6.42, -25.26%)	36.40 (-2.10, -5.45%)	22.90	33.00 (-2.57, -7.22%)	33.25	32.04 (-0.38, -1.16%)	29.75 (+1.05, +3.66%)	37.40 (-0.60, -1.58%)	21.97 (-5.27, -19.34%)	37.15	17.63 (-5.77, -24.64%)	34.05 (-8.05, -19.12%)	14.30 (-12.77, -47.17%)	35.00	21.20 (-5.40, -20.30%)	27.03	15.10	24.50	21.30 (-4.55, -17.59%)	23.75 (-10.35, -30.35%)	32.23 (-5.73, -15.10%)	33.60 (-3.65, -9.80%)	31.45 (-14.95, -32.22%)	2.35	22.37	35.43	39.35	18.17	38.95 (-0.10, -0.26%)	25.39 (-7.77, -23.42%)	32.20 (-13.60, -29.69%)	30.15 (-2.45, -7.52%)	22.83 (-5.03, -18.06%)	30.50	17.40 (-13.25, -43.23%)	35.81 (-0.20, -0.56%)	18.33 (-3.93, -17.66%)	41.50 (-5.15, -11.04%)	25.60	40.00 (-4.87, -10.85%)	26.00 (-6.25, -19.38%)	21.80	31.91 (-0.47, -1.44%)	23.25 (-5.90, -20.24%)	16.80	16.97 (-13.90, -45.03%)	28.09 (-3.92, -12.24%)
opusmt	32.70	29.60 (-1.96, -6.22%)	28.18 (-1.90, -6.32%)	34.37	33.90 (-0.60, -1.74%)	11.10	23.53 (-1.97, -7.71%)	26.70 (+2.69, +11.18%)	28.50 (-1.27, -4.26%)	N/A	19.35	8.10	N/A	35.90 (+0.47, +1.32%)	N/A	N/A	26.15 (-0.38, -1.41%)	20.65	27.44	25.75 (-7.25, -21.97%)	N/A	29.65 (-3.55, -10.70%)	N/A	N/A	N/A	20.57 (-1.00, -4.63%)	24.70 (-2.20, -8.18%)	N/A	28.30 (-1.14, -3.88%)	43.00 (-2.55, -5.60%)	25.30 (-2.00, -7.33%)	N/A	43.75 (-1.70, -3.74%)	N/A	28.00 (-1.10, -3.78%)	20.10	N/A	25.82 (-6.10, -19.11%)	N/A	12.60	N/A	31.50 (-0.87, -2.68%)	N/A	29.70 (-12.50, -29.62%)	18.75	29.65 (0.00, 0.00%)	N/A	N/A	36.30 (-2.20, -5.71%)	20.13	27.77 (-7.80, -21.93%)	12.75	32.73 (+0.31, +0.96%)	N/A	29.75 (-8.25, -21.71%)	18.80 (-8.43, -30.97%)	26.35	25.37 (+1.97, +8.40%)	32.50 (-9.60, -22.80%)	N/A	21.40	20.84 (-5.76, -21.64%)	12.43	6.65	N/A	23.99 (-1.86, -7.19%)	N/A	31.37 (-6.60, -17.38%)	N/A	46.10 (-0.30, -0.65%)	N/A	N/A	17.40	43.75	17.33	34.25 (-4.80, -12.29%)	34.09 (+0.93, +2.82%)	45.10 (-0.70, -1.53%)	32.25 (-0.35, -1.07%)	26.80 (-1.07, -3.83%)	N/A	N/A	36.31 (+0.30, +0.83%)	N/A	37.85 (-8.80, -18.86%)	N/A	N/A	30.35 (-1.90, -5.89%)	8.50	32.58 (+0.20, +0.62%)	N/A	12.93	N/A	29.91 (-2.10, -6.56%)