Interesting model 3.6x smaller and 4x faster based on m2m 1.2B

Argos Translate
6

I have tried:

# wget https://huggingface.co/alirezamsh/small100/raw/main/tokenization_small100.py
# ct2-transformers-converter --model alirezamsh/small100 --output_dir model
from tokenization_small100 import SMALL100Tokenizer
import ctranslate2


translator = ctranslate2.Translator("model")
tokenizer = SMALL100Tokenizer.from_pretrained("alirezamsh/small100")
tokenizer.src_lang = "en"

source = tokenizer.convert_ids_to_tokens(tokenizer.encode("Hello world, it's a small test"))
target_prefix = [tokenizer.lang_code_to_token["fr"]]
results = translator.translate_batch([source], target_prefix=[target_prefix])
target = results[0].hypotheses[0][1:]

print(tokenizer.decode(tokenizer.convert_tokens_to_ids(target)))

but the model return “ku на, it’s a small test” xD

1 Like