LibreTranslate Community

Finetuning bigger models with LoRa (Low-Rank Adaptation) in OpenNMT-py

Argos Translate

opennmt, research

argosopentech April 13, 2023, 11:33pm 1

OpenNMT – 13 Apr 23

Finetuning bigger models with LoRa (Low-Rank Adaptation) in OpenNMT-py

Hello Users, With the new version 3.1.1 it is possible to finetune a bigger model. As you know the issue is as follow: When training / finetuning a 3B parameters in fp16 mode, it will require: 3B x 2 bytes = 6 GB just to load the model about the...

Reading time: 1 mins 🕑 Likes: 2 ❤

Home
Categories
FAQ/Guidelines
Terms of Service
Privacy Policy

Powered by Discourse, best viewed with JavaScript enabled