You can use semantic similarity (cosine sim) using SentenceTransformers to measure how close the translations are. I’ve found there are still some issues and it’s not always picking up on issues (eg noisy random punctuation missing in src or tgt dont reflect much in the similarity)
2 Likes