A Generative Al Engineer has built an LLM-based system that will automatically translate user text...

The task is to benchmark LLMs for text translation using an evaluation set with known high-quality examples, requiring a performant metric. Let’s evaluate the options.

Option A: ROUGE metric

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures overlap between generated and reference texts, primarily for summarization. It’s less suited for translation, where precision and word order matter more.

Databricks Reference : " ROUGE is commonly used for summarization, not translation evaluation " ( " Generative AI Cookbook, " 2023).

Option B: BLEU metric

BLEU (Bilingual Evaluation Understudy) evaluates translation quality by comparing n-gram overlap with reference translations, accounting for precision and brevity. It’s widely used, performant, and appropriate for this task.

Databricks Reference : " BLEU is a standard metric for evaluating machine translation, balancing accuracy and efficiency " ( " Building LLM Applications with Databricks " ).

Option C: NDCG metric

NDCG (Normalized Discounted Cumulative Gain) assesses ranking quality, not text generation. It’s irrelevant for translation evaluation.

Databricks Reference : " NDCG is suited for ranking tasks, not generative output scoring " ( " Databricks Generative AI Engineer Guide " ).

Option D: RECALL metric

Recall measures retrieved relevant items but doesn’t evaluate translation quality (e.g., fluency, correctness). It’s incomplete for this use case.

Databricks Reference : No specific extract, but recall alone lacks the granularity of BLEU for text generation tasks.

Conclusion : Option B (BLEU) is the best metric for translation evaluation, offering a performant and standard approach, as endorsed by Databricks’ guidance on generative tasks.