Pith.

Computation and Language · arXiv:2409.03454 v2 · submitted 5 Sep 2024

How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizes

Inacio Vieira, Will Allred, Séamus Lankford, Sheila Castilho, Andy Way

The paper says

“Our findings reveal improvement in translation performance with larger datasets across all metrics.”

from the abstract · picked out as the main claim by a classifier

Pith says

Pith has not reviewed this paper.

Get a Pith Review, $5

Reviews

Reviews by people

No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Decoder-only LLMs have shown impressive performance in MT due to their ability to learn from extensive datasets and generate high-quality translations. However, LLMs often struggle with the nuances and style required for organisation-specific translation. In this study, we explore the effectiveness of fine-tuning Large Language Models (LLMs), particularly Llama 3 8B Instruct, leveraging translation memories (TMs), as a valuable resource to enhance accuracy and efficiency. We investigate the impact of fine-tuning the Llama 3 model using TMs from a specific organisation in the software sector. Our experiments cover five translation directions across languages of varying resource levels (English to Brazilian Portuguese, Czech, German, Finnish, and Korean). We analyse diverse sizes of training datasets (1k to 207k segments) to evaluate their influence on translation quality. We fine-tune separate models for each training set and evaluate their performance based on automatic metrics, BLEU, chrF++, TER, and COMET. Our findings reveal improvement in translation performance with larger datasets across all metrics. On average, BLEU and COMET scores increase by 13 and 25 points, respectively, on the largest training set against the baseline model. Notably, there is a performance deterioration in comparison with the baseline model when fine-tuning on only 1k and 2k examples; however, we observe a substantial improvement as the training dataset size increases. The study highlights the potential of integrating TMs with LLMs to create bespoke translation models tailored to the specific needs of businesses, thus enhancing translation quality and reducing turn-around times. This approach offers a valuable insight for organisations seeking to leverage TMs and LLMs for optimal translation outcomes, especially in narrower domains.

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23 · no language-model review

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.03 · states limitation 0.73 · human participants 0.03 · controlled comparison 0.96 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.94 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Decoder-only LLMs have shown impressive performance in MT due to their ability to learn from extensive datasets and generate high-quality translations. background 1.00 0.04 0.02 asserted 1.00
However, LLMs often struggle with the nuances and style required for organisation-specific translation. background 0.66 0.06 0.02 asserted 0.82
In this study, we explore the effectiveness of fine-tuning Large Language Models (LLMs), particularly Llama 3 8B Instruct, leveraging translation memories (TMs), as a valuable resource to enhance accuracy and efficiency. contribution 0.97 0.59 0.05 asserted 0.75
We investigate the impact of fine-tuning the Llama 3 model using TMs from a specific organisation in the software sector. method 0.55 0.88 0.03 asserted 0.85
Our experiments cover five translation directions across languages of varying resource levels (English to Brazilian Portuguese, Czech, German, Finnish, and Korean). method 0.96 0.92 0.02 asserted 0.96
We analyse diverse sizes of training datasets (1k to 207k segments) to evaluate their influence on translation quality. method 0.96 0.94 0.03 asserted 0.85
We fine-tune separate models for each training set and evaluate their performance based on automatic metrics, BLEU, chrF++, TER, and COMET. method 1.00 0.97 0.03 asserted 0.97
Our findings reveal improvement in translation performance with larger datasets across all metrics. result 1.00 0.96 0.59 asserted 1.00
On average, BLEU and COMET scores increase by 13 and 25 points, respectively, on the largest training set against the baseline model. result 1.00 0.96 0.19 asserted 1.00
Notably, there is a performance deterioration in comparison with the baseline model when fine-tuning on only 1k and 2k examples; however, we observe a substantial improvement as the training dataset size increases. result 1.00 0.96 0.26 asserted 1.00
The study highlights the potential of integrating TMs with LLMs to create bespoke translation models tailored to the specific needs of businesses, thus enhancing translation quality and reducing turn-around times. interpretation 1.00 0.35 0.46 hedged possibility 0.65
This approach offers a valuable insight for organisations seeking to leverage TMs and LLMs for optimal translation outcomes, especially in narrower domains. interpretation 0.99 0.20 0.20 asserted 0.62

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v2

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 8,910 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Cited by 3 Pith papers
How the literature uses it

6 arXiv papers cite this in 7 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 5 sentences without a confident label

Cites it as support 2 papers

What it cites · 8 sentences without a confident label

Uses the method or tool of 3 papers
Compares with 1 paper
Cites as background 1 paper

Challenges and discussion

Know something about this paper? Leave a comment.

No comments yet.

Sign in to comment