Computation and Language · arXiv:2409.03454 v2 · submitted 5 Sep 2024
How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizes
The paper says
“Our findings reveal improvement in translation performance with larger datasets across all metrics.”
from the abstract · picked out as the main claim by a classifier
Reviews
Reviews by people
No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Decoder-only LLMs have shown impressive performance in MT due to their ability to learn from extensive datasets and generate high-quality translations. However, LLMs often struggle with the nuances and style required for organisation-specific translation. In this study, we explore the effectiveness of fine-tuning Large Language Models (LLMs), particularly Llama 3 8B Instruct, leveraging translation memories (TMs), as a valuable resource to enhance accuracy and efficiency. We investigate the impact of fine-tuning the Llama 3 model using TMs from a specific organisation in the software sector. Our experiments cover five translation directions across languages of varying resource levels (English to Brazilian Portuguese, Czech, German, Finnish, and Korean). We analyse diverse sizes of training datasets (1k to 207k segments) to evaluate their influence on translation quality. We fine-tune separate models for each training set and evaluate their performance based on automatic metrics, BLEU, chrF++, TER, and COMET. Our findings reveal improvement in translation performance with larger datasets across all metrics. On average, BLEU and COMET scores increase by 13 and 25 points, respectively, on the largest training set against the baseline model. Notably, there is a performance deterioration in comparison with the baseline model when fine-tuning on only 1k and 2k examples; however, we observe a substantial improvement as the training dataset size increases. The study highlights the potential of integrating TMs with LLMs to create bespoke translation models tailored to the specific needs of businesses, thus enhancing translation quality and reducing turn-around times. This approach offers a valuable insight for organisations seeking to leverage TMs and LLMs for optimal translation outcomes, especially in narrower domains.
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23 · no language-model review
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.03 · states limitation 0.73 · human participants 0.03 · controlled comparison 0.96 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.94 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Decoder-only LLMs have shown impressive performance in MT due to their ability to learn from extensive datasets and generate high-quality translations. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| However, LLMs often struggle with the nuances and style required for organisation-specific translation. | background 0.66 | 0.06 | 0.02 | asserted 0.82 |
| In this study, we explore the effectiveness of fine-tuning Large Language Models (LLMs), particularly Llama 3 8B Instruct, leveraging translation memories (TMs), as a valuable resource to enhance accuracy and efficiency. | contribution 0.97 | 0.59 | 0.05 | asserted 0.75 |
| We investigate the impact of fine-tuning the Llama 3 model using TMs from a specific organisation in the software sector. | method 0.55 | 0.88 | 0.03 | asserted 0.85 |
| Our experiments cover five translation directions across languages of varying resource levels (English to Brazilian Portuguese, Czech, German, Finnish, and Korean). | method 0.96 | 0.92 | 0.02 | asserted 0.96 |
| We analyse diverse sizes of training datasets (1k to 207k segments) to evaluate their influence on translation quality. | method 0.96 | 0.94 | 0.03 | asserted 0.85 |
| We fine-tune separate models for each training set and evaluate their performance based on automatic metrics, BLEU, chrF++, TER, and COMET. | method 1.00 | 0.97 | 0.03 | asserted 0.97 |
| Our findings reveal improvement in translation performance with larger datasets across all metrics. | result 1.00 | 0.96 | 0.59 | asserted 1.00 |
| On average, BLEU and COMET scores increase by 13 and 25 points, respectively, on the largest training set against the baseline model. | result 1.00 | 0.96 | 0.19 | asserted 1.00 |
| Notably, there is a performance deterioration in comparison with the baseline model when fine-tuning on only 1k and 2k examples; however, we observe a substantial improvement as the training dataset size increases. | result 1.00 | 0.96 | 0.26 | asserted 1.00 |
| The study highlights the potential of integrating TMs with LLMs to create bespoke translation models tailored to the specific needs of businesses, thus enhancing translation quality and reducing turn-around times. | interpretation 1.00 | 0.35 | 0.46 | hedged possibility 0.65 |
| This approach offers a valuable insight for organisations seeking to leverage TMs and LLMs for optimal translation outcomes, especially in narrower domains. | interpretation 0.99 | 0.20 | 0.20 | asserted 0.62 |
What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v2
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 8,910 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Cited by 3 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- INVAR-Agri: Detecting Discrepancies in Multilingual Operator Manuals of Agricultural Implements with a Fine-Tuned LLM cs.CL · Oct 2026
- Beyond Benchmarks: Evaluating Large Language Models and Agentic Patterns for Static Code Analysis Warning Classification cs.SE · Nov 2026
- LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights cs.CR · Feb 2025
How the literature uses it
6 arXiv papers cite this in 7 sentences
Papers citing it · 5 sentences without a confident label
Cites it as support 2 papers
-
We intentionally opt for a compact language model rather than a larger LLM, as prior work has shown that for high-resource language pairs, domain adaptation yields diminishing returns beyond moderate model sizes when high-quality in-domain data is available [ref] .
Translating Under Pressure: Domain-Aware LLMs for Crisis Communication · Training · 1.00
-
Our IPD dataset contains only 551 parameters, which is insufficient for FFT, as prior research has shown that fewer than 1,000 examples do not yield significant performance improvements with FFT [ref] .
LlamaRestTest: Effective REST API Testing with Small Language Models · Parameter-Efficient Fine-Tuning · 0.98
What it cites · 8 sentences without a confident label
Uses the method or tool of 3 papers
-
We use QLoRA [ref] for efficient fine-tuning with 4-bit quantisation using Hugging Face Transformers.
LoRA: Low-Rank Adaptation of Large Language Models · Model · 1.00
-
We use QLoRA [ref] for efficient fine-tuning with 4-bit quantisation using Hugging Face Transformers.
QLoRA: Efficient Finetuning of Quantized LLMs · Model · 1.00
-
At inference time, we use many of the recommended parameters from previous work [ref] and model documentation to produce translation outputs from the baseline model and the fine-tuned versions (cf. Appendix [ref]).
Fine-tuning Large Language Models for Adaptive Machine Translation · Prompting · 0.93
Compares with 1 paper
-
The test and training sets used come from a much narrower corpus of data than in similar experiments that deal with wider domains, i.e. medicine [ref] .
Fine-tuning Large Language Models for Adaptive Machine Translation · Conclusions · 0.84
Cites as background 1 paper
-
In recent years, decoder-only large language models (LLMs) have revolutionised the machine translation (MT) field due to their ability to learn from vast amounts of data and generate high-quality translations [ref] .
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis · Introduction · 0.96
Challenges and discussion
Know something about this paper? Leave a comment.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment