LLM-Based Evaluation of Low-Resource Machine Translation: A Reference-less Dialect Guided Approach with a Refined Sylheti-English Benchmark

Md. Atiqur Rahman; Mushfiqul Haque Omi; Sabrina Islam

REVIEW 6 major objections 6 minor 20 references

LLM-Based Evaluation of Low-Resource Machine Translation: A Reference-less Dialect Guided Approach with a Refined Sylheti-English Benchmark

T0 review · 6 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A dialect-glossed prompt, an extended tokenizer, and a regression head let reference-less LLM evaluation of Sylheti-English track human quality scores, improving Spearman correlation by up to +0.1083 over the annotation-guided baseline.

desk verdict Useful new Sylheti-English benchmark, but the central comparison is unverifiable because the paper never discloses whether the reported correlations are out-of-sample. read the letter →

arxiv 2505.12273 v1 pith:AEDPBQWT submitted 2025-05-18 cs.CL

Md. Atiqur Rahman , Sabrina Islam , Mushfiqul Haque Omi This is my paper · ORCID

classification cs.CL

keywords MachineTranslationEvaluationLargeLanguageModels(LLMs)Low-ResourceLanguagesPromptEngineeringDialect-GuidedPromptingSylhetiDirectAssessmentReference-less

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Evaluating machine translation normally needs human reference translations, which are costly and often nonexistent for dialect-rich low-resource languages like Sylheti. This paper tries to establish that general-purpose LLMs, guided properly, can score such translations without any reference at all: give the model a glossary of dialect terms, extend its tokenizer with dialect vocabulary, and map its frozen embeddings to human Direct Assessment scores through a small regression head. On an extended Sylheti-English benchmark of 1,500 sentence pairs with native-speaker scores, the pipeline raises rank correlation with human judgments by up to 0.1083 over the prior annotation-guided prompting baseline across three of four open LLMs. If the claim holds, MT developers for underrepresented dialects gain a cheap, reference-free way to know which translations are good enough to use, removing a bottleneck that keeps such languages out of translation systems.

What carries the argument

The carrying mechanism is the dialect-guided (DG) prompt, a three-part instruction that hands the LLM a glossary of Sylheti words with their English glosses alongside the original Sylheti-influenced Bengali source sentence and the machine-translated English sentence to be scored. The glossary is built by cross-referencing each source word against a curated Sylheti-English dictionary, so the model never has to infer dialect vocabulary from a tokenizer that barely knows it. Two supporting mechanisms complete the pipeline: a byte-level BPE tokenizer the authors retrain on Sylheti sentences and merge with the original LLM vocabulary, and a lightweight regression head, trained with mean-squared error on 1,200 samples of frozen LLM embeddings, that converts the model's representations into a scalar DA score. The ablation isolates each piece — tokenizer, head, and prompt — and shows the prompt stage delivering the final and largest jump in correlation.

What would settle it

Use the released data and code to recompute Pearson and Spearman correlations on the subset of the 1,500-pair pool that the regression head never saw during training, and compare the DG gain over AG there; if the gain shrinks or flips sign on that held-out subset, the central improvement claim collapses. A second, cheaper check: compute inter-annotator agreement between the two native speakers — if the pipeline's correlation with averaged scores approaches or exceeds that agreement, the benchmark's ground truth is the limiting factor, not the pipeline.

Watch

Extended reading notes

Core claim

The paper's central claim is that dialect awareness, exposed explicitly to the model, is what unlocks reference-less LLM evaluation for dialect-rich languages. Its demonstration: for Llama-2-7B, Llama-2-13B, and OpenChat, the dialect-guided (DG) prompt correlates more strongly with human DA scores than the annotation-guided (AG) prompt of prior work, in both Pearson and Spearman terms — the largest Spearman gain is +0.1083 (0.1496 to 0.2579) on Llama-2-13B, and OpenChat reaches the best absolute correlations of 0.2826 Pearson and 0.2852 Spearman. The exception is Gemma, which scores marginally higher under AG; the paper attributes this to Gemma lacking chat tuning. The ablation on OpenChat attributes the total effect to the stacking of components: the untouched model sits at -0.1446 Pearson correlation, the dialect-aware tokenizer alone brings it to 0.0336, the regression head to 0.2312, and the DG prompt to 0.2734. The paper also contributes the benchmark itself: 520 new Sylheti source sentences, NLLB-200 translations for all 1,500 pairs, and Direct Assessment scores averaged over two native speakers.

Load-bearing premise

The load-bearing premise is that the reported correlations are computed on data the regression head did not train on: the paper never states the train/test split, reports no inter-annotator agreement for the two native speakers whose averaged DA scores serve as ground truth, and evaluates only NLLB-200 translations, so if the evaluation subset overlaps the 1,200-sample training set, the +0.1083 gain over the annotation-guided baseline reflects the training distribution rather than reference-less evaluation skill.

Editorial extensions

If this is right

Reference-free evaluation for dialect-rich languages no longer requires full fine-tuning: only a small regression head is trained, on 1,200 samples, while the LLM stays frozen.
Dialect cues in the prompt carry more signal than the official annotation guidelines: DG beats DAG (guidelines plus dialect cues) on every model tested, so prompt design for low-resource evaluation should lead with dialect context.
MT developers for Sylheti can run the released pipeline and obtain human-aligned quality scores for new translations, with no human references needed for scoring.
The tokenizer extension alone moves OpenChat from a negative to a positive correlation with human scores, indicating that lexical coverage is a first-order barrier for low-resource dialects.
Larger chat-tuned models gain most from dialect guidance, so scaling model size and dialect-aware prompting are complementary rather than competing routes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same glossary mechanism should transfer to sibling dialect continua — Chittagong and Barisal Bengali, or Arabic and Chinese dialect groups — as long as a dialect-to-English glossary exists; a direct test would build a DG prompt from such a dictionary and measure the Spearman gain over the annotation-guided baseline, which the released code makes straightforward.
The ablation result suggests that what cripples LLM evaluation on low-resource dialects may be surface-form recognition more than reasoning ability: a vocabulary-only intervention flips OpenChat's correlation from -0.1446 to +0.0336, so cheaper fixes than larger models or fine-tuning deserve attention.
Since every DA label in the benchmark anchors to NLLB-200's error profile, the pipeline's ranking fidelity on other MT systems' outputs is an open question; scoring translations from several engines against the same 1,500 sources would test whether the +0.1083 gain is specific to that distribution.
Editorial note on the manuscript itself: Section 4.1 cites the annotation-guided baseline with a bare '[cite]' marker rather than a numbered reference, so the comparison's provenance must be read back from the background section; the numbers in this summary take that baseline to be the AG prompting of the prior work the paper extends.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read it. The genuinely new thing here is the Sylheti-English benchmark: 520 new sentence pairs, NLLB-200 outputs, and native-speaker DA scores layered on the ONUBAD material. That is real, useful data for a dialect that basically has no MT evaluation resources. The dialect-glossary prompt variation and the tokenizer extension are sensible engineering moves, and the authors ship code and data, which makes the work easy to build on. Credit where it is due: the resource itself is a contribution.

The soft spots are in the evaluation. Section 3.4 says the regression head is trained on 1,200 samples; Section 4.1 reports Pearson and Spearman correlations without ever saying whether those correlations are computed on a held-out subset, on the training pool, or on all 1,500 sentences. If the numbers are in-sample, the +0.1083 Spearman gain and the ablation increments in Table 3 are artifacts, not evidence. This is not a minor omission; it is load-bearing for the central claim. The stress-test note is right to flag it. Also, the abstract says "consistently outperforms," but Gemma does better with AG than with DG in both tables, so the headline claim is overstated as written.

Other weaknesses are more fixable: no standard QE baselines (an XLM-R or COMET-style reference-free system), no inter-annotator agreement reported, only NLLB-200 as the MT source, and no significance tests for the ablation. The absolute correlations are low (under 0.29 for the best DG setting), so the practical value is modest even if the split is clean.

The authors need to state the split, report held-out numbers, add at least one standard baseline, and temper the language. If the split is proper, the result is a decent incremental contribution for a low-resource dialect. If it is not, the main result collapses. The paper deserved a serious referee, but only with a demand for the train/test disclosure and a rerun on the held-out 300. This is the kind of work a peer reviewer can meaningfully improve.

Referee Report

6 major / 6 minor

Summary. This paper proposes a reference-less machine translation (MT) evaluation pipeline for Sylheti–English, a low-resource, dialect-rich language pair. The authors extend the ONUBAD corpus to 1,500 Sylheti–English sentence pairs, obtain NLLB-200 translations, and collect Direct Assessment (DA) scores from two native speakers. The proposed pipeline has three components: a Dialect Guided (DG) prompting strategy that injects a Sylheti–English glossary into the prompt, a dialect-aware tokenizer extension that adds unique Sylheti tokens, and a regression head trained on frozen LLM embeddings to predict scalar DA scores. The system is compared with Annotation Guided (AG) prompting and a combined DAG variant across Llama-2-7B, Llama-2-13B, OpenChat, and Gemma, using Pearson and Spearman correlations as evaluation metrics. The abstract claims consistent outperformance with a maximum Spearman gain of +0.1083. The paper does not disclose the train/test split used for the regression head or the provenance of the glossary and tokenizer vocabulary relative to the evaluation set.

Significance. If the reported numbers are out-of-sample, the paper offers a genuinely useful contribution to low-resource MT evaluation: a native-annotated benchmark for a severely underrepresented dialect, a concrete method for incorporating dialect-specific lexical knowledge into LLM prompts, and evidence that a lightweight regression head on frozen LLM features improves DA alignment. The public release of the dataset and code is a strength. The significance is nevertheless conditional on resolving a central methodological ambiguity: every comparative claim depends on whether the regression head, tokenizer vocabulary, and glossary were derived from the evaluation sentences. As written, the evidence for the headline claim is incomplete, and the Gemma results contradict the 'consistently outperforms' wording.

major comments (6)

[3.4 and 4.1] Section 3.4 states that the regression head was fine-tuned on 1,200 samples, but Section 4.1 reports correlations without disclosing whether Tables 1 and 2 are computed on a held-out 300-sentence subset, on the training pool, or on all 1,500 sentences. If the reported correlations include the training samples, the +0.1083 Spearman gain and the Table 3 increments are in-sample artifacts, not evidence for a working evaluation pipeline. Please specify the split exactly and, ideally, report cross-validated or separate test-set correlations.
[3.2 and 3.3] The DG prompt glossary and the dialect-aware tokenizer vocabulary are described as built from 'Sylheti sentences,' but the paper never states that these resources were constructed without access to the evaluation sentences. If the glossary or the tokenizer vocabulary was derived from the test sentences, the improvements attributed to the DG prompt and the tokenizer would be partially in-sample. Please clarify the data provenance of both resources relative to the evaluation set.
[Abstract, Tables 1 and 2] The abstract's claim that the proposed pipeline 'consistently outperforms' existing methods is contradicted by the Gemma rows: in Table 1, AG achieves 0.3247 vs DG 0.3079, and in Table 2, AG achieves 0.3566 vs DG 0.3488. The Section 5 discussion acknowledges the Gemma exception, so the abstract and any summary of the results should be revised to reflect gains on three of four models rather than consistent outperformance.
[4.2] The ablation study in Table 3 does not report variance, number of runs, or any significance test, so the incremental benefit of each component cannot be distinguished from noise. In addition, the '× × ×' baseline reports Pearson = -0.1446 for OpenChat 'without any modifications,' which is inconsistent with the AG OpenChat result of 0.2299 in Table 1 unless AG is treated as a separate modification; this needs clarification. The relative change from -0.1446 to 0.0336 described as 'approximately 123%' is not a meaningful percentage improvement for a correlation coefficient.
[4.1] The experimental comparison contains only the AG baseline; no standard reference-free QE systems such as COMETKiwi or XCOMET are used. The abstract's phrase 'outperforms existing methods' therefore overstates the support provided by the experiments, which at most show a gain over AG prompting under the reported settings.
[3.1] The benchmark's ground truth consists of DA scores from only two native Sylheti speakers, and the paper reports no inter-annotator agreement (e.g., Pearson correlation between annotators or Krippendorff's alpha). Without a reliability estimate for the averaged scores, the quality of the benchmark itself is not established, which weakens the validity of any downstream correlation claim.

minor comments (6)

[4.1] The text contains the placeholder '[cite]' next to 'SOTA AG'; this must be replaced with a proper citation.
[References] References 4 and 14 are the same item; deduplicate them.
[3.1] Section 3.1 says ONUBAD contains '1,540 words, 130 clauses, and 980 sentences per dialect,' but the paper does not explain how the 980 extracted sentence pairs relate to the 1,500 final pairs beyond adding 520 new ones; clarify the extraction and filtering process.
[3.5 and 5] The paper should specify which Gemma checkpoint (e.g., Gemma-7B) was used, since Section 5 attributes behavior to lack of instruction tuning.
[Figure 2] Figure 2 is described in the text but not present in the provided version; ensure the final PDF includes the DG prompt figure.
[Throughout] Use a consistent spelling for 'Llama' throughout (e.g., 'Llama-2' rather than 'LLaMA-2').

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the benchmark and baselines are external, though train/test separation is undisclosed and reporting has internal inconsistencies.

full rationale

The paper's claimed contribution is an empirical comparison: it builds a Sylheti-English benchmark from ONUBAD plus new human DA annotations, augments the tokenizer vocabulary, trains a light regression head on DA scores, and compares DG/DAG prompting against an external AG baseline. None of these components is defined in terms of the paper's own success metric; the human DA scores are external ground truth, and the AG baseline comes from Sindhujan et al. (ref [8]), not from the authors' own prior results. I found no step where a 'prediction' equals its input by construction, no load-bearing self-citation, and no uniqueness or ansatz imported from the authors' own prior work. The legitimate concerns are reporting and statistical-validity issues rather than circularity: Section 3.4 says the regression head was fine-tuned on 1200 samples, but Section 4.1 reports Pearson/Spearman correlations without stating whether Tables 1-3 are computed on a held-out 300, on the training pool, or on all 1500; if the evaluation set overlaps the training set, the reported gains are partly in-sample fit. This is a real leakage risk, but the text does not actually state that the evaluation uses the training set, so it cannot be exhibited as a circular reduction. Similarly, the dialect-aware tokenizer and glossary are built from Sylheti sentences whose relationship to the evaluation sentences is not disclosed; again this is a leakage risk, not a definitional equivalence. The paper also contains a missing citation placeholder in Section 4.1 ('AG (Annotation Guided) [cite]'), and the abstract's 'consistently outperforms' is contradicted by the paper's own Tables 1 and 2, where Gemma scores higher under AG than DG for both Pearson (0.3247 vs 0.3079) and Spearman (0.3566 vs 0.3488). These flaws should be weighed in a correctness review, but they do not make the central derivation circular.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new entities or forces. The central empirical claim rests on a small, hand-curated benchmark, a learned regression head, and a dialect glossary, all of which are dataset or model choices rather than physical or mathematical postulates.

free parameters (4)

Regression head parameters = learned on 1200 DA-annotated samples
The light-weight head is fine-tuned to map LLM hidden states to human DA scores; the claim depends on this fit. No held-out split is disclosed, so reported correlations may reflect in-sample fit.
Dialect glossary entries in DG prompt = hand-curated, not enumerated
The DG prompt inserts a Sylheti-English dictionary for matched words; which words and glosses are included is a manual choice affecting results.
Augmented tokenizer vocabulary = unique Sylheti tokens added, count unspecified
Tokens are added based on a Sylheti corpus; the set and size of additions is a design choice.
Hyperparameters = lr 2e-5, max length 512, 4-bit quantization, fp16
Chosen by hand for efficiency; small dataset may be sensitive to these choices.

assumptions (6)

standard math Pearson and Spearman correlations measure agreement with human DA scores.
The evaluation uses these standard correlation coefficients as the metric.
domain assumption Averaged DA scores from two native Sylheti speakers form reliable gold labels.
No inter-annotator agreement is reported; Section 3.1.
domain assumption Sylheti is treated as a Bengali dialect, so a Bengali source with Sylheti features is the appropriate input.
The dataset adapts ONUBAD's Sylheti-English pair, treating Sylheti as a Bengali dialect; Section 3.1.
domain assumption NLLB-200 translations are representative of MT output quality for the evaluation task.
All machine translations in the benchmark come from NLLB-200; Section 3.1.
domain assumption Frozen LLM embeddings contain sufficient signal for DA regression.
The regression head relies on LLM contextual outputs; Section 3.4.
domain assumption The curated Sylheti glossary covers the dialectal terms needed for successful prompting.
The DG prompt's effectiveness depends on the coverage of this hand-built dictionary; Section 3.2.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLM-Based Evaluation of Low-Resource Machine Translation: A Reference-less Dialect Guided Approach with a Refined Sylheti-English Benchmark." pith.science (2026). https://pith.science/paper/AEDPBQWT

@misc{pith2026250512273,
  author       = {Pith},
  title        = {Pith review of: LLM-Based Evaluation of Low-Resource Machine Translation: A Reference-less Dialect Guided Approach with a Refined Sylheti-English Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AEDPBQWT}},
  note         = {Machine review of arXiv:2505.12273}
}

read the original abstract

Evaluating machine translation (MT) for low-resource languages poses a persistent challenge, primarily due to the limited availability of high quality reference translations. This issue is further exacerbated in languages with multiple dialects, where linguistic diversity and data scarcity hinder robust evaluation. Large Language Models (LLMs) present a promising solution through reference-free evaluation techniques; however, their effectiveness diminishes in the absence of dialect-specific context and tailored guidance. In this work, we propose a comprehensive framework that enhances LLM-based MT evaluation using a dialect guided approach. We extend the ONUBAD dataset by incorporating Sylheti-English sentence pairs, corresponding machine translations, and Direct Assessment (DA) scores annotated by native speakers. To address the vocabulary gap, we augment the tokenizer vocabulary with dialect-specific terms. We further introduce a regression head to enable scalar score prediction and design a dialect-guided (DG) prompting strategy. Our evaluation across multiple LLMs shows that the proposed pipeline consistently outperforms existing methods, achieving the highest gain of +0.1083 in Spearman correlation, along with improvements across other evaluation settings. The dataset and the code are available at https://github.com/180041123-Atiq/MTEonLowResourceLanguage.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 11 canonical work pages

[1]

Automatic Machine Trans- lation Evaluation using a Source and Reference Sentence with a Cross -lingual Language Model

Takahashi, Kosuke, Katsuhito Sudoh, and Satoshi Nakamura. "Automatic Machine Trans- lation Evaluation using a Source and Reference Sentence with a Cross -lingual Language Model." Journal of Natural Language Processing 29.1 (2022): 3-22

work page 2022
[2]

Survey of low -resource machine translation

Haddow, Barry, et al. "Survey of low -resource machine translation." Computational Lin- guistics 48.3 (2022): 673-732

work page 2022
[3]

Large language models are state -of-the-art evalu- ators of translation quality

Kocmi, Tom, and Christian Federmann. "Large language models are state -of-the-art evalu- ators of translation quality." arXiv preprint arXiv:2302.14520 (2023)

arXiv 2023
[4]

Machine translation human evaluation: an investigation of eval- uation based on post -editing and its relation with direct assessment

Bentivogli, Luisa, et al. "Machine translation human evaluation: an investigation of eval- uation based on post -editing and its relation with direct assessment." Proceedings of the 15th International Workshop on Spoken Language Translation (IWSLT 2018). 2018. 9

work page 2018
[5]

Findings of the WMT 2023 Shared Task on Quality Estimation

Blain, Frederic, et al. "Findings of the WMT 2023 Shared Task on Quality Estimation." Eight conference on machine translation. Association for Computational Linguistics, 2023

work page 2023
[6]

Atlas of the World's Languages in Danger

Moseley, Christopher, ed. Atlas of the World's Languages in Danger. Unesco, 2010

work page 2010
[7]

Unsupervised cross-lingual representation learning at scale

Conneau, Alexis, et al. "Unsupervised cross-lingual representation learning at scale." arXiv preprint arXiv:1911.02116 (2019)

arXiv 2019
[8]

When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages

Sindhujan, Archchana, et al. "When LLMs Struggle: Reference -less Translation Evalua- tion for Low-resource Languages." arXiv preprint arXiv:2501.04473 (2025)

work page Pith review arXiv 2025

Show all 20 references

[9]

LEGAL -BERT: The muppets straight out of law school

Chalkidis, Ilias, et al. "LEGAL -BERT: The muppets straight out of law school." arXiv preprint arXiv:2010.02559 (2020)

2020 arXiv
[10]

Getting the most out of your to- kenizer for pre-training and domain adaptation

Dagan, Gautier, Gabriel Synnaeve, and Baptiste Roziere. "Getting the most out of your to- kenizer for pre-training and domain adaptation." arXiv preprint arXiv:2402.01035 (2024)

2024 arXiv
[11]

Understanding LLM Embeddings for Regression

Tang, Eric, Bangding Yang, and Xingyou Song. "Understanding LLM Embeddings for Regression." arXiv preprint arXiv:2411.14708 (2024)

2024 arXiv
[12]

ONUBAD: A comprehensive dataset for automated conversion of Bangla regional dialects into standard Bengali dialect

Sultana, Nusrat, et al. "ONUBAD: A comprehensive dataset for automated conversion of Bangla regional dialects into standard Bengali dialect." Data in Brief 58 (2025): 111276

2025
[13]

No language left behind: Scaling human -centered machine translation

Costa-Jussà, Marta R., et al. "No language left behind: Scaling human -centered machine translation." arXiv preprint arXiv:2207.04672 (2022)

2022 arXiv
[14]

Machine translation human evaluation: an investigation of eval- uation based on post -editing and its relation with direct assessment

Bentivogli, Luisa, et al. "Machine translation human evaluation: an investigation of eval- uation based on post -editing and its relation with direct assessment." Proceedings of the 15th International Workshop on Spoken Language Translation (IWSLT 2018). 2018

2018
[15]

Neural machine translation with byte - level subwords

Wang, Changhan, Kyunghyun Cho, and Jiatao Gu. "Neural machine translation with byte - level subwords." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 05. 2020

2020
[16]

Llama 2: Open foundation and fine -tuned chat models

Touvron, Hugo, et al. "Llama 2: Open foundation and fine -tuned chat models." arXiv pre- print arXiv:2307.09288 (2023)

2023 arXiv
[17]

Openchat: Advancing open -source language models with mixed - quality data

Wang, Guan, et al. "Openchat: Advancing open -source language models with mixed - quality data." arXiv preprint arXiv:2309.11235 (2023)

2023 arXiv
[18]

Gemma: Open models based on gemini research and technology

Team, Gemma, et al. "Gemma: Open models based on gemini research and technology." arXiv preprint arXiv:2403.08295 (2024)

2024 arXiv
[19]

Pearson correlation coefficient

Cohen, Israel, et al. "Pearson correlation coefficient." Noise reduction in speech processing (2009): 1-4

2009
[20]

Spearman rank correlation

Zar, Jerrold H. "Spearman rank correlation." Encyclopedia of biostatistics 7 (2005)

2005

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Automatic Machine Trans- lation Evaluation using a Source and Reference Sentence with a Cross -lingual Language Model

Takahashi, Kosuke, Katsuhito Sudoh, and Satoshi Nakamura. "Automatic Machine Trans- lation Evaluation using a Source and Reference Sentence with a Cross -lingual Language Model." Journal of Natural Language Processing 29.1 (2022): 3-22

work page 2022

[2] [2]

Survey of low -resource machine translation

Haddow, Barry, et al. "Survey of low -resource machine translation." Computational Lin- guistics 48.3 (2022): 673-732

work page 2022

[3] [3]

Large language models are state -of-the-art evalu- ators of translation quality

Kocmi, Tom, and Christian Federmann. "Large language models are state -of-the-art evalu- ators of translation quality." arXiv preprint arXiv:2302.14520 (2023)

arXiv 2023

[4] [4]

Machine translation human evaluation: an investigation of eval- uation based on post -editing and its relation with direct assessment

Bentivogli, Luisa, et al. "Machine translation human evaluation: an investigation of eval- uation based on post -editing and its relation with direct assessment." Proceedings of the 15th International Workshop on Spoken Language Translation (IWSLT 2018). 2018. 9

work page 2018

[5] [5]

Findings of the WMT 2023 Shared Task on Quality Estimation

Blain, Frederic, et al. "Findings of the WMT 2023 Shared Task on Quality Estimation." Eight conference on machine translation. Association for Computational Linguistics, 2023

work page 2023

[6] [6]

Atlas of the World's Languages in Danger

Moseley, Christopher, ed. Atlas of the World's Languages in Danger. Unesco, 2010

work page 2010

[7] [7]

Unsupervised cross-lingual representation learning at scale

Conneau, Alexis, et al. "Unsupervised cross-lingual representation learning at scale." arXiv preprint arXiv:1911.02116 (2019)

arXiv 2019

[8] [8]

When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages

Sindhujan, Archchana, et al. "When LLMs Struggle: Reference -less Translation Evalua- tion for Low-resource Languages." arXiv preprint arXiv:2501.04473 (2025)

work page Pith review arXiv 2025