Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages

T0 review · 2 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that for reference-less, segment-level translation-quality estimation across eight low-resource language pairs, prompt-based LLM approaches are consistently outperformed by fine-tuned encoder-based QE models.

desk verdict The paper's real contribution is a solid negative result—fine-tuned encoders beat open LLMs for low-resource QE—while the claimed AG-prompt win is an OpenChat effect that does not survive Table 1. read the letter →

arxiv 2501.04473 v1 pith:Z2WFYDRM submitted 2025-01-08 cs.CL

Archchana Sindhujan , Diptesh Kanojia , Constantin Orasan , Shenbin Qian This is my paper · ORCID
classification cs.CL
keywords qualityestimationlow-resourcemachinetranslationreference-lessevaluationlargelanguagemodelspromptengineeringannotationguidelinesin-contextlearningtokenization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to test whether open-weight large language models can do reference-less quality estimation—scoring a machine translation's quality on a 0–100 scale with no human reference—for low-resource language pairs, and to improve their performance with a new prompt built from human annotation guidelines. Across eight WMT pairs (English to Gujarati, Hindi, Marathi, Tamil, Telugu; Estonian, Nepali, Sinhala to English), it compares zero-shot, in-context learning, and LoRA instruction fine-tuning of Gemma-7B, Llama-2-7B/13B, and OpenChat-3.5 against fine-tuned encoder-based QE systems. Its central result is that the prompt-based LLM approaches are outperformed by the encoder-based fine-tuned models in almost all settings, with the gap closing but not closing fully after instruction fine-tuning. The paper's own contribution is the AG prompt, which improves zero-shot correlation for most pairs, plus a tokenization analysis tracing part of the failure to LLM tokenizers inflating token counts for morphologically rich non-Latin scripts. If true, it means the current wave of small open LLMs is not yet a drop-in replacement for encoder QE models in low-resource settings, and that cross-lingual quality scoring deserves a place in LLM evaluation suites.

What carries the argument

The central object is the AG prompt (annotation-guidelines prompt), a prompt template that inserts the human direct-assessment annotation rubric—score ranges and scoring instructions—into the instruction context, asking the LLM to score a translation using the same criteria human annotators use. The AG prompt carries the argument in three ways: it is the intervention that improves zero-shot and in-context-learning performance relative to the GEMBA and translation-evaluator prompts, it is the instruction template used for LoRA fine-tuning of the LLMs, and it is the shared basis on which the LLM results are compared against fine-tuned encoder models (TransQuest with InfoXLM, CometKiwi with XLM-R-XL) in both unified multilingual training and independent language-pair training settings. The comparison against those encoders, measured by Spearman correlation with human DA scores, is what establishes the paper's headline gap.

What would settle it

Re-run the zero-shot and in-context-learning evaluations without dropping unparseable outputs—treating missing scores as failed predictions (e.g., imputing the worst score or using rank-based handling)—and check whether the human DA scores of dropped segments are systematically lower than those of retained segments; if they are, the reported Spearman correlations (e.g., AG-prompt gains over GEMBA) would shrink or reverse.

Watch

Extended reading notes

Core claim

The paper claims that for reference-less, segment-level quality estimation (predicting a 0–100 direct-assessment score without a reference translation) on eight WMT low-resource pairs, open-weight LLMs under 13B parameters are consistently outperformed by fine-tuned encoder-based QE systems such as TransQuest (InfoXLM) and CometKiwi (XLM-R-XL). A new prompt that embeds the human annotation guidelines (the AG prompt) improves zero-shot LLM correlation for most pairs, and instruction fine-tuning narrows the gap but does not close it: in the unified multilingual setting the encoder models still win on every pair, while in the language-pair-specific setting LLMs beat the encoder baseline on En-Hi, En-Ta, and En-Te. The paper attributes part of the remaining gap to tokenization, showing that LLM tokenizers inflate token counts for morphologically rich non-Latin-script languages, which harms cross-lingual semantic matching, and that English-target pairs consistently score higher than Indic-target pairs.

Load-bearing premise

The paper drops every LLM output that does not contain a parseable numeric score before computing correlations in the zero-shot and in-context-learning settings, and in several of those settings more than 10% of outputs are dropped; if score omission happens more often for hard or low-quality segments, the reported correlations are computed on an unrepresentative subset and the prompt comparisons could be biased.

Editorial extensions

If this is right

  • If the result holds, reference-less segment-level QE for low-resource pairs should keep using fine-tuned encoder models rather than prompting sub-13B open LLMs.
  • The AG prompt—annotation guidelines inserted as scoring instructions—is a cheap, zero-shot improvement over existing prompts for most pairs, so guideline-based prompting can be reused wherever rubric-based scoring is needed.
  • Instruction fine-tuning with the AG prompt narrows the gap and, in the independent language-pair setting, lets LLMs beat the encoder baseline on En-Hi, En-Ta, and En-Te, so the encoder lead is not absolute.
  • Tokenization skew for morphologically rich, non-Latin-script languages is a concrete bottleneck: improving tokenizers or pre-training data distribution for those scripts should improve LLM-based QE more than adding prompt examples.
  • Because English-target pairs consistently score higher, the result raises the question of how much of LLM 'multilingual' capability is actually target-English proficiency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the practice of dropping LLM outputs without a parseable score before computing correlations (over 10% in several zero-shot and ICL cells, as the paper's own tables show) may make the reported correlations optimistic, and a robustness check that imputes or penalizes missing scores could change the relative ordering of prompts.
  • Editorial inference: the tokenization analysis implies a testable extension—retokenizing inputs via transliteration to Latin script or using a morpheme-aware tokenizer before QE prompting should improve correlations for Tamil, Telugu, Marathi, and Hindi if the paper's mechanism is correct.
  • Editorial inference: the AG prompt's success suggests annotation rubrics act as a form of soft supervision, and the same design could transfer to other regression-style evaluation tasks (e.g., summarization quality, grammaticality scoring) where official guidelines exist.
  • Editorial inference: the English-target advantage hints that reference-less QE with English as the translation language may partly reduce to monolingual fluency judgment; reversing language-pair directions on matched data would isolate target-language effects from source-language effects.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper evaluates reference-less segment-level quality estimation (QE) for eight low-resource language pairs using open-weight LLMs (Gemma-7B, Llama-2-7B/13B, OpenChat-3.5) in zero-shot, in-context learning (ICL), and adapter-based instruction fine-tuning (ILT and UMT settings). It proposes an annotation-guideline-based prompt (AG) and compares against the GEMBA and translation-evaluator (TE) prompts. The main findings are that fine-tuned encoder-based QE models (TransQuest, CometKiwi) outperform the LLM-based approaches, and that the AG prompt improves zero-shot performance for some language pairs when the best-performing model (OpenChat) is selected. The paper also reports a tokenization analysis and an error analysis for En-Ta, and releases the fine-tuned adapters and datasets.

Significance. If the main claim holds, the paper provides a useful, reproducible benchmark showing that, for low-resource reference-less QE, sub-13B open-weight LLMs are not yet competitive with fine-tuned encoder-based models, and it offers practical instruction-tuning adapters and public data. The fine-tuned comparison (Table 2) is complete and supports the central conclusion that encoder-based models dominate. The paper also contributes a concrete error analysis and tokenization analysis that point to pre-training and tokenization issues for morphologically rich low-resource languages. However, the zero-shot contribution is overstated: the claimed AG-prompt improvement is driven by a single model (OpenChat), and the evaluation methodology for zero-shot/ICL excludes a substantial number of unparseable outputs, which weakens the confidence in the prompt-level comparisons.

major comments (2)
  1. [§4, Table 1] The claim that the AG prompt improves zero-shot performance is not supported by the per-model results in Table 1. For Gemma-7B, AG's Spearman correlation is below GEMBA in all 8 language pairs; for Llama-2-7B it is below in 6 of 8; for Llama-2-13B it is below in 4 of 8; and for OpenChat it is below in 3 of 8. Across the 32 model-pair cells, AG beats GEMBA in fewer than half of the comparisons, and every asterisked best zero-shot score belongs to OpenChat. The paper's statement in Section 4 that 'Our proposed AG prompt achieved the highest scores in the zero-shot setting for most language pairs' is therefore an artifact of selecting the best model per language pair, rather than a consistent per-model effect. Moreover, the statistical significance markers (†) are based on a paired t-test between predictions and human scores, not between prompts or models, so no significance evidence is provided for the AG-vs-GEMBA differences. The zero-shot contribution should be re-scoped to OpenChat, or supported with a per-model comparison and corresponding significance tests.
  2. [§3.4, Appendices F and G] In zero-shot and ICL experiments, outputs that do not contain a parseable score are excluded before computing correlations. The appendix tables show that this affects more than 10% of inferences in several settings (e.g., TE prompt: En-Gu with Llama-2-13B drops 102/1000, En-Ta with Llama-2-13B drops 143/1000, Ne-En with Llama-2-13B drops 114/1000; 3-ICL-AG: En-Mr with Gemma-7B drops 111/1000). If the model is more likely to omit a score for difficult or marginal segments, the reported correlations are computed on an unrepresentative subset and could overstate or invert the prompt-level comparisons. Since the zero-shot and few-shot results are used to support the AG-prompt improvement claim, the main tables should report the number of dropped rows per cell (or a summary), and the authors should analyze whether dropped segments differ systematically from retained ones. The fine-tuned results in Table 2 are not affected, as all fine-tuned outputs contained scores.
minor comments (4)
  1. [§5, Tokenization analysis] The tokenization analysis is based on 100 sentences per language pair with no error bars or statistical significance testing. Since the claim that tokenization discrepancies explain QE performance is correlational, the authors should either add variability estimates or soften the causal language in the discussion.
  2. [Appendix C, Table 3] The language pair labeled 'Nepalis - English (Ne-En)' should be 'Nepali - English'.
  3. [§4, first paragraph] The phrase 'For En to {Hi, Te} the AG prompt scores were very close to those of the best scores' is imprecise: for En-Hi the best zero-shot score is GEMBA (0.254) and AG is 0.253, but for En-Te the best is GEMBA (0.145) and AG is 0.121, which is not particularly close. Please clarify what 'close' means with a numerical tolerance.
  4. [§5, Error Analysis] The error analysis is conducted only for En-Ta using the OpenChat model, and the percentages in Figure 3 sum to 98.6% rather than 100%. Please clarify whether the remaining 1.4% corresponds to an 'other' category or annotator disagreement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark evaluation against external WMT DA scores, with no fitted parameters or construction-based predictions.

full rationale

This paper is an empirical study in which all conclusions are measured against externally provided human DA scores from the WMT QE shared tasks. There is no derivation of a result from its own inputs: the AG prompt is a newly proposed prompt template, not fitted to the test labels; the zero-shot, ICL, and fine-tuned results are all computed correlations between model outputs and independent human annotations. The self-citations present in the paper, such as Sindhujan et al. (2023) for the TransQuest framework and the WMT shared-task findings papers co-authored by some of the present authors, are used as dataset/background references and baseline implementations, not as load-bearing justifications for the paper's conclusions. No uniqueness theorem is imported from the authors' prior work, no ansatz is smuggled in via citation, and no known result is simply renamed. The paper's central claims, including the relative performance of encoder-based QE models versus LLMs, are directly supported by external benchmark numbers. A potential weakness regarding the exclusion of unparseable LLM outputs before correlation computation is a methodological and correctness concern, not a circularity concern. Therefore, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard QE benchmark assumptions, on the treatment of dropped LLM outputs, and on framework correctness. There are no invented entities and no fitted parameters in the sense of a derivation; the listed free parameters are hyperparameters chosen by hand.

free parameters (3)
  • LoRA rank = 64
    Chosen by hand in Section 3.3 as a balance between adaptation quality and resource use; affects fine-tuned LLM results but not the qualitative conclusion.
  • Temperature = 0
    Selected in Section 3.3 for stable outputs; not fitted to data.
  • ICL example counts = 3, 5, 7
    Design choice in Section 3.2 with examples drawn from DA score ranges; no search is reported, so these are not fitted values.
assumptions (5)
  • domain assumption Human DA mean scores are a valid ground truth for translation quality.
    Standard WMT QE setup; the paper uses the mean of at least three human annotations as the regression target (Section 1).
  • domain assumption WMT train/test splits do not leak into LLM pretraining.
    LLMs are evaluated zero-shot on WMT22/23 test sets without checking contamination; if test segments appeared in pretraining, zero-shot scores would be inflated (Section 3.1).
  • domain assumption Correlations computed after dropping outputs without scores are representative of the full test set.
    Load-bearing for zero-shot and ICL comparisons; Section 3.4 excludes outputs lacking a parseable score, and several settings drop more than 10% of rows.
  • domain assumption The vLLM and LLaMA-Factory implementations faithfully reproduce the released model behavior.
    All zero-shot, ICL, and fine-tuning results rely on these frameworks with default settings (Section 3.3).
  • domain assumption The tokenization sample of 100 sentences per language pair is representative.
    Tokenization conclusions in Section 5 are drawn from a 100-sentence sample per pair with no reported sampling error or significance test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages." pith.science (2026). https://pith.science/paper/Z2WFYDRM

@misc{pith2026250104473,
  author       = {Pith},
  title        = {Pith review of: When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z2WFYDRM}},
  note         = {Machine review of arXiv:2501.04473}
}
read the original abstract

This paper investigates the reference-less evaluation of machine translation for low-resource language pairs, known as quality estimation (QE). Segment-level QE is a challenging cross-lingual language understanding task that provides a quality score (0-100) to the translated output. We comprehensively evaluate large language models (LLMs) in zero/few-shot scenarios and perform instruction fine-tuning using a novel prompt based on annotation guidelines. Our results indicate that prompt-based approaches are outperformed by the encoder-based fine-tuned QE models. Our error analysis reveals tokenization issues, along with errors due to transliteration and named entities, and argues for refinement in LLM pre-training for cross-lingual tasks. We release the data, and models trained publicly for further research.

Figures

Figures reproduced from arXiv: 2501.04473 by the authors.

Figure 1
Figure 1. The proposed AG prompt which augments scoring instructions within the context. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Error types and their percent contribution. shows enhanced performance with English as the target language and the data distribution for other language pairs is a concern for most pre-training setups (Uthus et al., 2023), including those of encoder models. This observation is in line with the study of Nguyen et al. (2024) and indicates that language models are likely more proficient when English is the target langua… view at source ↗
Figure 4
Figure 4. The graphs compare the original word counts with the model-generated token counts for selected inputs, as described in Section 5. This comparison includes both low-resource language pairs (En-Ta, Et-En) and a high-resource language pair (En-De). A detailed image covering all language pairs is provided in the Appendix E. that observed for other low-resource languages. This holds true even for LLMs as well. This reduc… view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Our proposed AG prompt for in-context learning. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: GEMBA prompt (Kocmi and Federmann, 2023) The GEMBA prompt is part of the GEMBA (GPT Estimation Metric Based Assessment) method, which uses GPT-based language models to evaluate translation quality. The GEMBA prompt evaluates translation quality by scoring each translat…
Figure 7
Figure 7. Figure 7: TE prompt (Mujadia et al., 2023) The TE (Translation Evaluator) prompt instructs the model to act as an experienced translation evaluator, explicitly presenting the source language, source text, target language, and translated text. The prompt concludes with the model …
Figure 8
Figure 8. Figure 8: This image shows the number of data belonging to each DA score range of each language pair in the train [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: A comparative analysis of the total word count of source and target sentences versus the count of [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: The examples are taken from our study (See in section [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: The above graphs show how the Spearman scores varied for each experimental setting with different [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: This bar graph shows the size (number of parameters) of the large language models we have utilized for our [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: This bar graph shows the disk space utilization of the large language models we have utilized for our experiments [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LLM-Based Evaluation of Low-Resource Machine Translation: A Reference-less Dialect Guided Approach with a Refined Sylheti-English Benchmark

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A dialect-guided prompt with tokenizer extension and a regression head improves reference-less machine translation evaluation for Sylheti-English, with best Spearman gain +0.1083 over an annotation-guided baseline.

Reference graph

Works this paper leans on

43 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Guerreiro, Diptesh Kanojia, Jos \'e G

    Frederic Blain, Chrysoula Zerva, Ricardo Rei, Nuno M. Guerreiro, Diptesh Kanojia, Jos \'e G. C. de Souza, Beatriz Silva, T \^a nia Vaz, Yan Jingxuan, Fatemeh Azadi, Constantin Orasan, and Andr \'e Martins. 2023. https://doi.org/10.18653/v1/2023.wmt-1.52 Findings of the WMT 2023 shared task on quality estimation . In Proceedings of the Eighth Conference on...

    doi:10.18653/v1/2023.wmt-1.52 2023

  3. [3]

    Jiuhai Chen, Lichang Chen, Chen Zhu, and Tianyi Zhou. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.745 How many demonstrations do you need for in-context learning? In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11149--11159, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-emnlp.745 2023

  4. [4]

    Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. Noise reduction in speech processing, pages 1--4

    2009

  5. [5]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. https://arxiv.org/abs/2305.14314 Qlora: Efficient finetuning of quantized llms . Preprint, arXiv:2305.14314

    arXiv 2023

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171--4186

    2019

  7. [7]

    Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, Andr \'e Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. https://doi.org/10.18653/v1/2023.wmt-1.100 The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation . In Proceedings of the Eighth Confer...

    doi:10.18653/v1/2023.wmt-1.100 2023

  8. [8]

    Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. 2021. https://arxiv.org/abs/2105.00572 Larger-scale transformers for multilingual masked language modeling . CoRR, abs/2105.00572

    arXiv 2021

Show all 43 references
  1. [9]

    Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. https://aclanthology.org/W13-2305 Continuous measurement scales in human evaluation of machine translation . In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, p...

    2013

  2. [10]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    2021 arXiv

  3. [11]

    Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting. arXiv preprint arXiv:2305.07004

    2023 arXiv

  4. [12]

    Xu Huang, Zhirui Zhang, Xiang Geng, Yichao Du, Jiajun Chen, and Shujian Huang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.211 Lost in the source language: How large language models evaluate the quality of machine translation . In Findings of the Association for Comput...

    2024 doi

  5. [13]

    Cong-Phap Huynh, Christian Boitet, and Herv \'e Blanchon. 2008. Sectra\_w. 1: an online collaborative system for evaluating, post-editing and presenting mt translation corpora. In LREC, volume 8, pages 28--30

    2008

  6. [14]

    Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, To...

    2023 doi

  7. [15]

    Tom Kocmi and Christian Federmann. 2023. https://aclanthology.org/2023.eamt-1.19 Large language models are state-of-the-art evaluators of translation quality . In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193--203, Tam...

    2023

  8. [16]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  9. [17]

    Mirella Lapata. 2006. https://doi.org/10.1162/coli.2006.32.4.471 Automatic evaluation of information ordering: Kendall ' s tau . Computational Linguistics, 32(4):471--484

    2006 doi

  10. [18]

    Yuang Li, Chang Su, Ming Zhu, Mengyao Piao, Xinglin Lyu, Min Zhang, and Hao Yang. 2023. https://doi.org/10.18653/v1/2023.wmt-1.72 HW - TSC 2023 submission for the quality estimation shared task . In Proceedings of the Eighth Conference on Machine Translation, pages 835--840, S...

    2023 doi

  11. [19]

    Chi-kiu Lo, Meriem Beloucif, Markus Saers, and Dekai Wu. 2014. Xmeant: Better semantic mt evaluation without reference translations. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 765--771

    2014

  12. [20]

    Arle Lommel, Aljoscha Burchardt, Maja Popovi \'c , Kim Harris, Eleftherios Avramidis, and Hans Uszkoreit. 2014. https://aclanthology.org/2014.eamt-1.38 Using a new analytic measure for the annotation and analysis of MT errors on real data . In Proceedings of the 17th Annual Co...

    2014

  13. [21]

    Shushen Manakhimova, Eleftherios Avramidis, Vivien Macketanz, Ekaterina Lapshinova-Koltunski, Sergei Bagdasarov, and Sebastian M \"o ller. 2023. https://doi.org/10.18653/v1/2023.wmt-1.23 Linguistically motivated evaluation of the 2023 state-of-the-art machine translation: Can ...

    2023 doi

  14. [22]

    Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros,...

    2024 arXiv

  15. [23]

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. https://arxiv.org/abs/1710.03740 Mixed precision training . Preprint, arXiv:1710.03740

    2018 arXiv

  16. [24]

    Vandan Mujadia, Pruthwik Mishra, Arafat Ahsan, and Dipti M. Sharma. 2023. https://aclanthology.org/2023.icon-1.28 Towards large language model driven reference-less translation evaluation for E nglish and I ndian language . In Proceedings of the 20th International Conference o...

    2023

  17. [25]

    Xuan-Phi Nguyen, Mahani Aljunied, Shafiq Joty, and Lidong Bing. 2024. https://aclanthology.org/2024.acl-long.192 Democratizing LLM s for low-resource languages by leveraging their E nglish dominant abilities with linguistically-diverse prompts . In Proceedings of the 62nd Annu...

    2024

  18. [26]

    Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2024. Language model tokenizers introduce unfairness between languages. Advances in Neural Information Processing Systems, 36

    2024

  19. [27]

    Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020. https://doi.org/10.18653/v1/2020.coling-main.445 T rans Q uest: Translation quality estimation with cross-lingual transformers . In Proceedings of the 28th International Conference on Computational Linguistics, p...

    2020 doi

  20. [28]

    Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, Andr \'e F. T. Martins, and Alon Lavie. 2021. https://aclanthology.org/2021.wmt-1.111 Are references really needed? unbabel- IST 2021 submission for the metrics shared t...

    2021

  21. [29]

    Guerreiro, Jos \ A Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, Jos \'e G

    Ricardo Rei, Nuno M. Guerreiro, Jos \ A Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, Jos \'e G. C. de Souza, and Andr \'e Martins. 2023. https://doi.org/10.18653/v1/2023.wmt-1.73 Scaling up C omet K iwi: Unbabel- IST 2023 submission for the quality estimation shared t...

    2023 doi

  22. [30]

    Fran c ois Remy, Pieter Delobelle, Hayastan Avetisyan, Alfiya Khabibullina, Miryam de Lhoneux, and Thomas Demeester. 2024. Trans-tokenization and cross-lingual vocabulary transfers: Language adaptation of llms for low-resource nlp. arXiv preprint arXiv:2408.04303

    2024 arXiv

  23. [31]

    Mortensen, and Graham Neubig

    Nathaniel Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. 2023. https://doi.org/10.18653/v1/2023.wmt-1.40 C hat GPT MT : Competitive for high- (but not low-) resource languages . In Proceedings of the Eighth Conference on Machine Translation, pages 392--418, Sing...

    2023 doi

  24. [32]

    Philip Sedgwick. 2014. Spearman’s rank correlation coefficient. Bmj, 349

    2014

  25. [33]

    Archchana Sindhujan, Diptesh Kanojia, Constantin Orasan, and Tharindu Ranasinghe. 2023. https://doi.org/10.18653/v1/2023.wmt-1.74 S urrey AI 2023 submission for the quality estimation shared task . In Proceedings of the Eighth Conference on Machine Translation, pages 849--855,...

    2023 doi

  26. [34]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  27. [35]

    David Uthus, Santiago Ontanon, Joshua Ainslie, and Mandy Guo. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.628 m L ong T 5: A multilingual and efficient text-to-text transformer for longer sequences . In Findings of the Association for Computational Linguistics: EMNLP...

    2023 doi

  28. [36]

    Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023. https://arxiv.org/abs/arXiv:2309.11235 Openchat: Advancing open-source language models with mixed-quality data

    2023 arXiv

  29. [37]

    Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Wang, and Lei Li. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.365 INSTRUCTSCORE : Towards explainable text generation evaluation with automatic feedback . In Proceedings of the 2023 Conference ...

    2023 doi

  30. [38]

    Chrysoula Zerva, Fr \'e d \'e ric Blain, Ricardo Rei, Piyawat Lertvittayakumjorn, Jos \'e G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Or a san, Marina Fomicheva, Andr \'e F. T. Martins, and Lucia Specia. 2022. https://aclanthology.org/2022.wmt-1.3 F...

    2022

  31. [39]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong W...

    2023 arXiv

  32. [40]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma. 2024. http://arxiv.org/abs/2403.13372 Llamafactory: Unified efficient fine-tuning of 100+ language models . arXiv preprint arXiv:2403.13372

    2024 arXiv

  33. [41]

    Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.968 Context-faithful prompting for large language models . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14544--14556, Singapore. As...

    2023 doi

  34. [42]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  35. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.