{"id":"85f00370-6be6-4ddc-b92b-0aa2742606be","arxiv_id":"2501.04473","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A systematic comparison shows prompt-based LLMs underperform fine-tuned encoder models for reference-less translation quality estimation across eight low-resource language pairs.","lead":"The paper tests whether large language models can score machine translation quality without reference translations for eight low-resource language pairs. It finds that fine-tuned encoder models still beat prompt-based LLMs, and it releases prompts, data, and trained models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed zero-shot improvement from the AG prompt is an OpenChat effect: across Gemma and Llama models, AG usually underperforms GEMBA, so the paper's central contribution claim overstates the evidence.","rationale":"The reader's selective-exclusion concern is legitimate, but it mostly affects the zero-shot/ICL sub-analyses and would, if anything, make the fine-tuned-encoder advantage conservative, since dropped LLM outputs are excluded from the LLM side. The more direct threat to the paper's stated contribution is that Table 1 itself undercuts the general claim that the AG prompt improves zero-shot performance. The best-per-language-pair asterisks are all OpenChat, and AG is numerically worse than GEMBA for the majority of Gemma and Llama model-pair cells. This is a claim-level issue that can be settled by reanalyzing the published table, without additional experiments. The main encoder-versus-prompt conclusion remains plausible, and the released adapters and benchmark analysis are useful, so the appropriate outcome is still conditional: the AG-prompt contribution and any significance statements built on the paired t-test need to be corrected or appropriately qualified.","tokens_in":19862,"tokens_out":13559,"duration_ms":137033,"concrete_test":"From Table 1, form the 32 paired differences d = rho_AG - rho_GEMBA for every model and language pair. Compute the mean difference and a paired sign test; then recompute after excluding OpenChat. If the mean difference is not positive, or is positive only because of OpenChat, revise Contribution 1 and the Section 4 sentence to state that AG yields the best zero-shot score for a majority of language pairs only when paired with OpenChat, and that AG does not improve over GEMBA for the Gemma and Llama models. No new inference runs are needed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's Contribution 1 and Section 4 assert that the AG prompt improves zero-shot QE performance and achieves the highest zero-shot scores for most language pairs. The supporting evidence in Table 1 is only consistent with this when the best-performing model, OpenChat-3.5, is selected per language pair; every asterisked best zero-shot score in Table 1 belongs to OpenChat. Per model, AG is not an improvement: for Gemma-7B, AG's Spearman correlation is below GEMBA in all 8 language pairs; for Llama-2-7B, below in 6 of 8; for Llama-2-13B, below in 4 of 8; and for OpenChat, below in 3 of 8. Across the 32 model-pair cells, AG beats GEMBA in fewer than half of the comparisons. The paper reports no paired significance test over model-pair cells, so the 'AG improves zero-shot performance' contribution rests on an implicit best-model selection that is not stated as such. This is an internal inconsistency with the authors' own Table 1, not a disagreement with prior work.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates reference-less segment-level quality estimation (QE) for eight low-resource language pairs using open-weight LLMs (Gemma-7B, Llama-2-7B/13B, OpenChat-3.5) in zero-shot, in-context learning (ICL), and adapter-based instruction fine-tuning (ILT and UMT settings). It proposes an annotation-guideline-based prompt (AG) and compares against the GEMBA and translation-evaluator (TE) prompts. The main findings are that fine-tuned encoder-based QE models (TransQuest, CometKiwi) outperform the LLM-based approaches, and that the AG prompt improves zero-shot performance for some language pairs when the best-performing model (OpenChat) is selected. The paper also reports a tokenization analysis and an error analysis for En-Ta, and releases the fine-tuned adapters and datasets.","tokens_in":20097,"tokens_out":3198,"duration_ms":29959,"significance":"If the main claim holds, the paper provides a useful, reproducible benchmark showing that, for low-resource reference-less QE, sub-13B open-weight LLMs are not yet competitive with fine-tuned encoder-based models, and it offers practical instruction-tuning adapters and public data. The fine-tuned comparison (Table 2) is complete and supports the central conclusion that encoder-based models dominate. The paper also contributes a concrete error analysis and tokenization analysis that point to pre-training and tokenization issues for morphologically rich low-resource languages. However, the zero-shot contribution is overstated: the claimed AG-prompt improvement is driven by a single model (OpenChat), and the evaluation methodology for zero-shot/ICL excludes a substantial number of unparseable outputs, which weakens the confidence in the prompt-level comparisons.","major_comments":[{"comment":"The claim that the AG prompt improves zero-shot performance is not supported by the per-model results in Table 1. For Gemma-7B, AG's Spearman correlation is below GEMBA in all 8 language pairs; for Llama-2-7B it is below in 6 of 8; for Llama-2-13B it is below in 4 of 8; and for OpenChat it is below in 3 of 8. Across the 32 model-pair cells, AG beats GEMBA in fewer than half of the comparisons, and every asterisked best zero-shot score belongs to OpenChat. The paper's statement in Section 4 that 'Our proposed AG prompt achieved the highest scores in the zero-shot setting for most language pairs' is therefore an artifact of selecting the best model per language pair, rather than a consistent per-model effect. Moreover, the statistical significance markers (†) are based on a paired t-test between predictions and human scores, not between prompts or models, so no significance evidence is provided for the AG-vs-GEMBA differences. The zero-shot contribution should be re-scoped to OpenChat, or supported with a per-model comparison and corresponding significance tests.","section":"§4, Table 1"},{"comment":"In zero-shot and ICL experiments, outputs that do not contain a parseable score are excluded before computing correlations. The appendix tables show that this affects more than 10% of inferences in several settings (e.g., TE prompt: En-Gu with Llama-2-13B drops 102/1000, En-Ta with Llama-2-13B drops 143/1000, Ne-En with Llama-2-13B drops 114/1000; 3-ICL-AG: En-Mr with Gemma-7B drops 111/1000). If the model is more likely to omit a score for difficult or marginal segments, the reported correlations are computed on an unrepresentative subset and could overstate or invert the prompt-level comparisons. Since the zero-shot and few-shot results are used to support the AG-prompt improvement claim, the main tables should report the number of dropped rows per cell (or a summary), and the authors should analyze whether dropped segments differ systematically from retained ones. The fine-tuned results in Table 2 are not affected, as all fine-tuned outputs contained scores.","section":"§3.4, Appendices F and G"}],"minor_comments":[{"comment":"The tokenization analysis is based on 100 sentences per language pair with no error bars or statistical significance testing. Since the claim that tokenization discrepancies explain QE performance is correlational, the authors should either add variability estimates or soften the causal language in the discussion.","section":"§5, Tokenization analysis"},{"comment":"The language pair labeled 'Nepalis - English (Ne-En)' should be 'Nepali - English'.","section":"Appendix C, Table 3"},{"comment":"The phrase 'For En to {Hi, Te} the AG prompt scores were very close to those of the best scores' is imprecise: for En-Hi the best zero-shot score is GEMBA (0.254) and AG is 0.253, but for En-Te the best is GEMBA (0.145) and AG is 0.121, which is not particularly close. Please clarify what 'close' means with a numerical tolerance.","section":"§4, first paragraph"},{"comment":"The error analysis is conducted only for En-Ta using the OpenChat model, and the percentages in Figure 3 sum to 98.6% rather than 100%. Please clarify whether the remaining 1.4% corresponds to an 'other' category or annotator disagreement.","section":"§5, Error Analysis"}],"recommendation":"major_revision","confidential_remarks":"The main fine-tuning comparison is solid and the release of adapters/data is a good contribution. The AG-prompt zero-shot claim is the primary weakness: it is an OpenChat-specific effect and is not supported by the per-model table. The dropped-output issue also needs to be addressed quantitatively before the zero-shot/ICL conclusions can be accepted. The paper is within scope for a computational linguistics venue and, with a more careful presentation of the zero-shot results, could become a useful reference for the QE community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the negative finding is real, and the positive finding is oversold. If you read only the abstract, you'd think the AG prompt is a clear win. Table 1 says otherwise: AG beats GEMBA in only 11 of 32 model-pair cells, and every best zero-shot score comes from OpenChat. Per model, Gemma is worse with AG on all eight pairs; Llama-2-7B worse on six; Llama-2-13B worse on four; OpenChat better on five. The paper's Contribution 1 is an implicit best-model selection dressed up as a prompt effect. The stress-test note is right.\n\nWhat's actually good: the fine-tuning comparison is complete and convincing. TransQuest and CometKiwi beat all four LLMs on almost every pair in both ILT and UMT settings, and that central conclusion is unaffected by the AG issue. The benchmark itself is useful: eight low-resource pairs, four open LLMs, zero-shot/ICL/fine-tuning in one place, plus released data and adapters. The appendices are unusually transparent about dropped outputs and statistical significance, which earns real credit.\n\nSoft spots in proportion: the row dropping is the second real problem. Zero-shot and ICL correlations are computed on subsets after removing outputs without parseable scores; in several settings more than 10% of rows vanish (TE prompt, several Gemma ICL runs). If score omission correlates with segment difficulty, the prompt comparisons sit on unrepresentative data. The authors flag this in Appendix F but still build Contribution 1 on it. The tokenization analysis uses 100 unverified sentences per pair with no error bars; it is suggestive, not evidence. No code is released, though data and models are.\n\nWho this is for: people working on QE, low-resource MT evaluation, or prompt design for LLM assessment. It is a solid data point that under-13B open LLMs are not competitive with fine-tuned encoders for reference-less segment-level QE, but it needs revision before acceptance: re-frame the AG claim, report paired significance over model-pair cells, and handle dropped rows with robustness checks.\n\nRecommendation: yes, send it to peer review. The empirical core is publishable and a serious referee can push the framing into line with the evidence.","headline":"The paper's real contribution is a solid negative result—fine-tuned encoders beat open LLMs for low-resource QE—while the claimed AG-prompt win is an OpenChat effect that does not survive Table 1.","tokens_in":20601,"tokens_out":4475,"would_cite":true,"duration_ms":40932,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that for reference-less, segment-level translation-quality estimation across eight low-resource language pairs, prompt-based LLM approaches are consistently outperformed by fine-tuned encoder-based QE models.","keywords":["quality estimation","low-resource machine translation","reference-less evaluation","large language models","prompt engineering","annotation guidelines","in-context learning","tokenization"],"falsifier":"Re-run the zero-shot and in-context-learning evaluations without dropping unparseable outputs—treating missing scores as failed predictions (e.g., imputing the worst score or using rank-based handling)—and check whether the human DA scores of dropped segments are systematically lower than those of retained segments; if they are, the reported Spearman correlations (e.g., AG-prompt gains over GEMBA) would shrink or reverse.","tokens_in":2255,"feed_emoji":"🎯","tokens_out":2508,"duration_ms":92452,"temperature":0.7,"pith_summary":"The paper sets out to test whether open-weight large language models can do reference-less quality estimation—scoring a machine translation's quality on a 0–100 scale with no human reference—for low-resource language pairs, and to improve their performance with a new prompt built from human annotation guidelines. Across eight WMT pairs (English to Gujarati, Hindi, Marathi, Tamil, Telugu; Estonian, Nepali, Sinhala to English), it compares zero-shot, in-context learning, and LoRA instruction fine-tuning of Gemma-7B, Llama-2-7B/13B, and OpenChat-3.5 against fine-tuned encoder-based QE systems. Its central result is that the prompt-based LLM approaches are outperformed by the encoder-based fine-tuned models in almost all settings, with the gap closing but not closing fully after instruction fine-tuning. The paper's own contribution is the AG prompt, which improves zero-shot correlation for most pairs, plus a tokenization analysis tracing part of the failure to LLM tokenizers inflating token counts for morphologically rich non-Latin scripts. If true, it means the current wave of small open LLMs is not yet a drop-in replacement for encoder QE models in low-resource settings, and that cross-lingual quality scoring deserves a place in LLM evaluation suites.","feed_headline":"Small LLMs lose to fine-tuned encoders on MT quality scoring","feed_subtitle":"On eight low-resource pairs, even fine-tuned open LLMs stay behind encoder QE models.","key_machinery":"The central object is the AG prompt (annotation-guidelines prompt), a prompt template that inserts the human direct-assessment annotation rubric—score ranges and scoring instructions—into the instruction context, asking the LLM to score a translation using the same criteria human annotators use. The AG prompt carries the argument in three ways: it is the intervention that improves zero-shot and in-context-learning performance relative to the GEMBA and translation-evaluator prompts, it is the instruction template used for LoRA fine-tuning of the LLMs, and it is the shared basis on which the LLM results are compared against fine-tuned encoder models (TransQuest with InfoXLM, CometKiwi with XLM-R-XL) in both unified multilingual training and independent language-pair training settings. The comparison against those encoders, measured by Spearman correlation with human DA scores, is what establishes the paper's headline gap.","core_discovery":"The paper claims that for reference-less, segment-level quality estimation (predicting a 0–100 direct-assessment score without a reference translation) on eight WMT low-resource pairs, open-weight LLMs under 13B parameters are consistently outperformed by fine-tuned encoder-based QE systems such as TransQuest (InfoXLM) and CometKiwi (XLM-R-XL). A new prompt that embeds the human annotation guidelines (the AG prompt) improves zero-shot LLM correlation for most pairs, and instruction fine-tuning narrows the gap but does not close it: in the unified multilingual setting the encoder models still win on every pair, while in the language-pair-specific setting LLMs beat the encoder baseline on En-Hi, En-Ta, and En-Te. The paper attributes part of the remaining gap to tokenization, showing that LLM tokenizers inflate token counts for morphologically rich non-Latin-script languages, which harms cross-lingual semantic matching, and that English-target pairs consistently score higher than Indic-target pairs.","pith_inferences":["Editorial inference: the practice of dropping LLM outputs without a parseable score before computing correlations (over 10% in several zero-shot and ICL cells, as the paper's own tables show) may make the reported correlations optimistic, and a robustness check that imputes or penalizes missing scores could change the relative ordering of prompts.","Editorial inference: the tokenization analysis implies a testable extension—retokenizing inputs via transliteration to Latin script or using a morpheme-aware tokenizer before QE prompting should improve correlations for Tamil, Telugu, Marathi, and Hindi if the paper's mechanism is correct.","Editorial inference: the AG prompt's success suggests annotation rubrics act as a form of soft supervision, and the same design could transfer to other regression-style evaluation tasks (e.g., summarization quality, grammaticality scoring) where official guidelines exist.","Editorial inference: the English-target advantage hints that reference-less QE with English as the translation language may partly reduce to monolingual fluency judgment; reversing language-pair directions on matched data would isolate target-language effects from source-language effects."],"forward_implications":["If the result holds, reference-less segment-level QE for low-resource pairs should keep using fine-tuned encoder models rather than prompting sub-13B open LLMs.","The AG prompt—annotation guidelines inserted as scoring instructions—is a cheap, zero-shot improvement over existing prompts for most pairs, so guideline-based prompting can be reused wherever rubric-based scoring is needed.","Instruction fine-tuning with the AG prompt narrows the gap and, in the independent language-pair setting, lets LLMs beat the encoder baseline on En-Hi, En-Ta, and En-Te, so the encoder lead is not absolute.","Tokenization skew for morphologically rich, non-Latin-script languages is a concrete bottleneck: improving tokenizers or pre-training data distribution for those scripts should improve LLM-based QE more than adding prompt examples.","Because English-target pairs consistently score higher, the result raises the question of how much of LLM 'multilingual' capability is actually target-English proficiency."],"supporting_citations":[{"why":"Supplies the GEMBA prompt, the zero-shot reference-less baseline that the AG prompt is compared against and reported to beat for most pairs.","marker":"[Kocmi and Federmann, 2023]"},{"why":"Provides the WMT23 QE shared-task data and human DA annotations for En-Gu, En-Hi, En-Mr, En-Ta, and En-Te used in all training and test splits.","marker":"[Blain et al., 2023]"},{"why":"Provides the WMT22 QE shared-task data and DA annotations for Et-En, Ne-En, and Si-En, plus the high-resource pairs used in tokenization comparisons.","marker":"[Zerva et al., 2022]"},{"why":"Defines CometKiwi, the encoder-based fine-tuned QE model that outperforms all LLMs in the unified multilingual setting.","marker":"[Rei et al., 2023]"},{"why":"Defines TransQuest, the encoder-based cross-lingual transformer QE framework used as the fine-tuned encoder comparison in both ILT and UMT settings.","marker":"[Ranasinghe et al., 2020]"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method used to adapt the LLMs with the AG prompt in ILT and UMT settings.","marker":"[Hu et al., 2021]"},{"why":"Defines Llama-2-7B and Llama-2-13B, two of the LLMs evaluated in zero-shot, ICL, and fine-tuning settings.","marker":"[Touvron et al., 2023]"},{"why":"Defines Gemma-7B, an LLM evaluated across all settings and the best LLM in UMT for four language pairs.","marker":"[Mesnard et al., 2024]"},{"why":"Defines OpenChat-3.5, the LLM that achieves the highest zero-shot and ICL correlations among LLMs.","marker":"[Wang et al., 2023]"},{"why":"Defines XLM-R-XL, the encoder used by CometKiwi and in the tokenization analysis that motivates the paper's tokenization conclusions.","marker":"[Goyal et al., 2021]"}],"fun_headline_variants":["LLMs fall short in reference-less MT quality scoring","Encoder QE models still beat LLMs on low-resource pairs","Tokenization trips up LLMs in translation quality estimation","LLMs lose to fine-tuned encoders for MT quality scoring","Fine-tuned LLMs still trail encoder QE on eight language pairs"],"cache_read_input_tokens":22784,"weakest_assumption_plain":"The paper drops every LLM output that does not contain a parseable numeric score before computing correlations in the zero-shot and in-context-learning settings, and in several of those settings more than 10% of outputs are dropped; if score omission happens more often for hard or low-quality segments, the reported correlations are computed on an unrepresentative subset and the prompt comparisons could be biased.","fun_headline_variants_meta":{"raw":{"variants":["LLMs fall short in reference-less MT quality scoring","Encoder QE models still beat LLMs on low-resource pairs","Tokenization trips up LLMs in translation quality estimation","LLMs lose to fine-tuned encoders for MT quality scoring","Fine-tuned LLMs still trail encoder QE on eight language pairs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000373,"raw_usage":{"total_tokens":1961,"prompt_tokens":880,"completion_tokens":1081,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":995}},"tokens_in":496,"tokens_out":1081,"duration_ms":8210,"temperature":1.0,"reasoning_tokens":995,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:31:19.192842+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the zero-shot and in-context-learning evaluations without dropping unparseable outputs—treating missing scores as failed predictions (e.g., imputing the worst score or using rank-based handling)—and check whether the human DA scores of dropped segments are systematically lower than those of retained segments; if they are, the reported Spearman correlations (e.g., AG-prompt gains over GEMBA) would shrink or reverse.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the WMT22 QE shared-task data and DA annotations for Et-En, Ne-En, and Si-En, plus the high-resource pairs used in tokenization comparisons."}],"review_version":1}