Pith. sign in

REVIEW 3 major objections 5 minor 20 references

This paper argues that prompt scope and demonstration strategy are first-class variables in LLM translation evaluation, showing that dedicated MT systems still lead on Germanic pairs while few-shot prompts help some local LLMs and hurt othe

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:14 UTC pith:QIS7GJK6

load-bearing objection Solid empirical evaluation of prompt scope and demonstration selection for local LLM translation; core findings hold, but cross-model ordering rests on a single unreported configuration and needs artifact release and sharper statistics. the 3 major comments →

arxiv 2607.26286 v1 pith:QIS7GJK6 submitted 2026-07-28 cs.CL

Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation

Mihael Arcan This is my paper
classification cs.CL
keywords local LLM machine translationprompt scopefew-shot promptingdemonstration retrievalembedding similaritystructured output complianceFLORES benchmarkmulti-target translation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper treats prompt scope (one target language vs. an entire language family in a single JSON response) and demonstration selection (random, lexical, or embedding-based) as experimental variables rather than fixed evaluation details. On FLORES English-to-nine-EU-language translation, it finds that dedicated MT systems remain strongest, that few-shot prompting consistently helps the two larger local LLMs while degrading the smallest, and that embedding retrieval is the best average strategy but only modestly better than random or lexical examples. The central message is that LLM translation evaluation should report structured-output compliance and prompt scope alongside BLEU, chrF++, and COMET scores.

Core claim

The central claim is that prompt scope and demonstration strategy materially change local LLM translation behavior, so evaluation of LLM translators must include them as variables. Concretely: dedicated MT baselines (OPUS-MT, NLLB-200) remain strongest overall, especially for Germanic languages; k=5 few-shot prompting helps mistral:latest and qwen2.5:14b but harms llama3.2:3b; embedding-based retrieval yields the best average gains for the models that benefit from demonstrations, yet its margin over random and lexical retrieval is small; and family-scope JSON prompting is feasible for the stronger models but causes large structured-output failures in the smallest model, particularly for Germ

What carries the argument

The experimental machinery is a controlled comparison of three local instruction-tuned LLMs (llama3.2:3b, mistral:latest, qwen2.5:14b) run via Ollama at temperature 0 on the FLORES devtest split, under two prompt scopes: single-target prompts (one language per call) and family-scope prompts (all languages of a language family in one JSON object). Few-shot conditions use k=5 demonstrations from FLORES dev selected randomly, by lexical similarity, or by embedding similarity. Quality is measured with sacreBLEU, chrF++, and COMET, and the family-scope condition adds compliance metrics: coverage, complete-output count, missing-target count, and prompt-echo count. These components define 'prompt s

Load-bearing premise

The cross-model ordering assumes that one fixed inference configuration—Ollama at temperature 0 with default prompt templates and the unpinned mistral:latest tag—gives stable, representative behavior for each model; if template details, quantization, context length, or model version drift alter which models benefit from few-shot examples or follow JSON output, the qualitative results would not generalize.

What would settle it

Re-run llama3.2:3b on the Germanic family-scope condition with a single JSON-formatted example in the prompt; if coverage rises from about 0.34 to above 0.9, the paper's attribution of structured-output failure to model size would be contradicted in favor of a formatting/template explanation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • LLM translation evaluation should include prompt scope and structured-output compliance measures, not just quality metrics.
  • Few-shot demonstration benefit is model-dependent; a random-demonstration control is necessary to separate format effects from relevance effects.
  • Dedicated MT systems (OPUS-MT, NLLB) remain a strong baseline for local consumer-hardware translation, particularly for Germanic languages.
  • Family-scope prompting can return multiple translations in one generation call, but only for models that reliably follow the JSON schema; smaller models may omit required languages.
  • Sentence-level similarity scores do not predict translation gains, so retrieval strategies should be reported as averages, not assumed per-sentence benefits.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The near-zero sentence-level correlation suggests demonstrations help mainly by anchoring format and output style rather than by semantic relevance; a test that uses format-only (unrelated) examples could isolate this mechanism.
  • A practical extension is to add a lightweight JSON-validation and re-prompt loop at inference time to recover missing targets in small models, potentially changing deployment decisions for multilingual local systems.
  • The paper's findings are likely to transfer to other structured multi-output tasks (e.g., generating translations plus summaries), where output-schema compliance is part of the quality definition.
  • The Germanic family-scope failure for the smallest model may stem from tokenizer or alignment issues rather than capacity alone; per-language coverage analysis would tell.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper reports an empirical evaluation of local instruction-tuned LLMs (llama3.2:3b, mistral:latest, qwen2.5:14b) for English-to-Romance and English-to-Germanic translation on the full FLORES devtest split. It compares zero-shot prompting against k=5 few-shot prompting with random, lexical-similarity, and embedding-similarity demonstration selection, and compares single-target prompts with JSON-formatted family-scope prompts across nine EU languages. The main claims are that dedicated MT systems remain strongest overall (especially for Germanic languages), few-shot prompting helps mistral:latest and qwen2.5:14b but hurts llama3.2:3b, embedding retrieval is best on average for the stronger LLMs but its margin over lexical and random retrieval is modest, and family-scope prompting is feasible for stronger LLMs but exposes structured-output failures in smaller models. The paper also reports compliance metrics, paired bootstrap confidence intervals for selected comparisons, and sentence-level retrieval diagnostics.

Significance. If the claims hold, the paper is a useful methodological contribution to LLM-based MT evaluation: it shows that prompt scope, demonstration selection, and structured-output compliance can materially change evaluation conclusions for local LLMs. The study has notable strengths: it uses the full FLORES devtest split, includes dedicated MT baselines, separates translation quality from output compliance, uses random demonstrations as a necessary control, reports sentence-level diagnostics, and organizes reproducible task files. However, the central model-level ordering and several significance claims currently rest on a single partially unreported inference configuration and on test-set-based selection without proper uncertainty quantification. With those concerns addressed, the paper would provide actionable guidance for practitioners evaluating local LLM translation systems.

major comments (3)
  1. [§4.2, §6, Tables 1/5/9] The cross-model comparison is run with a single partially unreported configuration: 'mistral:latest' is unpinned; no model digest, quantization, context length, Ollama version, or exact prompt template is given. The paper's own limitation section acknowledges that other decoding settings, context lengths, quantizations, or instruction templates may change quality and compliance. This is load-bearing because the headline results are model-level orderings (few-shot helps mistral/qwen but hurts llama; family-scope compliance differs sharply). As written, those claims may be properties of the specific configuration rather than of the models. Please pin exact checkpoints and report the full configuration; ideally add a small robustness check across templates/quantizations or explicitly re-scope all conclusions to the tested configuration.
  2. [Table 10] Paired comparisons select the best local LLM condition and the chrF++-best MT baseline per target on the same FLORES devtest, then report p-values/CIs for the selected pair. This selection invalidates nominal coverage/error rates; e.g., the headline 'Spanish gap not significant (p=0.520)' is a post-selection statement. Use pre-specified conditions, a validation split, or a multiple-comparison correction, and report the selection rule.
  3. [Tables 1/5/9] Central few-shot delta claims lack uncertainty. Table 9 reports llama3.2:3b mean deltas of -0.58/-0.60/-0.56 chrF++ across nine targets; at 1012 test sentences per target these magnitudes could easily be within sampling noise, yet the paper concludes 'few-shot prompting hurts llama3.2:3b.' Report paired bootstrap CIs or sentence-level paired tests for Table 9, and ideally test the model×condition interaction rather than reading separate model columns. This also affects the 'embedding best on average' claim, whose margins are smaller than model differences.
minor comments (5)
  1. [§4.3] The reproducibility statement should include exact software versions (Ollama, SacreBLEU, COMET) and model hashes or digests, not just the task-file structure.
  2. [Table 10] The caption uses 'chrF-best' without defining how the baseline is selected. Also clarify whether the CI and p-values are for chrF++ or COMET.
  3. [Tables 4/8] Because quality is computed over produced translations, low-coverage conditions (e.g., llama3.2:3b Germanic zero-shot) have quality scores that are not directly comparable to high-coverage conditions. Consider also reporting quality on the subset of complete outputs.
  4. [§5.5, Table 11] The sentence-level correlations are reported without p-values or CIs. Please add them or state explicitly that these values are descriptive only.
  5. [References] Several references have incomplete venue/proceedings information (e.g., [5] and [17]). Please normalize the bibliography.

Circularity Check

0 steps flagged

No circularity: all central claims are direct empirical comparisons against external baselines; self-citations are background only.

full rationale

This paper is an empirical evaluation rather than a derivation. Its central findings—dedicated MT systems remain strongest, few-shot prompting helps some models but hurts llama3.2:3b, embedding retrieval is best on average, and family-scope prompting causes structured-output failures—are direct readings of measured BLEU/chrF++/COMET and compliance tables on FLORES devtest. No fitted parameter is subsequently renamed as a prediction, and no claimed result is defined in terms of another result. The few-shot demonstrations are drawn from FLORES dev while evaluation uses devtest, avoiding leakage. The author's self-citations appear only in related work, where they motivate the study of lexical/domain context; the paper explicitly states that the benefit of additional knowledge 'must be measured against random examples and zero-shot prompts rather than assumed.' The comparison against OPUS-MT and NLLB-200 is externally anchored, and the sentence-level retrieval analysis even checks whether the retrieval similarity score predicts gains, finding near-zero correlation—so the paper does not assume that its retrieval criterion creates the observed benefit. The acknowledged limitation that a single inference stack, temperature 0, and an unpinned mistral:latest tag may affect generalization is a reproducibility/validity concern, not a circularity one. The empirical ordering may be configuration-dependent, but it is not derived from or equivalent to its own inputs.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central claims rest on benchmark representativeness, automatic-metric validity, a fixed inference stack, and post-selection validity in significance tests. No fitted constants or invented entities are used.

free parameters (2)
  • few-shot count k = 5
    All few-shot conditions use k=5 demonstrations from FLORES dev; no sensitivity analysis over k is reported.
  • decoding temperature = 0
    Generation runs with temperature 0 via Ollama; this is a hand-chosen decoding setting that removes sampling variance but may not reflect default user configurations.
axioms (4)
  • domain assumption FLORES devtest and its reference translations are a valid proxy for the translation tasks users give local LLMs.
    All evaluation is on FLORES devtest (§4.1); if this benchmark is not representative of practical LLM MT use, the comparative findings generalize poorly.
  • domain assumption Automatic metrics (sacreBLEU, chrF++, COMET) are sufficient for the paper's comparative claims.
    The paper reports only automatic metrics and acknowledges that human adequacy and fluency assessment is absent (§6).
  • domain assumption The three models run through Ollama at temperature 0 with the stated prompts represent local LLM translation behavior.
    Results depend on the inference stack, model tags, and prompt templates; 'mistral:latest' is unpinned (§4.2, §6).
  • domain assumption Selecting the best local-LLM condition per target and the chrF++-best MT baseline on the same test set does not invalidate paired bootstrap p-values.
    Table 10 reports p-values without multiple-comparison or post-selection adjustment; the closest result (Spanish, p=0.520) depends on this selection.

pith-pipeline@v1.3.0-alltime-deepseek · 16046 in / 9604 out tokens · 88004 ms · 2026-08-01T00:14:31.661241+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are increasingly used as general-purpose translation systems, but their behavior is usually evaluated under a single prompt shape: translate one source sentence into one target language. In practice, users may ask for one target language, for several related languages at once, or for translations conditioned on examples. This paper studies prompt scope and demonstration selection as experimental variables for local LLM machine translation. We evaluate English-to-Romance and English-to-Germanic translation on the full FLORES devtest split for nine official European Union languages. We compare three local instruction-tuned LLMs, llama3.2:3b, mistral:latest, and qwen2.5:14b, against dedicated MT baselines from OPUS-MT and NLLB-200. We test zero-shot prompting and k=5 few-shot prompting with random, lexical-similarity, and embedding-similarity demonstration selection. We also compare single-target prompts with JSON-formatted family-scope prompts that request all languages in a family at once. Results show that dedicated MT systems remain strongest overall, especially for Germanic languages. Few-shot prompting helps mistral:latest and qwen2.5:14b, but hurts llama3.2:3b; embedding retrieval is best on average for the stronger LLMs, but its advantage over random and lexical examples is modest. Family-scope prompting is feasible for stronger local LLMs but exposes structured-output failures in smaller models. These findings motivate evaluating LLM translation not only by language pair and metric, but also by prompt scope, retrieval strategy, and multi-target compliance.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 5 linked inside Pith

  1. [1]

    Mihael Arcan, Daniel Torregrosa, Sina Ahmadi, and John P. McCrae. Inferring translation candidates for multilingual dictionary generation with multi-way neural machine translation. InProceedings of the Translation Inference Across Dictionaries 2019 Shared Task, volume 2493 ofCEUR Workshop Proceedings, pages 13–23, 2019. 16

    2019

  2. [2]

    Translating terminological expressions in knowledge bases with neural machine translation.arXiv preprint arXiv:1709.02184, 2017

    Mihael Arcan, Daniel Torregrosa, and Paul Buitelaar. Translating terminological expressions in knowledge bases with neural machine translation.arXiv preprint arXiv:1709.02184, 2017

    Pith/arXiv arXiv 2017

  3. [3]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in Neural Information Processing Systems, 33:1877–1901, 2020

    1901

  4. [4]

    Bharathi Raja Chakravarthi, Mihael Arcan, and John P. McCrae. WordNet gloss translation for under-resourced languages using multilingual neural machine translation. InProceedings of the Second Workshop on Multilingualism at the Intersection of Knowledge Bases and Machine Translation, pages 1–7, Dublin, Ireland, 2019. European Association for Machine Translation

    2019

  5. [5]

    Enhancing translation quality by leveraging semantic diversity in multimodal machine translation

    Ali Hatami, Mihael Arcan, and Paul Buitelaar. Enhancing translation quality by leveraging semantic diversity in multimodal machine translation. InProceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 154–166, Chicago, USA, 2024. Association for Machine Translation in the Americas

    2024

  6. [6]

    How good are GPT models at machine translation? a comprehensive evaluation

    Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. How good are GPT models at machine translation? a comprehensive evaluation. InarXiv preprint arXiv:2302.09210, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, Andre F. T. Martins, and Alexandra Birch. Marian: Fast neural machine translation in C++. InProceedings of ACL 2018, System Demonstrations, pages 116–121. Association for Computational Li...

    2018

  8. [8]

    What makes good in-context examples for GPT-3? InProceedings of Deep Learning Inside Out, pages 100–114

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? InProceedings of Deep Learning Inside Out, pages 100–114. Association for Computational Linguistics, 2022

    2022

  9. [9]

    McCrae, Mihael Arcan, and Paul Buitelaar

    John P. McCrae, Mihael Arcan, and Paul Buitelaar. Linking knowledge graphs across languages with semantic similarity and machine translation. InProceedings of the First Workshop on Multi-Language Processing in a Globalising World, 2017

    2017

  10. [10]

    NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, et al. No language left behind: Scaling human-centered machine translation.arXiv preprint arXiv:2207.04672, 2022

    Pith/arXiv arXiv 2022

  11. [11]

    chrF: Character n-gram F-score for automatic MT evaluation

    Maja Popovic. chrF: Character n-gram F-score for automatic MT evaluation. InProceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal,

  12. [12]

    chrF++: Words helping character n-grams

    Maja Popovic. chrF++: Words helping character n-grams. InProceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark, 2017. Association for Computational Linguistics

    2017

  13. [13]

    A call for clarity in reporting BLEU scores

    Matt Post. A call for clarity in reporting BLEU scores. InProceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium, 2018. Association for Computational Linguistics. 17

    2018

  14. [14]

    Ricardo Rei, Jose G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C. Farinha, Taisiya Glushkova, Alon Lavie, and Luisa Coheur. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. InProceedings of the Seventh Conference on Machine Translation, pages 578–585. Association for Computational Linguistics, 2022

    2022

  15. [15]

    Farinha, and Alon Lavie

    Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2685–2702. Association for Computational Linguistics, 2020

    2020

  16. [16]

    The OPUS-MT project: Building open translation services for the world

    Jörg Tiedemann and Santhosh Thottingal. The OPUS-MT project: Building open translation services for the world. InProceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480, 2020

    2020

  17. [17]

    Leveraging rule-based machine translation knowledge for under-resourced neural machine translation models

    Daniel Torregrosa, Nivranshu Pasricha, Maraim Masoud, Bharathi Raja Chakravarthi, Juan Alonso, Noe Casas, and Mihael Arcan. Leveraging rule-based machine translation knowledge for under-resourced neural machine translation models. InProceedings of Machine Translation Summit XVII: Translator, Project and User Tracks, pages 125–133, Dublin, Ireland, 2019. E...

    2019

  18. [18]

    Daniel Torregrosa, Nivranshu Pasricha, Maraim Masoud, Bharathi Raja Chakravarthi, Juan Alonso, Noe Casas, and Mihael Arcan. Aspects of terminological and named entity knowledge within rule-based machine translation models for under-resourced neural machine translation scenarios.arXiv preprint arXiv:2009.13398, 2020

    Pith/arXiv arXiv 2009

  19. [19]

    Prompting large language model for machine translation: A case study.arXiv preprint arXiv:2301.07069, 2023

    Biao Zhang, Barry Haddow, and Alexandra Birch. Prompting large language model for machine translation: A case study.arXiv preprint arXiv:2301.07069, 2023. 18

    Pith/arXiv arXiv 2023

  20. [2015]

    Association for Computational Linguistics