REVIEW 3 major objections 3 minor
From SALAMANDRA to SALAMANDRATA: BSC Submission for WMT25 General Machine Translation Shared Task
T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Two-step training yields a 38-language translator from the SALAMANDRA base
desk verdict Abstract-only WMT25 system paper that promises open 7B/2B MT models for 38 European languages; the recipe is sensible but the central performance claim is entirely unevidenced in the abstract. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the two-stage training recipe: (1) continual pre-training on parallel data to realign the model toward translation, and (2) supervised fine-tuning on high-quality instructions to teach it to follow translation prompts. A third component, vocabulary adaptation, extends the tokenizer to cover the additional non-European languages before the second round of training. The decoding side adds quality-aware selection: Minimum Bayes Risk Decoding and tuned reranking with COMET/COMET-KIWI.
What would settle it
Run a head-to-head ablation on the released 7B model: train a control model with the same supervised fine-tuning data but without the parallel-data continual pre-training, and evaluate both on the same held-out translation benchmark across the 38 European languages. If the control matches or exceeds the full recipe on BLEU/COMET, the paper's central claim about the recipe's contribution collapses.
Extended reading notes
Core claim
The central claim is that a general-purpose multilingual LLM can be turned into a strong translation model by two successive training passes: continual pre-training on parallel sentence pairs, followed by supervised fine-tuning on high-quality translation instructions. Applied to the SALAMANDRA 7B model, after adapting its vocabulary to the non-European languages required by the WMT25 task, this recipe yields competitive translation quality across 38 European and additional non-European languages. The authors also introduce a second released variant, SALAMANDRATA-V2, and report that quality-aware decoding strategies—Minimum Bayes Risk Decoding and tuned reranking with COMET and COMET-KIWI—ar
Load-bearing premise
The load-bearing premise is that the two-stage recipe—parallel-data continual pre-training then instruction fine-tuning—improves translation for every target language without damaging the base model's existing abilities; the abstract asserts this but gives no comparative evidence.
Editorial extensions
If this is right
- If the recipe is effective, then a large general-purpose LLM can be specialized into a strong translator with continual pre-training on parallel data, rather than needing a dedicated translation model trained from scratch.
- The vocabulary-extension step shows how to add new languages to an existing tokenizer without full retraining.
- The two scales (2B and 7B) suggest the recipe is robust to model size, at least across that range.
- The public release enables direct evaluation and reuse of the models by other researchers and practitioners.
- The use of MBR and COMET/COMET-KIWI reranking at decoding time suggests that the same models can be improved further by better inference-time search.
Reading between the lines
- The recipe is not translation-specific: continual pre-training on task-parallel data followed by instruction fine-tuning is a general blueprint for specializing a base LLM, so the same two-stage approach could plausibly be applied to other structured-generation tasks (e.g., summarization, code translation).
- A controlled ablation that removes the parallel-data continual pre-training (keeping only the SFT step) would isolate the contribution of each stage; the paper's abstract does not report such an ablation, so the recipe's individual step contributions remain untested here.
- Because the abstract reports no human evaluation, the 'strong performance' claim rests on the shared task's automatic metrics; a human study would test whether the quality holds up for real-world use.
- The SALAMANDRATA-V2 model is mentioned but not described; comparing it to the base SALAMANDRATA would show how much the added training phase changes behavior.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript, available only as an abstract, announces the SALAMANDRATA family of multilingual translation models (2B and 7B) from the Barcelona Supercomputing Center for the WMT25 General Machine Translation shared task. The proposed recipe is two-stage: continual pre-training on parallel data followed by supervised fine-tuning on high-quality instructions, with a vocabulary adaptation step for additional non-European languages. The submission is based on the 7B variant, and decoding uses Minimum Bayes Risk decoding and COMET/COMET-KIWI-based tuned re-ranking. The abstract claims 'strong performance' across 38 European languages and additional non-European languages, and states that both 2B and 7B models, plus a newer SALAMANDRATA-V2, are publicly released on Hugging Face.
Significance. If the claimed performance is substantiated in the full text, this work would be a useful open-source contribution to multilingual machine translation, particularly for European languages and the WMT shared task. The practical strengths that are visible from the abstract include the public release of models, the use of two quality-aware decoding strategies, and a clearly stated two-stage training recipe. However, because the abstract contains no quantitative evaluation results, no baselines, and no language-level breakdown, the central empirical claim is currently unverified. The significance of the work cannot be properly assessed without the evaluation section; the potential contribution is real but conditional on the missing evidence.
major comments (3)
- [Abstract] The central claim 'strong performance' is asserted without any supporting evaluation numbers, such as BLEU or COMET scores, and without comparison to a baseline system. For a shared-task system description, official test-set results are the load-bearing evidence. Please add a results section with at least aggregate scores over the task languages and, if space permits, per-language or per-language-family results, along with the evaluation protocol (test sets, metrics, number of systems compared).
- [Abstract] The training recipe is described as 'carefully designed to optimize performance across all translation directions,' but no evidence is provided for this claim. Critical details such as the composition and size of the parallel data, the vocabulary adaptation procedure, the fine-tuning instruction data, and the hyperparameters are absent from the abstract. If these are present in the full text, please make them explicit; if they are not, include a description of the design and, ideally, an ablation or at least a discussion of why the design was chosen.
- [Abstract] The abstract does not address the risk of catastrophic forgetting during continual pre-training and fine-tuning, which is a common concern when adapting a multilingual model to new languages and tasks. Since the claim is that performance is strong 'across all translation directions,' the manuscript should report whether performance on previously supported languages or general capabilities was maintained. A simple evaluation before and after adaptation would strengthen the claim.
minor comments (3)
- [Abstract] The phrase 'Tuned Re-ranking using COMET and COMET-KIWI respectively' is grammatically unclear. The sentence lists two decoding strategies but the 'respectively' suggests a pairing that is not fully specified. Please rephrase, e.g., 'Minimum Bayes Risk Decoding and tuned re-ranking with COMET/COMET-KIWI.'
- [Abstract] The abstract introduces 'SALAMANDRATA-V2' without explaining its relationship to the 2B and 7B models. If this model is part of the submission, clarify its role; otherwise, it may be a distraction in an already dense abstract.
- [Abstract] Minor typographical/presentation issue: the Hugging Face reference is followed by '1' without a visible footnote marker in the abstract text. Please ensure footnote formatting is consistent.
Circularity Check
No significant circularity found
full rationale
This is an abstract-only system description for a WMT shared task submission. There are no equations, no derived quantities, and no formal derivation chain that could reduce to its own inputs. The central claim is an empirical one: that SALAMANDRATA 7B, trained with continual pre-training on parallel data followed by supervised fine-tuning, achieves strong translation performance. The training recipe is a method, not a fitted parameter renamed as a prediction; the claimed outcome is an evaluation result that is not reported in the abstract, so it is unsupported but not circular. The only citation is to the prior SALAMANDRA work (Gonzalez-Agirre et al., 2025), which is appropriately used as the base model lineage rather than as a load-bearing justification for the present results. Even if the cited authors overlap with the current submission, the abstract does not invoke any uniqueness theorem, ansatz, or derived equality from that citation. No step in the available text defines a quantity in terms of the claim it is supposed to support, and no 'prediction' is statistically forced by a fit to the same target data. The absence of evaluation evidence is a completeness concern, not a circularity concern. Therefore the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (2)
- domain assumption Continual pre-training on parallel data followed by supervised fine-tuning improves translation quality across all 38 languages.
- domain assumption The adapted vocabulary supports additional non-European languages without degrading performance on existing ones.
Cite this review
Pith. "Pith review of From SALAMANDRA to SALAMANDRATA: BSC Submission for WMT25 General Machine Translation Shared Task." pith.science (2026). https://pith.science/paper/ZLK6LO6H
@misc{pith2026250812774,
author = {Pith},
title = {Pith review of: From SALAMANDRA to SALAMANDRATA: BSC Submission for WMT25 General Machine Translation Shared Task},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZLK6LO6H}},
note = {Machine review of arXiv:2508.12774}
}
read the original abstract
In this paper, we present the SALAMANDRATA family of models, an improved iteration of SALAMANDRA LLMs (Gonzalez-Agirre et al., 2025) specifically trained to achieve strong performance in translation-related tasks for 38 European languages. SALAMANDRATA comes in two scales: 2B and 7B parameters. For both versions, we applied the same training recipe with a first step of continual pre-training on parallel data, and a second step of supervised fine-tuning on high-quality instructions. The BSC submission to the WMT25 General Machine Translation shared task is based on the 7B variant of SALAMANDRATA. We first adapted the model vocabulary to support the additional non-European languages included in the task. This was followed by a second phase of continual pre-training and supervised fine-tuning, carefully designed to optimize performance across all translation directions for this year's shared task. For decoding, we employed two quality-aware strategies: Minimum Bayes Risk Decoding and Tuned Re-ranking using COMET and COMET-KIWI respectively. We publicly release both the 2B and 7B versions of SALAMANDRATA, along with the newer SALAMANDRATA-V2 model, on Hugging Face1
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.