{"id":"cc9479b9-46da-4adb-a42b-4e6723ccb21e","arxiv_id":"2607.19181","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"On Swiss legal translation, reinforcement learning with a ChrF reward improves small open models more than supervised fine-tuning, but frontier reasoning models still score higher.","lead":"This paper compares three ways to make small AI translation models better at Swiss legal texts: prompting them to think first, fine-tuning on example translations, and training with rewards. It finds reward-based training works best, beating fine-tuning but still losing to the largest commercial reasoning models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RL claim hinges on automatic metrics, but the paper's own human eval shows ChrF (the RL reward) has near-zero expert correlation; a human preference test is needed to confirm real quality gains.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the validity of ChrF as a proxy for legal translation quality. This concern is central because the RL training objective is ChrF, and the evaluation uses ChrF alongside other metrics. The paper's Table 3 directly shows ChrF has essentially no correlation with a legal expert (r=0.05), and even COMET—the most human-aligned metric—is weak (r=0.24). If the metrics do not reflect expert judgment, the RL-better-than-SFT ranking may be an artifact of reference overfitting rather than a genuine improvement in legal translation quality. The human evaluation reported is insufficient to resolve this: 240 samples, one expert, no system-level human score comparison. Therefore, I agree with the reader that the verdict should be CONDITIONAL: the claim is plausible (RL consistently improves all metrics, and the direction is uniform) but requires additional validation through a human preference study before it can be accepted. No verdict change is needed from the reader's CONDITIONAL. I do not find a different concern that would be more load-bearing: the dataset-size inconsistency is minor, the missing NMT baselines are peripheral, and the SFT/RL compute asymmetry does not negate the empirical finding. The metric-validity concern is the crux, and it is best settled by a targeted human preference test.","tokens_in":12958,"tokens_out":5306,"duration_ms":65305,"concrete_test":"Run a double-blind pairwise preference evaluation: sample 200+ segments per language pair (or at least 300 total) from the test set, present RL and Simple SFT outputs side-by-side to 2–3 legal experts, and ask which translation better preserves legal meaning, terminology, and fluency. Then test whether the proportion of human preferences favoring RL is significantly above chance (e.g., exact binomial test). Also report human scores per system and correlate per-segment human preference with ChrF/COMET differences to see if either metric predicts expert preference. If human experts show no significant preference for RL, the central claim would be severely weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—RL with verifiable rewards surpasses SFT for legal MT—is supported only by automatic metrics (Table 2). But the reward used for RL is ChrF (§4.3), and the training data is filtered by ChrF (§3.1). The paper's own human evaluation (Table 3) shows ChrF correlates with a legal expert at only r=0.05 (p=0.54), and even the best metric, COMET, correlates at r=0.24. This opens a concrete risk: RL may be optimizing for reference-specific character n-gram overlap, not for human-valued legal translation quality. While RL also improves COMET, METEOR, and MetricX, all of these metrics show weak or even negative human correlation in this study. The human evaluation is too thin to rescue the claim: 240 segments total, a single expert, no per-system human scores, and no pairwise comparison between RL and SFT. If legal experts do not actually prefer RL outputs over SFT outputs, the main conclusion is unsupported regardless of metric improvements. This is not an accusation of metric gaming; it is a validity threat that the current evidence does not rule out.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether structured reasoning can improve legal machine translation for Swiss legal texts. The authors compare three strategies: prompting frontier reasoning models, supervised fine-tuning (Simple SFT and Reasoning SFT) of small language models (Qwen3.5 4B/9B, Gemma 3 12B), and GRPO-based reinforcement learning with rule-based rewards that include a ChrF term. They release a dataset of 40k automatically generated reasoning-augmented translation pairs and report that RL consistently outperforms both SFT variants across ChrF, COMET, METEOR, and MetricX, while noting diminishing returns with model scale. They also present a small human evaluation showing that COMET correlates best with a legal expert. The central claim is that RL with verifiable rewards surpasses SFT for legal translation and narrows the gap to frontier models.","tokens_in":13323,"tokens_out":3545,"duration_ms":44854,"significance":"If the central result holds, the paper offers a practical recipe for improving small open-weight models on a high-stakes domain and releases a useful dataset and codebase. The comparison across three model families, four automatic metrics, and multiple training paradigms is systematic and reproducible. The main significance is the demonstration that GRPO with a continuous lexical reward can improve LLM-based translation beyond SFT, although this is less surprising given that the reward is also used as an evaluation metric. The human-evaluation evidence is too thin to establish that the RL gains reflect expert-valued quality, so the significance depends on additional validation.","major_comments":[{"comment":"The RL reward is ChrF and the training data are filtered by ChrF > median, while Table 2 reports ChrF as a headline metric; this creates a reward-fitting risk. The paper's own Table 3 shows ChrF correlates with a legal expert at r=0.05 (p=0.54), and even COMET correlates only at r=0.24. Thus the automatic-metric improvements do not establish that RL outputs are better in human-valued legal translation quality. The human evaluation covers only 240 segments with a single expert and no per-system means or pairwise RL-vs-SFT comparison. I recommend a direct human preference test between RL and SFT outputs, with multiple experts, before the central claim can be accepted.","section":"§4.3, §3.1, Table 3"},{"comment":"The abstract claims RL 'surpasses the translation quality of supervised fine-tuning', but the comparison is limited to LoRA-based SFT of instruction-tuned LLMs; no conventional NMT baseline (e.g., NLLB, M2M-100, or a standard transformer) is included. Simple SFT often underperforms the base model (e.g., Qwen3.5 4B ChrF 50.03 vs 50.53; Gemma 12B ChrF 55.86 vs 56.73), suggesting the SFT configuration may not be representative. Without a standard NMT baseline, the practical significance of the RL gains relative to widely deployed NMT systems is unclear.","section":"§4.2, Table 2, Abstract"},{"comment":"The training dataset is constructed by taking the first 40,000 sentence pairs and retaining only those with ChrF > median (64.19), yielding 19,979 pairs. There is no control experiment with a random subset of the same size. Because the same metric is used as the RL reward, the filtering may bias the training distribution toward examples that are particularly ChrF-favorable, weakening the generalization claims of the released dataset and the RL recipe. A random-subset control would disentangle data-selection effects from method effects.","section":"§3.1"}],"minor_comments":[{"comment":"MetricX has a negative correlation with human judgments (r=-0.13), but the text only notes that COMET correlates best; the negative correlation is not discussed. This is relevant to the choice of metrics for the central comparison.","section":"Table 3"},{"comment":"The training set is heavily skewed (e.g., fr→it has 8,152 pairs while en→de, en→fr, en→it each have only 63–65), and the test set excludes Romansh after filtering. The paper does not report the language-pair distribution of the final test set, so the aggregate results in Table 2 may not represent balanced legal translation performance.","section":"Table 4 / Evaluation"},{"comment":"For Gemma 3 12B, the base model technically surpasses RL on it→en by a small margin in Figure 2, and several RL-vs-SFT gaps are small (e.g., COMET 83.08 vs 82.45). No pairwise significance tests are reported; the confidence intervals alone do not establish significance for all differences.","section":"Table 2 / Figure 2"},{"comment":"The table lists 'OpenAI 4o' while the text mentions 'o4-mini', 'o4', and 'o3'; the naming is inconsistent and could confuse readers about which OpenAI model is which.","section":"Table 1"},{"comment":"There is a typo, 'self-reaffine', presumably 'self-refinement'. Also, the claim that reasoning steps from DeepSeek-R1 'resemble the workflow of a human translator' is anecdotal and not supported by analysis.","section":"§6.1"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a straightforward empirical comparison, but the central claim is currently supported only by automatic metrics, one of which is also the training reward. The human evaluation is too limited to resolve the validity threat. The revision should add a human preference test and ideally a standard NMT baseline; with those, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a useful, honest empirical comparison of RL vs SFT for Swiss legal MT, with a new reasoning-augmented dataset, but its headline claim — RL beats SFT — rests on automatic metrics that barely track the one expert human judge, and the SFT baseline looks undertuned. The direction is consistent across model families, so the result is probably real as an optimization effect, but it may not mean what the authors claim about translation quality.\n\nWhat’s new: the CH-Law-Trans reasoning dataset (40k generated reasoning traces, filtered to ~20k), the GRPO with ChrF reward recipe for legal MT, and the finding that reasoning SFT degrades quality relative to simple SFT. That last point is a useful negative result. The paper reports code and models, and the authors are candid about limitations.\n\nSoft spots, in rough order:\n1. Reward/evaluation overlap. ChrF is both the RL reward and a headline metric. Training data was filtered by ChrF above median. The paper’s own human eval shows ChrF correlates r=0.05 with a legal expert. COMET, the metric that drives most of the narrative, correlates only 0.24. So the RL gains may be optimizing for n-gram overlap that experts don’t value. A pairwise human preference test between RL and SFT outputs (not just metric correlations on 240 segments) is needed.\n2. The SFT baseline looks weak. Simple SFT fails to beat the base model on COMET for Qwen 4B and 9B, and only ties for Gemma. That is unusual and suggests the SFT recipe is undertrained or misconfigured. If so, RL > SFT is partly a comparison to a poor SFT. The paper should compare against a stronger SFT (more epochs, proper hyperparams) and a conventional NMT baseline.\n3. Dataset size inconsistency: abstract says 40k pairs; filtered is 19,979; 90/10 split should give ~17,981 training, but Appendix Table 4 sums to 18,981. Off by 1000. Should be fixed before publication.\n4. Minor: single expert for human eval, 240 segments; no per-system scores.\n\nNone of this sinks the empirical contribution — the release and the recipe are valuable. But the central claim needs stronger evidence.\n\nWho this is for: people building legal MT with small open models, or researchers interested in RL for translation. It deserves peer review, but with expectation of major revision. I’d bring it to reading group as a cautionary example of metric circularity.","headline":"Solid empirical comparison with a useful dataset, but the RL-vs-SFT claim rests on automatic metrics that barely correlate with expert judgment and a possibly undertrained SFT baseline.","tokens_in":13788,"tokens_out":2815,"would_cite":true,"duration_ms":32260,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement learning with verifiable rewards improves legal machine translation beyond supervised fine-tuning, the paper reports.","keywords":["legal machine translation","reinforcement learning","GRPO","supervised fine-tuning","reasoning","ChrF","COMET","Swiss legal domain"],"falsifier":"Collect expert ratings on a larger sample of RL versus SFT outputs large enough to detect a difference; if expert-preferred translations do not favor the RL models—or if replacing the ChrF reward with a more expert-correlated metric like COMET erases the gap—the central claim fails.","tokens_in":12875,"feed_emoji":"⚖️","tokens_out":2505,"duration_ms":31199,"temperature":0.7,"pith_summary":"The paper claims that for legal translation, reinforcement learning using simple verifiable rewards improves quality beyond supervised fine-tuning, while explicit reasoning steps in SFT actually hurt. Across three small models, RL with GRPO consistently outperforms both Simple SFT and Reasoning SFT on Swiss legal translation, narrowing the gap to frontier reasoning models. The sympathetic reader should care because this suggests a cost-effective path to high-quality domain translation with open, deployable models.","feed_headline":"RL beats supervised fine-tuning for legal translation","feed_subtitle":"Small open models trained with verifiable rewards approach frontier quality on Swiss legal texts.","key_machinery":"The central mechanism is a GRPO training recipe with two rule-based rewards: a format reward that enforces reasoning inside <think> tags before a final translation inside <translation> tags, and a continuous translation-quality reward equal to the ChrF character n-gram F-score against the reference. This connects the verifiable-reward approach from math reasoning to neural machine translation.","core_discovery":"The authors report that, on the SwissLegalTranslations test set, GRPO-trained models using a format reward for reasoning tags plus a ChrF-based translation-quality reward surpass both Simple SFT and Reasoning SFT on all four evaluation metrics and across all three model sizes. The best RL model, Gemma 3 12B, reaches COMET 83.08, within about two points of frontier models, while Reasoning SFT consistently degrades quality relative to Simple SFT.","pith_inferences":["Because the paper's own human evaluation finds ChrF correlates only r=0.05 with a legal expert, the RL advantage may be an artifact of optimizing a weak proxy; an expert-based evaluation could rank methods differently.","The failure of Reasoning SFT suggests the reasoning traces themselves are not the active ingredient; the reward signal is. A testable extension is RL with only a translation-quality reward and no explicit thinking format.","The released reasoning-augmented dataset could be repurposed for other legal NLP tasks, such as summarization or legal error detection, beyond translation.","The same GRPO + ChrF recipe may transfer to other high-stakes, terminology-dense domains like medical or patent translation, where a reference-based lexical metric is available."],"forward_implications":["RL-trained small models come within roughly two COMET points of frontier reasoning models on Swiss legal translation, using far fewer parameters.","Supervised fine-tuning on automatically generated reasoning steps not only fails to help; it systematically hurts translation quality.","RL gains are largest for the smallest model and shrink with scale, suggesting the main beneficiaries are compact, deployable models.","The released 40k reasoning-augmented legal translation dataset provides a resource for future research on reasoning in domain-specific NMT.","The GRPO + ChrF reward recipe offers a straightforward blueprint for applying RL to other terminology-dense translation domains."],"fun_headline_variants":["Verifiable-reward RL outshines SFT for legal translation","Small models, RL, and near-frontier Swiss legal translation","RL beats SFT on all metrics for legal MT","Legal translation: RL-tuned small models approach frontier"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire RL advantage rests on ChrF against a single test-set reference being a trustworthy proxy for legal translation quality; the paper's own human evaluation shows ChrF correlates only 0.05 with a legal expert, the weakest of the four metrics tested.","fun_headline_variants_meta":{"raw":{"variants":["Verifiable-reward RL outshines SFT for legal translation","Small models, RL, and near-frontier Swiss legal translation","RL beats SFT on all metrics for legal MT","Legal translation: RL-tuned small models approach frontier"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000581,"raw_usage":{"total_tokens":2565,"prompt_tokens":728,"completion_tokens":1837,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":1781}},"tokens_in":472,"tokens_out":1837,"duration_ms":14095,"temperature":1.0,"reasoning_tokens":1781,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T13:10:06.423417+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect expert ratings on a larger sample of RL versus SFT outputs large enough to detect a difference; if expert-preferred translations do not favor the RL models—or if replacing the ChrF reward with a more expert-correlated metric like COMET erases the gap—the central claim fails.","supporting_citations":[],"review_version":1}