{"id":"90145fcf-d841-4724-af2c-81a63c14096c","arxiv_id":"2607.19226","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Reasoning traces improve legal MT mainly when enabled at inference, and training with reasoning keeps those traces compact enough to be cost-effective.","lead":"This paper flips the \"thinking\" switch in reinforcement-trained legal machine translation models, during both training and use, and measures quality against cost. It finds that reasoning at inference reliably helps, but only if the model was trained to reason—otherwise the thinking output balloons and costs more without better translations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cost-quality claim omits training-cost amortization; break-even volume not established.","rationale":"The reader's CONDITIONAL verdict is appropriate; I would not move it. The strongest part of the paper is the empirical quality claim: across all four model/training combinations, enabling explicit thinking at inference raises COMET with small standard errors, and the token-count reduction from thinking-aware training is large and plausibly robust. That part is well supported. The distinctive contribution is the cost-quality tradeoff, and there the analysis is incomplete: Fig. 2 and the §5 conclusion depend on treating training cost as sunk, but no break-even volume is given. The reader flagged the $0.79/hour cost assumption as fragile; that is related but not the same as the amortization gap. The empty <think>-tag operationalization is a legitimate secondary concern, but the paper frames the claim as being about the explicit reasoning trace, so I do not treat it as decisive. The proposed total-cost check would settle whether the central cost-quality conclusion survives at realistic deployment volumes.","tokens_in":6611,"tokens_out":12151,"duration_ms":138190,"concrete_test":"Recompute the Pareto analysis using total cost = training_cost + N × inference_cost for N = 1, 10, 100, 1000 test-set equivalents (1 test set = 18.1k sentence pairs), and report the smallest N at which [T:✓, I:✓] enters the frontier ahead of [T:✗, I:✓]. Also re-run with the hourly GPU rate varied over $0.20–$2.00/h. If matched-mode dominance persists beyond the intended deployment scale, the conclusion stands; otherwise it must be explicitly volume-conditioned.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central cost-quality conclusion (§4.2, Fig. 2, §5) is plotted against inference cost only, justified by the statement that training is a one-time fixed cost and therefore inference cost decides. Table 1 shows training costs dominate one-test-set inference costs: for Qwen3.5 9B, [T:✓, I:✓] costs $11.61 to train vs $0.28 to infer on the 18.1k test set, while [T:✗, I:✓] costs $4.54 to train vs $1.03 to infer. Thus the asserted advantage of 'always thinking' over 'inference-only thinking' appears only after roughly (11.61 − 4.54)/(1.03 − 0.28) ≈ 9.4 test-set equivalents (~170k sentence pairs); for the 4B model it is ≈11.5 test sets (~207k pairs). Below that volume, [T:✗, I:✓] is cheaper and only slightly lower in COMET, so the headline tradeoff is not scale-free. The paper provides no break-even analysis and no sensitivity analysis for the standardized $0.79/hour rate, so the cost-quality ranking is conditional on deployment volume and hardware assumptions, even though the quality result itself is consistent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether explicit reasoning traces ('thinking') in RLVR-trained LLMs for legal NMT should be enabled during training and/or inference. Using Qwen3.5 4B/9B with GRPO/QLoRA on SwiLTra-Bench, the authors permute thinking on/off in the two phases, evaluate with chrF, COMET, METEOR, and MetricX, and measure training/inference cost and output tokens. They find that inference-time thinking improves translation quality regardless of the training condition, that training without thinking produces much longer thinking traces when thinking is enabled at inference, and that the cost-quality Pareto frontier favors matched modes (never thinking or always thinking). The central quantitative claim is that training with thinking is economically preferable mainly because it keeps inference-time reasoning traces compact, despite higher training cost.","tokens_in":6919,"tokens_out":5713,"duration_ms":62214,"significance":"The factorial design cleanly separates the role of the reasoning trace at training vs. inference, which is valuable for deployment decisions in legal MT. Strengths include open code, four complementary metrics with small standard errors, an explicit cost model borrowed from Zhuang et al., and training-sample curves (Fig. 1) supporting the plateau claim. If the conclusions hold, the result that RLVR's benefit is largely realized at inference time and that RL training compacts reasoning traces is practically important. The main weakness is that the cost-quality tradeoff is presented against inference cost only, without amortizing the substantial training-cost differences.","major_comments":[{"comment":"The central cost-quality conclusion is based on inference cost only. The text argues that training is a one-time fixed cost and inference cost becomes the deciding factor, but no amortization or break-even analysis is provided. From Table 1, for Qwen3.5 9B the [T:✓, I:✓] configuration costs $11.61 to train + $0.28 to infer, while [T:✗, I:✓] costs $4.54 + $1.03. On a single SwiLTra-Bench test set the former is $6.32 more expensive; the claimed advantage appears only after ≈9.4 test-set equivalents (~170k sentence pairs), and ≈11.5 test sets (~207k pairs) for the 4B model. The Pareto frontier in Fig. 2 should include amortized training cost or a break-even volume, and should be accompanied by sensitivity analysis for the standardized $0.79/hour rate, since absolute cost differences are cents.","section":"§4.2, Fig. 2, Table 1, Eq. (1)"},{"comment":"The paper states 'we select the best checkpoints' but does not specify the selection criterion. Which metric and which validation/test set were used? If any test-set information influenced checkpoint selection, Table 1 and Fig. 2 overstate the achievable quality; if a validation COMET curve was used, this should be stated and the number of training samples for each selected checkpoint should be reported. This is load-bearing because the quality-cost ordering depends on the chosen checkpoint for each of the four training/inference configurations.","section":"§4.2, Table 1"},{"comment":"The 'without thinking' condition is implemented by prefilling an empty <think></think> tag, and the paper later states that this 'does not destroy the thinking ability that the base model exhibits.' The manipulation therefore isolates the visible reasoning trace, not the presence or absence of reasoning as a cognitive process. The abstract and conclusion should phrase the finding as 'including a reasoning trace in the output' rather than 'reasoning has a positive effect,' unless a control for implicit reasoning is provided. This does not invalidate the quality comparisons, but it narrows the theoretical claim.","section":"§3, §4.1"}],"minor_comments":[{"comment":"The sentence 'training without thinking ... does not destroy the thinking ability' is an interpretation; the evidence only shows that enabling thinking at inference still improves quality after such training. Please rephrase as an empirical observation about transfer, not an internal-ability claim.","section":"§4.1"},{"comment":"Report for each row the number of training samples used for the selected checkpoint. Currently the training costs vary (e.g., 4B [T:✗] costs $5.46 while 9B [T:✗] costs $4.54), which is difficult to interpret without knowing the training data volume and the number of samples per checkpoint.","section":"Table 1"},{"comment":"State the hardware used for timing measurements (GPU type, vLLM configuration, batch settings) and the commercial API pricing version used in Fig. 2, so that the cost comparison is reproducible.","section":"§2.3, §4.2"},{"comment":"Provide GRPO/QLoRA hyperparameters (learning rate, LoRA rank/alpha, group size is already given as 4, number of steps, etc.) and the exact number of total training samples considered; Figure 1 shows up to 2,500 but the stopping criterion is not stated.","section":"§3"},{"comment":"There is a formatting issue in the abstract ('bothtrainingandinference'); correct to 'both training and inference.'","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for DocEng and the factorial experiment is well designed. The main revision should focus on making the cost-quality claim conditional on deployment volume and on documenting checkpoint selection; once those are addressed, the result will be substantially stronger."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth a serious look. The core empirical finding holds up: enabling reasoning at inference improves translation quality regardless of whether the model was trained with reasoning, and training with reasoning keeps inference-time reasoning traces compact. The 2x2 factorial ablation is a genuine extension of the authors' prior work, and the token-compaction effect (e.g., Qwen3.5 9B output tokens dropping from 20.67M to 6.27M) is a non-obvious, useful observation. Quality results are consistent across four metrics with small standard errors, which is solid evidence. Code and data are available, and the experiments are reproducible in principle.\n\nThe main soft spot is the cost-quality framing. The Pareto analysis in Figure 2 plots inference cost only, justified by training being a one-time fixed cost. That justification is fine at large scale, but the paper does not quantify the scale. From Table 1, the always-thinking 9B model costs about $7 more to train than the inference-only-thinking model, but saves only about $0.75 per test-set inference. So you need roughly 170k translated sentence pairs before that training premium pays off. Below that volume, inference-only thinking is cheaper, and the COMET loss is only a hair (82.32 vs 82.50). The paper should report break-even volumes or explicitly frame the conclusion as \"for high-volume deployment.\" The standardized $0.79/hour GPU cost is also a single point estimate with no sensitivity analysis; the Pareto ordering could shift under different hardware or electricity assumptions.\n\nTwo smaller issues: the best-checkpoint selection criterion is not stated, and the \"no thinking\" operationalization via empty <think> tags is a reasonable proxy but does not rule out implicit reasoning. These are all addressable in revision.\n\nFor a short DocEng paper, this is a clean, honest empirical contribution. It is not ground-breaking, but it gives practitioners a concrete deployment rule for legal MT and similar RLVR tasks. I would send it to peer review, and I would expect acceptable after the authors add a break-even analysis, sensitivity on the hourly rate, and state the checkpoint selection protocol.","headline":"Solid, clean ablation showing inference-time reasoning helps and training-with-reasoning compacts traces; the cost-quality headline is conditional on deployment volume and the single hourly rate, and the paper should say so.","tokens_in":7329,"tokens_out":2397,"would_cite":true,"duration_ms":39058,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Enabling reasoning at inference improves translation quality, but matched training and inference modes dominate the cost-quality tradeoff for legal machine translation.","keywords":["neural machine translation","legal translation","reinforcement learning with verifiable rewards","reasoning traces","cost-quality tradeoff","inference cost","GRPO","SwiLTra-Bench"],"falsifier":"Recompute the cost-quality comparison with a different hourly GPU price (e.g., $0.30 or $2.00) and check whether the matched [T:✓, I:✓] and [T:✗, I:✗] configurations still Pareto-dominate; or probe the models' internal activations while generating with empty <think> tags to test whether reasoning is truly disabled.","tokens_in":6527,"feed_emoji":"🧠","tokens_out":4613,"duration_ms":51043,"temperature":0.7,"pith_summary":"This paper asks whether the reasoning traces that improve legal machine translation are worth their cost. By switching reasoning on and off independently during training and inference, the authors show that enabling reasoning at inference consistently raises translation quality, even for models never trained to reason. But the reasoning mode must be consistent: models trained without thinking produce much longer, less effective reasoning at inference, tripling cost for slightly worse quality. The economic conclusion is that the cost-quality tradeoff favors matched configurations—both thinking or neither—and the main benefit of training with reasoning is keeping inference traces compact (up to 70% fewer tokens).","feed_headline":"Thinking at inference boosts translation; training tames its cost","feed_subtitle":"Matched reasoning modes—both on or both off—dominate the cost-quality frontier for legal machine translation.","key_machinery":"The experiment uses GRPO training with a structural reward that requires the model to put reasoning inside <think> tags, and an empty pre-filled <think> tag to disable reasoning during training and inference. Quality is measured with chrF (the training reward), COMET, METEOR, and MetricX, while cost is estimated from a standardized hourly GPU cost ($0.79/hour) multiplied by execution time, following a depreciation-plus-power-plus-maintenance formula. The empty-tag switch cleanly separates the reasoning trace from the training paradigm, and checkpoints every 100 training samples reveal how quality and cost scale with data.","core_discovery":"The central claim is that the value of reasoning in RLVR-trained translation models lies at inference, not in training. Enabling thinking during inference improves translation quality across all configurations, but training without thinking produces verbose reasoning traces at inference (up to 20.67M output tokens vs 6.27M for the 9B model) that cost far more and translate slightly worse. Therefore, the cost-quality tradeoff favors models that preserve the thinking mode between training and inference: either always think or never think. The best result is the 9B model trained and inferred with thinking, reaching a COMET score of 82.50, while its matched no-thinking variant sits on the same P","pith_inferences":["The $0.79/hour GPU price is the pivot: the absolute cost gaps between configurations are cents, so a sensitivity analysis across realistic hardware and electricity prices could reorder the Pareto frontier.","The empty-tag manipulation assumes reasoning is fully disabled when the tag is empty; if models reason implicitly, the 'no thinking' condition still contains hidden reasoning, making the attribution less clean than reported.","The trace-compaction effect suggests a testable extension: check whether the same 70% token reduction holds for other language pairs and domains, or whether it is specific to legal texts and the GRPO reward.","The Pareto frontier is a snapshot under one pricing model; changes in API prices or local hardware costs could shift which configurations are competitive."],"forward_implications":["Deployments should match the reasoning mode used during training; mismatched configurations are Pareto-dominated in cost and quality.","Training with reasoning is best justified as a cost-containment measure rather than a pure quality booster, reducing inference tokens by up to 70%.","RLVR's gains for translation do not require new reasoning capabilities; the base model already produces useful reasoning when asked at inference.","For the legal translation benchmark used, around 1,000–1,500 training pairs reach the quality plateau, so curating high-quality samples beats adding more data.","Matched local configurations can sit on the same cost-quality frontier as commercial API models."],"fun_headline_variants":["Reasoning at inference, not training, lifts translation quality","Mismatched reasoning training inflates cost, hurts quality","Always think or never: matched reasoning wins translation tradeoff","Inference thinking boosts translation; training mismatch doubles cost","Cost-quality sweet spot: keep reasoning mode consistent from train to test"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The cost-quality conclusion rests on a fixed $0.79-per-GPU-hour price, and since the absolute cost differences between configurations are under a dollar, a different hardware or electricity price could change which configuration wins.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning at inference, not training, lifts translation quality","Mismatched reasoning training inflates cost, hurts quality","Always think or never: matched reasoning wins translation tradeoff","Inference thinking boosts translation; training mismatch doubles cost","Cost-quality sweet spot: keep reasoning mode consistent from train to test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000693,"raw_usage":{"total_tokens":2938,"prompt_tokens":677,"completion_tokens":2261,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":421,"completion_tokens_details":{"reasoning_tokens":2193}},"tokens_in":421,"tokens_out":2261,"duration_ms":15436,"temperature":1.0,"reasoning_tokens":2193,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T13:01:46.370127+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the cost-quality comparison with a different hourly GPU price (e.g., $0.30 or $2.00) and check whether the matched [T:✓, I:✓] and [T:✗, I:✗] configurations still Pareto-dominate; or probe the models' internal activations while generating with empty <think> tags to test whether reasoning is truly disabled.","supporting_citations":[],"review_version":1}