{"id":"17bc10ab-e1aa-4816-b9d0-1913f0fa1d1e","arxiv_id":"2506.05017","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Weighting the EOS token in the loss during fine-tuning reduces too-long summaries on CNN/DailyMail and fixed-length XL-sum, but not on dynamic-length XL-sum.","lead":"This paper introduces a simple training trick: give the end-of-sentence token extra weight in the loss function, so a summarization model learns to stop at the desired length. The authors show this reduces overly long summaries across T5 and Llama models with little quality loss in most settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The XL-sum dynamic-length result directly contradicts the proposed causal mechanism and is explained only post hoc, so the method's generality and the 'always controls length' claim are not established.","rationale":"The paper presents a plausible empirical observation, and the fixed-length improvements are consistent across many settings, including the truncation-baseline analysis. The reader's weakest-assumption analysis correctly identifies the causal claim as the most fragile part of the argument. The XL-sum dynamic result is not merely a missing improvement; it is a case where the hypothesized mechanism should apply directly, and it fails. The authors' distribution-based explanation is reasonable but untested, and no released artifacts or formal verification substitute for that test. I do not think this warrants rejection: the abstract's qualified 'often' and the limitations section partially mitigate the overreach, and the fixed-length evidence is real. However, the Section 5 'always' statement goes beyond what is supported, and the dynamic-length contribution is not established as a general length-control mechanism. A focused experiment on XL-sum with a W sweep and a de-skewed training distribution would either rescue the mechanism or force the authors to narrow their claims, so the conditional verdict remains appropriate.","tokens_in":10250,"tokens_out":9760,"duration_ms":119853,"concrete_test":"Run the XL-sum dynamic-length experiment under two conditions: (i) sweep W over {2,5,10,50,100} instead of fixing W=10, and (ii) re-sample the XL-sum training set so that the summary-length distribution is de-skewed, for example uniform over the 25-character bins or truncated as in the fixed-length setup. If no W reduces % too-long below W=1 in either condition, the method's benefit depends on an unexplained distributional property and the causal hypothesis fails; if some W succeeds, the reported failure is a W-tuning artifact and the paper should report per-dataset W selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central hypothesis (Section 3) is that boosting the EOS loss component is sufficient to make the model follow the training length distribution. The XL-sum dynamic-length experiment (Table 6) is a direct counterexample to this hypothesis: the EOS position is the only training signal encoding where to stop, yet W=10 does not reduce the percentage of too-long summaries relative to W=1 (T5 Greedy 10.4% vs 10.4%; T5 Beam0 7.2% vs 8.0%; Llama-2 Greedy 7.2% vs 9.2%). The paper's explanation for this failure, that the XL-sum length distribution is heavily right-skewed and bimodal, is post hoc and is not implied by the proposed mechanism; no experiment manipulates that distribution to test the explanation. Since W=10 improves length control on the analogous CNN/DailyMail dynamic task (Table 5), the method's effect is dataset-dependent in a way the paper has not modeled. Consequently, the Section 5 sentence that the method 'always controls length better than the baseline, across architectures and decoding strategies' is unsupported beyond the fixed-length setup, and the causal story that gives the method its name remains unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a simple training-time method for controlling summarization length: during fine-tuning, the cross-entropy loss on the EOS token is upweighted by a hyperparameter W, together with a rescaling factor that preserves the expected loss norm. The method is tested on two datasets (CNN/DailyMail and XL-sum), two base models (T5-base and Llama-2 7B with qLoRA), and two settings: fixed-length (target a maximum number of characters) and dynamic-length (a prompt specifies the length limit). The authors report ROUGE-2 and BERTScore for quality and the percentage of too-long summaries as the primary length-control metric, across greedy and beam search decoding with length penalties. They report consistent improvements over the baseline for all fixed-length settings and for the CNN/DailyMail dynamic setting, but not for the XL-sum dynamic setting. They also compare against GPT-3.5-turbo and GPT-4o, which are found to overgenerate. The central claim is that upweighting the EOS loss improves adherence to length constraints without substantially harming quality.","tokens_in":10517,"tokens_out":6669,"duration_ms":77295,"significance":"The method is appealingly simple, architecture-agnostic, and complementary to decoding-time length penalties. If the effect is robust, it offers a practical tool for integrating length control into fine-tuning pipelines. The paper includes a useful ablation over W, covers multiple decoding strategies, and tests both encoder-decoder and decoder-only LLMs. However, the empirical support is weakened by a dataset-dependent failure (XL-sum dynamic), an implausible BERTScore scale, and the absence of statistical uncertainty. The core idea is worth considering, but the current evidence does not fully establish the broad claims made in the abstract and conclusions.","major_comments":[{"comment":"The XL-sum dynamic-length results directly contradict the central hypothesis of Section 3 that upweighting the EOS loss is sufficient to make the model follow the training length distribution. In Table 6, W=10 does not reduce the percentage of too-long summaries relative to W=1 for most rows (e.g., T5-base Greedy 10.4% vs 11.4%; Llama-2 Greedy 7.2% vs 9.2%; Llama-2 Beam0 3.4% vs 3.8%). The explanation given in Section 5, based on the right-skewed and bimodal training length distribution, is post hoc; no experiment manipulates the training distribution to test this explanation. Since the dynamic variant is part of the proposed methodology, the paper should either restrict its length-control claims to the fixed-length setting or provide a direct experimental test of the distribution-based explanation.","section":"Section 5, Table 6"},{"comment":"The reported BERTScore values are implausibly low for standard BERTScore: e.g., 26.1 for T5-base in Table 1, 28.1 for gpt-3.5-turbo in Table 3, and 41.8 for Llama-2 in Table 10. Typical BERTScore F1 values for summarization on CNN/DailyMail and XL-sum with DeBERTa-xlarge-mnli are above 0.85, and even a 100-fold rescaling would not produce values near 26-44. The paper does not state whether F1, precision, or recall is reported, nor any normalization. This strongly suggests an evaluation artifact (e.g., wrong layer, wrong tokenizer, or misaligned references). Since the paper uses BERTScore to support the claim that length control often does not harm quality, the quality comparisons must be recomputed or the metric must be clearly redefined.","section":"Section 4.4, Tables 1-10"},{"comment":"The hyperparameter W is selected after inspecting test-set results on the same CNN/DailyMail fixed-length setup used for the main comparisons. The text states 'This is also why we fixed W=10 for all subsequent experiments,' with no evidence of validation-based selection. This constitutes post-hoc selection on the test set and can inflate the apparent improvement in the percentage of too-long summaries. The authors should select W using a held-out validation split and only then report results on the test set.","section":"Section 5, Table 1"},{"comment":"No confidence intervals or significance tests are reported for any metric. Many comparisons are based on 500 test samples, where small percentage differences correspond to only a few summaries (e.g., XL-sum fixed-length Llama-2 Greedy: 1.0% vs 0.8%; XL-sum dynamic T5 Greedy: 10.4% vs 11.4%). The Conclusion states that the method 'significantly improves' adherence without statistical support. The authors should provide error bars, confidence intervals, or significance tests, particularly for the primary length metric.","section":"Sections 4.4, 5, 6"}],"minor_comments":[{"comment":"The rounding of K to multiples of 50 (CNN/DailyMail) or 25 (XL-sum) is described, but it is not stated whether the percentage of too-long summaries is computed against the rounded K or the original K. For a summary whose length is between K and the rounded K, this affects the metric, so the convention should be made explicit.","section":"Section 4.2"},{"comment":"There are typographical errors in Table 8: '18.10' should likely be '18.1' and '32.80' should be '32.8'. In addition, decimal formatting is inconsistent across tables (e.g., '9.80' vs '9.8').","section":"Table 8"},{"comment":"The learning rate schedule differs between models (cosine for Llama-2, linear for T5-base), and the OpenAI baselines use default generation parameters with a single prompt template. These choices make the cross-model comparisons (especially with GPT models) not fully controlled; the comparisons should be interpreted with this caveat in mind.","section":"Section 4.3"},{"comment":"The statement that gpt-3.5-turbo and gpt-4o 'failed to adhere' to length constraints is based on a single prompt and default decoding parameters. A more robust comparison would vary the prompt wording or include decoding-time length penalties for the OpenAI models.","section":"Section 5"},{"comment":"Reference [8] and Reference [9] are the same paper (Hermann et al., 'Teaching machines to read and comprehend') listed twice with different titles and venues; one of the two entries should be removed or consolidated.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The BERTScore values are the most immediate red flag; if they are not the result of a simple scaling/unit error, the quality comparisons are unreliable. The W-selection issue is fixable and should be addressed in revision. The XL-sum dynamic-length failure is load-bearing: the paper presents the method as a general length-control mechanism, and the only dataset-dependent failure must be either explained mechanistically or used to narrow the paper's claims. I would recommend a major revision rather than rejection because the fixed-length results are consistent and the method is simple enough to be worth evaluating after these corrections."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: the paper describes a genuinely simple and useful trick—weight the EOS token more heavily in the cross-entropy loss during fine-tuning—and shows it reduces the fraction of too-long summaries on two model families and several decoding settings. That part is real and worth knowing. The paper is less careful, though, when it generalizes: its own XL-sum dynamic-length results (Table 6) show no length-control benefit at W=10, sometimes slightly worse than baseline. The sentence in Section 5 that the method 'always controls length better' is not supported. The stress-test note is right about that.\n\nWhat the paper does well: the method is truly light-touch (no new architecture, no RL), and the experiments cover a reasonable spread: T5-base and Llama-2, greedy and beam search with length penalties, and a dynamic-length variant that prepends character limits. The authors also include a truncation baseline and show their method cuts off sentences less often, which is a thoughtful check. They are honest in the Limitations section about the XL-sum difficulty, even if that honesty doesn't square with the earlier 'always' claim.\n\nThe soft spots, in rough order of seriousness. First, the overclaim above; the mechanism story (EOS weighting teaches length distribution) is plausible but not proven, and the XL-sum dynamic case is a direct counterexample. Second, the BERTScore numbers are implausibly low—mid-20s to mid-40s—which suggests either a different computation or a bug. That casts doubt on the quality comparisons even where length control works. Third, W=10 was selected after seeing the test results, with no significance tests or confidence intervals. Fourth, no code or data are released, which for a method this simple is a fixable omission.\n\nNone of this kills the core finding. For fixed-length summarization, the effect is consistent and the quality costs look mild. The paper is a decent empirical contribution for practitioners who need to hit character budgets in UIs. It deserves a serious referee, but the authors should be asked to temper the claim, run proper significance tests (or report variance), justify or pre-register W selection, and either fix or explain the BERTScore values.\n\nMe, I'd rather see this revised than rejected—the core idea is too cheap and useful to sit in a drawer.","headline":"A genuinely simple EOS-weighting trick that works in many settings, undermined by an overbroad 'always' claim and a suspicious quality metric.","tokens_in":10988,"tokens_out":2684,"would_cite":true,"duration_ms":29772,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Raising the weight of the EOS token in the training objective gives fine-tuned summarization models better adherence to length limits, usually without a quality drop.","keywords":["text summarization","length control","EOS token weighting","cross-entropy loss","fine-tuning","ROUGE","BERTScore","decoding strategies"],"falsifier":"Fine-tune the same model on the same data with $W=1$ and $W=10$, then measure the share of summaries exceeding the length cap on a test set whose reference summaries sit just below the cap; if the $W=10$ model does not reduce that share, the central mechanism is falsified. The XL-sum dynamic-length condition already approximates this test, since EOS weighting there fails to improve length control.","tokens_in":10103,"feed_emoji":"📏","tokens_out":8200,"duration_ms":79225,"temperature":0.7,"pith_summary":"The paper tries to establish that a one-line change to the training objective — multiplying the loss contributed by the end-of-sentence (EOS) token by a weight $W>1$ — gives a model a direct signal about when to stop, and that this is enough to make generated summaries respect a length cap more often. This matters because existing length-control methods either alter the architecture, require training from scratch, or only act at decoding time, while this approach works by fine-tuning ordinary pretrained models and combines with decoding-time tricks. Across fine-tuned T5-base and Llama-2 7B models, greedy decoding and beam search with different length penalties, the weighted-loss model produced fewer over-length summaries in the fixed-length setting, often with roughly unchanged ROUGE-2 and BERTScore. In a dynamic-length setting where the target length is given in the prompt, the same weighting improved length adherence on CNN/Daily Mail but not on XL-sum, which the paper attributes to XL-sum's heavily right-skewed summary-length distribution. If the claim is right, length control becomes a hyperparameter of fine-tuning rather than a separate mechanism.","feed_headline":"Raising EOS weight cuts too-long summaries, keeps quality","feed_subtitle":"Fine-tuning with a heavier EOS loss makes generated summaries respect length caps without extra decoding tricks.","key_machinery":"The central object is the modified cross-entropy loss, $L_2 = -\\frac{R}{N}\\sum_{n=1}^{N} w_{y_n} \\log p(y_n)$, where $w_{y_n} = W$ if the ground-truth token is EOS and $1$ otherwise, and $R = N/(N+W-1)$ rescales the loss so the expected gradient norm stays put. This reweighted EOS term is the only loss component that directly teaches the model when to stop, so counting it $W$ times makes the model treat a missed EOS as $W$ times worse than any other token error. With $W$ growing, the predicted length distribution shifts left and the percentage of too-long summaries falls, with diminishing returns between $W=10$ and $W=100$ in the paper's ablation.","core_discovery":"The central claim is that the cross-entropy loss component computed on the EOS token is the only training signal that directly teaches a summarization model where to stop, and that this signal is normally diluted by averaging over dozens or hundreds of other token losses. The paper's method reweights that single component by a factor $W$ and rescales the loss by $R = N/(N+W-1)$ so that the update norm is unchanged in expectation, turning the loss into one computed over $N-1$ ordinary tokens plus the EOS token counted $W$ times. With $W=10$ the fine-tuned models consistently produced a lower percentage of too-long summaries than the $W=1$ baseline across both architectures and all tested decoding strategies in the fixed-length experiments, while summary quality stayed about the same for T5-base and showed a trade-off for Llama-2 7B on some settings. The paper also reports that the learned stopping behavior is not a trivial truncation: models trained with $W=10$ end summaries with punctuation about as often as the baseline and much more often than a hard 250-character truncation baseline. On the dynamic-length variant the method reliably reduced too-long output on CNN/Daily Mail but failed to improve length control on XL-sum, a failure the paper connects to the shape of the training length distribution.","pith_inferences":["A natural testable extension is to make $W$ depend on the requested length $K$ in the dynamic-length setting, e.g. a larger weight for shorter targets, which could counteract the pull of a right-skewed distribution like XL-sum's.","Because the method only penalizes false negatives on EOS, it enforces an upper bound rather than a target length; a symmetric penalty or a length-dependent weight could turn it into exact-length control.","The mechanism may transfer beyond summarization to any generation task with a hard output limit, such as code generation, translation, or constrained dialogue, wherever EOS is a single token.","Since the effect appears even under greedy decoding, the method changes the model's internal stopping policy rather than only the beam's ranking; inspecting EOS probability trajectories under $W=1$ and $W=10$ could reveal whether the model learns a length prior or a confidence threshold."],"forward_implications":["Any pretrained sequence-to-sequence model that can be fine-tuned can adopt length control by editing one term in the loss, with no architectural changes.","The length-control effect persists when beam search with length penalties is used, so EOS weighting stacks with inference-time methods rather than replacing them.","Models trained this way do not resort to hard truncation: summaries end in punctuation nearly as often as baseline summaries, so the controlled length is not achieved by cutting sentences off.","The method's effectiveness depends on the training length distribution; on datasets whose summaries are concentrated well below the cap, the too-long rate may not improve.","Higher $W$ values monotonically shorten output but with diminishing returns, so the weight can be tuned as a length-versus-quality dial."],"supporting_citations":[{"why":"supplies the CNN/Daily Mail corpus from which the fixed-length and dynamic-length summarization training sets are drawn.","marker":"[8, 24]"},{"why":"supplies XL-sum, the multilingual dataset whose English subset is used and whose right-skewed length distribution explains one failure case.","marker":"[6]"},{"why":"provides T5-base, one of the two pretrained architectures that are fine-tuned with and without EOS weighting.","marker":"[22]"},{"why":"provides Llama-2 7B, the large language model fine-tuned with low-rank adapters in the experiments.","marker":"[26]"},{"why":"enables efficient fine-tuning of Llama-2 7B via quantized low-rank adapters, so the method is tested on a modern GPT-style LLM.","marker":"[2]"},{"why":"introduces the decoding-time baseline that suppresses EOS to extend generation, the contrast for the paper's learning-based alternative.","marker":"[23]"},{"why":"provides length penalty in beam search, the inference-time technique shown to be orthogonal to EOS weighting.","marker":"[20]"},{"why":"defines ROUGE, the n-gram overlap metric used to measure summary quality.","marker":"[13]"},{"why":"defines BERTScore, the semantic similarity metric used alongside ROUGE to measure quality.","marker":"[29]"}],"fun_headline_variants":["EOS weight triage: cuts long summaries, keeps quality","Heavier EOS loss steers summary length, not quality","Reweighting EOS token loss tames overlong summaries","Simple EOS reweighting controls length, preserves quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that boosting the EOS token's share of the loss is enough, on its own, to make the fine-tuned model follow the training length distribution more closely, and the paper advances this as an empirical hypothesis rather than a derivation.","fun_headline_variants_meta":{"raw":{"variants":["EOS weight triage: cuts long summaries, keeps quality","Heavier EOS loss steers summary length, not quality","Reweighting EOS token loss tames overlong summaries","Simple EOS reweighting controls length, preserves quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000144,"raw_usage":{"total_tokens":1158,"prompt_tokens":913,"completion_tokens":245,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":175}},"tokens_in":529,"tokens_out":245,"duration_ms":3415,"temperature":1.0,"reasoning_tokens":175,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:28:09.029872+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune the same model on the same data with $W=1$ and $W=10$, then measure the share of summaries exceeding the length cap on a test set whose reference summaries sit just below the cap; if the $W=10$ model does not reduce that share, the central mechanism is falsified. The XL-sum dynamic-length condition already approximates this test, since EOS weighting there fails to improve length control.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines ROUGE, the n-gram overlap metric used to measure summary quality."},{"cited_title":"Raffel, N","cited_arxiv_id":null,"evidence_quote":"provides T5-base, one of the two pretrained architectures that are fine-tuned with and without EOS weighting."}],"review_version":1}