{"id":"b43e1c0e-161f-4ce2-94cd-07be2e515086","arxiv_id":"2505.18298","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An adaptive length-penalty reward for RL-trained LLMs reduces reasoning length by over 50% with small accuracy loss by automatically tightening and relaxing the penalty based on the model's current accuracy.","lead":"Researchers train a 1.5-billion-parameter math model to give shorter answers by making the reward for short responses adapt over time: compress hard when answers stay correct, ease off when accuracy drops. The method cuts reasoning length by more than half while losing a few accuracy points, which could lower the cost and latency of large reasoning models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reference accuracy accref is estimated from a single 64-prompt batch; with standard error ~0.06 and Section 5.3 showing ±0.1 miss collapses the method, the claimed >50% length reduction may not transfer across runs or new datasets.","rationale":"We agree with the reader that the weakest assumption is the reliability of acc_ref and the noisy per-step accuracy. The paper itself provides the strongest evidence: Section 5.3 demonstrates that ±0.1 errors in acc_ref cause the method to either stop reducing length or collapse. Since acc_ref is computed from one batch of 64 prompts, such errors are within the natural sampling noise of the estimate, so the failure modes are not edge cases but plausible outcomes of the proposed procedure. This is load-bearing because the central claim is that A-DLP consistently reduces length while maintaining accuracy; if the anchor is off, consistency breaks. We also note the absence of error bars or seed reporting, which makes it impossible to know whether the favorable case is typical. The S-DLP comparison issue is real but secondary: even if S-DLP is a single trajectory, the adaptive method does avoid collapse in the shown runs; the acc_ref sensitivity is the more direct threat to the central claim. These concerns do not falsify the method; they mean the paper's evidence is not yet sufficient, so the reader's CONDITIONAL verdict remains appropriate.","tokens_in":9976,"tokens_out":5251,"duration_ms":44831,"concrete_test":"Run the main experiment 10 times with identical hyperparameters but each time estimate acc_ref from a different randomly drawn 64-prompt batch (or bootstrap batches from the training set), and record final checkpoint accuracy and token length. If all 10 runs retain >50% length reduction and <0.04 accuracy loss vs the base model, the concern is refuted; if any run's acc_ref estimate deviates by ~0.1 from 0.62 and the final metrics move toward the Section 5.3 failure modes (plateau at long length or collapse), the headline claim is not robust to the paper's own acc_ref-setting procedure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2's update λ_{t+1}=max(0, λ_t+η(acc_t−acc_ref)) requires acc_ref to be a reliable anchor. In the main experiments (Section 4), acc_ref=0.62 is estimated from the accuracy of the first GRPO batch on the base model, i.e., from 64 prompts. For a Bernoulli accuracy near 0.6, the standard error of that estimate is about sqrt(0.6·0.4/64)≈0.06, so the true value can easily be 0.1 away. Section 5.3 (Effect of accref) shows that with accref=0.7 the penalty decays to zero within about 10 steps and length reduction plateaus, while with accref=0.5 the penalty overgrows and the model collapses into overly short, incorrect outputs. Thus the central claim—consistent >50% length reduction with <0.04 accuracy loss—holds only when the one-batch accref estimate happens to land within roughly ±0.05 of the true accuracy. No procedure is given for obtaining such an estimate without extra evaluation runs, which the paper explicitly says it wants to avoid, and no seed/error-bar analysis is reported to show this favorable setting is typical. Without a more robust way to set accref, the headline result is a conditional outcome rather than a general property of A-DLP.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Adaptive Direct Length Penalty (A-DLP), a reward-shaping method for RL fine-tuning of reasoning LLMs. The reward is R(x,y)=I{y=y*}-lambda_t*len(y), where the penalty coefficient is updated as lambda_{t+1}=max(0, lambda_t + eta*(acc_t - acc_ref)), with acc_ref estimated from the base model's first training batch. The paper compares A-DLP against a static length penalty (S-DLP), L1-Exact, and L1-Max on DeepScaleR-1.5B using GRPO, evaluating on AIME2025, MATH, AMC, Olympiad-Bench, and Minerva. The central claims are that A-DLP reduces token length by more than 50% while sacrificing less than 0.04 accuracy, that it avoids the collapse observed under S-DLP, and that it does so without manual tuning of the penalty coefficient. Section 5.3 reports sensitivity experiments showing that the method degrades when acc_ref is 0.1 above or below the value used in the main experiments, and that small eta can lead to collapse while large eta leads to oscillatory lambda_t.","tokens_in":10190,"tokens_out":8094,"duration_ms":65775,"significance":"If the empirical claims hold, A-DLP is an attractive drop-in intervention: it requires no architectural change, integrates into existing GRPO pipelines, and its core update rule is simple and interpretable. The paper also provides a useful negative result by documenting that the static S-DLP baseline collapses after prolonged training, and the sensitivity analysis in Section 5.3 is more candid than is typical. However, the general significance is currently conditional: the headline result rests on a single model scale, a single training run, figures without numeric tables or error bars, and an acc_ref anchor whose noise is comparable to the perturbation that the paper shows can make the method fail. The evaluation is genuinely external to the method's own parameters, so there is no circularity, but the robustness evidence is not yet sufficient to support the broad 'consistently and dramatically' claim in the abstract and conclusion.","major_comments":[{"comment":"The reference accuracy acc_ref is the load-bearing hyperparameter of Eq. (3), but the paper estimates it from a single GRPO batch of 64 prompts (Section 4). For a Bernoulli accuracy near 0.62, the standard error of that estimate is about 0.06, and Section 5.3 shows that setting acc_ref just 0.1 too high (0.7) drives lambda_t to zero within about 10 steps with length reduction plateauing, while setting it 0.1 too low (0.5) causes the penalty to overgrow and the model to collapse. The tested miss of ±0.1 is only about 1.6 standard errors away from the point estimate, yet one direction stalls and the other collapses; the paper gives no bootstrap, multi-batch averaging, or other robust estimation procedure. This makes the headline >50% length reduction a conditional outcome rather than a robust property of A-DLP.","section":"§3.2 and §5.3"},{"comment":"The central quantitative claims—more than 50% token-length reduction with less than 0.04 accuracy degradation, and reduction rates exceeding 55%—are stated only in prose and supported only by plotted curves. No numeric table reports per-dataset accuracy and length values, no error bars or confidence intervals are given, and no multiple-seed results are reported. Because the adaptive update in Eq. (3) is driven by a per-step accuracy estimate that Section 5.3 itself describes as noisy, the absence of any variance reporting is particularly consequential; the reader cannot tell whether the single displayed run is typical or favorable. The main results should be reported as tables with means and standard deviations over at least three seeds.","section":"§5.1 and Figures 1–3"},{"comment":"The comparison to S-DLP is based on a single training trajectory under one static penalty value, plotted as a fitted curve over checkpoints, rather than a sweep over lambda. The paper asserts monotonic length reduction to justify reading this trajectory as the full accuracy-length trade-off curve, but Figure 2 shows S-DLP undergoing a sharp collapse around step 100; a curve that includes a collapsed policy does not represent the attainable frontier of static penalties, because a smaller static lambda might avoid the collapse and yield better trade-offs. The value of lambda used for S-DLP is not explicitly reported, so the baseline comparison is underspecified; the claim that A-DLP lies above and to the left of S-DLP should be established against multiple static lambda values.","section":"§4, §5.1, and Figure 1"},{"comment":"All experiments use a single backbone, DeepScaleR-1.5B-Preview, with one training run per configuration. The limitations section acknowledges that generalization to larger models is untested, but the abstract and conclusion assert that A-DLP consistently and dramatically reduces length and is broadly applicable. Without evidence across model scales, model families, or datasets with substantially different base accuracy levels, that consistency claim is not established; the acc_ref sensitivity in Section 5.3 suggests the method's behavior could change materially on a model whose accuracy distribution differs from the one used here.","section":"§7 and §4"}],"minor_comments":[{"comment":"The phrase 'math mathematical reasoning' contains a duplicated word and should be corrected.","section":"Abstract"},{"comment":"The sentence beginning 'Although we expect A-DLP to generalize to larger models... scaling, it remains an important next step' is grammatically incomplete and should be rewritten.","section":"Section 7"},{"comment":"The caption reads 'For A-LDP' but should read 'For A-DLP'.","section":"Appendix Figure 8 caption"},{"comment":"The rule of thumb equating lambda_0 and eta is supported only by Figure 7, which varies both together; the independent variations in Figures 9–11 show a more complex interaction, so the guidance should be presented as an empirical heuristic rather than as a derived principle.","section":"Section 5.3"},{"comment":"The evaluation metric is described only as 'accuracy' over 16 sampled completions per question; the paper should state explicitly whether this is pass@1, majority vote, or some other aggregation.","section":"Section 4"},{"comment":"The validation-accuracy panel for AIME2024 is not anchored to the base model's AIME2024 accuracy in the text, so the reader cannot assess the magnitude of the observed drop from the figure alone.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional verdict matches my own reading. The main issue is not the idea itself, which is simple and plausible, but that the paper's own sensitivity analysis shows the method is fragile to acc_ref perturbations of the same size as the noise in the estimator used to set acc_ref, and the absence of numeric tables and multiple seeds leaves the headline claim insufficiently supported. I would be willing to reconsider after the authors provide a robust acc_ref estimation procedure, report quantitative results with variance over seeds, and strengthen the S-DLP baseline comparison with a lambda sweep."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The idea is straightforward and worth a look: instead of a fixed length penalty in GRPO, adjust lambda with an integral controller on the accuracy gap. That is the new bit — L1, O1-Pruner, and ShorterBetter all fix the coefficient. The experiments show the adaptive version stays above the static-penalty trajectory on most datasets, and the paper's own Section 5.2 dynamics are informative. Code is released, which is a real plus.\n\nThe soft spots are real, though. No numeric tables, no seeds, no error bars. All claims come from figures, and the key numbers (50% length reduction, less than 0.04 accuracy loss) cannot be checked from the text. The bigger problem is acc_ref. It is estimated from one 64-prompt batch. With accuracy near 0.6, the standard error is about 0.06, and Section 5.3 shows that a miss of +/-0.1 causes either no length reduction (accref too high) or collapse (accref too low). That means the headline result holds only if the one-batch estimate lands within roughly +/-0.05 of the true value, which is not guaranteed. The paper offers a rule of thumb (set lambda0 = eta and hope the deviation stays under 0.1) but no robust way to obtain accref without extra evaluation runs, which the authors explicitly want to avoid. The stress-test note is on target: the central claim is conditional, not a general property.\n\nAlso, the S-DLP baseline collapses by construction, which makes A-DLP's stability look natural. That is not fatal, but it weakens the 'avoids collapse' selling point.\n\nOn the other hand, the authors are honest about the sensitivity and include a limitation section. The method is simple and plausible, and the training dynamics analysis (lambda_t converging to zero, length stabilizing around 1500-2000 tokens) is a nice touch.\n\nWho is this for? Researchers working on RL-based efficient reasoning. It deserves a serious referee: the idea is relevant, the code exists, and the sensitivity issue, while serious, is fixable with more seeds, a proper table, and a better procedure for accref. I would not desk-reject this, but I would ask for substantial revision.","headline":"Adaptive length-penalty reward is a good idea, but the evidence here is too fragile to justify the headline claims without revision.","tokens_in":10818,"tokens_out":2303,"would_cite":true,"duration_ms":17867,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An adaptive length penalty in RL training cuts LLM reasoning tokens by more than half while losing less than 0.04 accuracy.","keywords":["adaptive reward shaping","length penalty","reinforcement learning","reasoning length","LLM reasoning","token efficiency","GRPO","overthinking"],"falsifier":"Run A-DLP on a different RL-trained reasoning model with $\\mathrm{acc}_{\\mathrm{ref}}$ estimated from its first training batch. If the trained model does not show the predicted coupling—token length falling by over 50% while accuracy drops less than 0.04, with the static-penalty curve below and to the right—then the claim fails to transfer. A more direct mechanistic check: deliberately set $\\mathrm{acc}_{\\mathrm{ref}}$ 0.1 above the true base accuracy; the paper predicts $\\lambda_t$ reaches zero within about 10 steps and length reduction plateaus, while 0.1 below predicts over-penalization and collapse. A run that avoids both failure modes would contradict the proposed control law.","tokens_in":9661,"feed_emoji":"⚖️","tokens_out":8267,"duration_ms":68152,"temperature":0.7,"pith_summary":"Reasoning models trained with reinforcement learning often emit thousands of tokens even for simple math questions, and fixed length penalties are hard to tune. This paper proposes a self-tuning reward shaper: when the model stays accurate, the length penalty strengthens and pushes output length down; when accuracy slips, the penalty relaxes so correctness is not traded away. The claimed result is an average token reduction of more than 50% with an accuracy loss under 0.04 on five math benchmarks, plus stable convergence instead of the collapse that fixed penalties trigger. That matters because inference cost and latency, not just benchmark accuracy, determine whether long-reasoning LLMs are practical to deploy.","feed_headline":"Adaptive reward halves LLM reasoning tokens, costs 0.04 accuracy","feed_subtitle":"The penalty tightens when accuracy holds and relaxes when it dips, avoiding the collapse fixed penalties cause.","key_machinery":"The central object is a reward signal, not a new model or architecture: the adaptive length penalty coefficient $\\lambda_t$ inside $R_{\\lambda_t}(x,y) = \\mathbb{I}\\{y=y^*\\} - \\lambda_t \\, \\mathrm{len}(y)$. The control law in Eq. (3), $\\lambda_{t+1} = \\max(0, \\lambda_t + \\eta(\\mathrm{acc}_t - \\mathrm{acc}_{\\mathrm{ref}}))$, turns the difference between observed and reference accuracy into a proportional adjustment to the penalty. That feedback loop is what carries the argument: it makes length compression aggressive early in training, when the model is verbose but still accurate, and gentle later, when further shortening would begin to cost correctness. The non-negativity clamp lets the penalty turn off completely, which the paper identifies as the mechanism that prevents training collapse.","core_discovery":"The central discovery is that replacing a fixed length penalty with a penalty coefficient that tracks the model's own accuracy produces a better accuracy–length frontier. The reward is $R_{\\lambda_t}(x,y) = \\mathbb{I}\\{y=y^*\\} - \\lambda_t \\, \\mathrm{len}(y)$, and the coefficient moves by the update $\\lambda_{t+1} = \\max(0, \\lambda_t + \\eta(\\mathrm{acc}_t - \\mathrm{acc}_{\\mathrm{ref}}))$. When the model is more accurate than the reference, the penalty grows and drives length down; when accuracy slips, the penalty decays so compression does not destroy correctness. In the paper's experiments this yields more than 50% token reduction with less than 0.04 accuracy loss on five math benchmarks, places the adaptive curve above and to the left of the static-penalty curve, and avoids the collapse that fixed-penalty training shows around step 100.","pith_inferences":["The feedback law in Eq. (3) is generic: any observable that should trade off against cost could replace accuracy, so the same update could shape other efficiency axes such as tool-use count or refusal rate.","A rolling estimate of the reference accuracy from recent training batches might remove the calibration sensitivity that makes $\\mathrm{acc}_{\\mathrm{ref}}$ a fragile choice in Section 5.3.","Because correct and incorrect responses shrink proportionally, a version with difficulty-aware penalties could compress easy questions more aggressively while preserving hard-question accuracy.","The results are demonstrated on a 1.5B-parameter model; the method's practical value would increase substantially if the accuracy loss stays below 0.04 when scaled to larger reasoning models."],"forward_implications":["A-DLP can be integrated into an existing RL pipeline through the reward function alone, requiring no architectural or inference-time changes.","Token cost on the tested math benchmarks drops by more than half while accuracy stays within 0.04 of the base model.","The method stabilizes naturally as training continues, removing the need to carefully choose a stopping checkpoint to avoid over-compression.","The penalty shortens both correct and incorrect responses roughly in proportion to their original lengths, with reported reductions exceeding 55% in both categories.","Compared with fixed-penalty baselines, the adaptive curve lies above and to the left on the accuracy–length plot, meaning comparable or better accuracy is reached with fewer tokens."],"supporting_citations":[{"why":"Supplies the base model, training dataset, and evaluation/validation splits used in all experiments.","marker":"[18]"},{"why":"Provides the GRPO policy-update algorithm used to train the model with the shaped reward.","marker":"[25]"},{"why":"Defines the L1-Exact and L1-Max baselines that fix a target length budget, the main comparison for A-DLP.","marker":"[1]"},{"why":"Formulates a constrained length-reduction baseline that enforces accuracy as a hard constraint.","marker":"[17]"},{"why":"Offers another fixed-penalty reward baseline that penalizes deviation from the shortest correct response.","marker":"[35]"},{"why":"Documents the overthinking phenomenon that motivates penalizing unnecessarily long reasoning traces.","marker":"[3]"},{"why":"Is the source of the DeepSeek-R1-Distill-Qwen base from which the DeepScaleR model is fine-tuned.","marker":"[9]"}],"fun_headline_variants":["Adaptive reward cuts AI math reasoning length by 50%","LLMs think faster with adaptive accuracy-length trade-off","Penalty adapts to accuracy, slashing reasoning tokens","Think fast, stay right: adaptive reward trims LLM reasoning","Adaptive length penalty halves token use, keeps accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single reference accuracy estimated from the first training batch is close enough to the model's true accuracy to drive the penalty update; Section 5.3 shows that an error of 0.1 in either direction either stalls length reduction or collapses the model.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive reward cuts AI math reasoning length by 50%","LLMs think faster with adaptive accuracy-length trade-off","Penalty adapts to accuracy, slashing reasoning tokens","Think fast, stay right: adaptive reward trims LLM reasoning","Adaptive length penalty halves token use, keeps accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000632,"raw_usage":{"total_tokens":2912,"prompt_tokens":934,"completion_tokens":1978,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":1896}},"tokens_in":550,"tokens_out":1978,"duration_ms":11041,"temperature":1.0,"reasoning_tokens":1896,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:33:10.313643+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run A-DLP on a different RL-trained reasoning model with $\\mathrm{acc}_{\\mathrm{ref}}$ estimated from its first training batch. If the trained model does not show the predicted coupling—token length falling by over 50% while accuracy drops less than 0.04, with the static-penalty curve below and to the right—then the claim fails to transfer. A more direct mechanistic check: deliberately set $\\mathrm{acc}_{\\mathrm{ref}}$ 0.1 above the true base accuracy; the paper predicts $\\lambda_t$ reaches zero within about 10 steps and length reduction plateaus, while 0.1 below predicts over-penalization and collapse. A run that avoids both failure modes would contradict the proposed control law.","supporting_citations":[{"cited_title":"Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica","cited_arxiv_id":null,"evidence_quote":"Supplies the base model, training dataset, and evaluation/validation splits used in all experiments."}],"review_version":1}