{"id":"920276d4-0241-4dee-b0db-92721dc0bcb7","arxiv_id":"2506.18237","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"AdapThink is an RL post-training framework that adaptively reduces overthinking and underthinking in reasoning language models by rewarding confidence-appropriate reasoning depth and diverse training samples.","lead":"This paper introduces AdapThink, a reinforcement learning post-training framework that teaches reasoning models to think adaptively: spend less effort on easy problems and keep exploring on hard ones. Instead of a fixed token budget, AdapThink uses the model's confidence on a group of sampled answers and counts of reflection words such as \"wait\" and \"alternatively\" to shape rewards.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 3's GRPR reward is never plugged into the GRPO advantage (Eq. 10); Algorithm 1 says 'with GRPR reward and accuracy reward' but gives no combination rule, so the reported gains are not attached to a defined objective.","rationale":"The reader's CONDITIONAL verdict is appropriate, and this stress-test identifies a sharper, more direct version of one of the reader's stated reasons: the training objective is under-specified. AdapThink is an RL method, so the reward function is the method; Eq. 3 cannot be left disconnected from Eq. 10. Algorithm 1's phrase 'with GRPR reward and accuracy reward' is not a formal specification, and no coefficient or normalization is given anywhere in the paper. This means the causal claim 'the GRPR reward induces adaptive thinking' cannot be evaluated, and the comparison to baselines cannot be reproduced even with code, because the code would have to fill in a missing design decision. The transition-word proxy identified by the reader is a real weakness, but it is subordinate: even if the words were perfect, the undefined reward combination would still block verification. Secondary observations reinforce rather than change the verdict: the 27% average improvement in Section 6.1 does not match any natural aggregation of Table 1, and token efficiency is not uniform across benchmarks (LCPO uses fewer tokens on AIME2024 and AMC, though with much lower accuracy). These are correctable presentation and specification issues, not evidence of fraud, so the verdict remains CONDITIONAL with no adjustment needed.","tokens_in":13006,"tokens_out":8959,"duration_ms":100915,"concrete_test":"Re-implement AdapThink strictly from Algorithm 1, Appendix C, and Table 4, and try explicit reward combinations: (a) A_i = (I(y_i=y) + β·r_GRPR(x,G,θ) - group_mean)/group_std; (b) A_i = I(y_i=y) + β·r_GRPR(x,G,θ); and (c) A_i = r_GRPR(x,G,θ), for β ∈ {0.1, 0.5, 1.0}. Train each variant on the 5K DeepScaleR subset with the 2K token limit and evaluate on AIME2025 with an 8K token limit. Report which combination reproduces the Table 1 AdapThink row (PASS@1 = 25.42, average tokens = 6455). If none does, the central claim is not attached to a fully specified method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on a fully specified RL update, but the paper never defines how the GRPR reward r(x,G,θ) from Eq. 3 enters the GRPO objective. Algorithm 1 (line 15) states 'Update πθ using Equation 9 with GRPR reward (Equation 3) and accuracy reward,' yet Equation 9 is the standard GRPO objective whose advantage A_i is defined in Equation 10 using only the binary accuracy reward r_i = I(y_i = y). No equation, appendix, or hyperparameter table specifies whether A_i = (I(y_i=y) + β·r_GRPR - group_mean)/group_std, whether r_GRPR is an additive reward term with some coefficient β, or whether r_GRPR replaces the accuracy reward. Because every PASS@1 and token count in Table 1 is produced by this undefined update, the headline comparison against LCPO/TLB/CosFn is not reproducible from the paper. This is more load-bearing than the transition-word proxy: even if the eight selected words are a perfect proxy for reasoning quality, the reader still cannot tell what objective was optimized. The missing justification for the 27% summary statistic in Section 6.1 (the average relative gain from Table 1 is about 20.7% or 31.6% depending on aggregation, not 27%) and the absence of error bars are secondary; the undefined reward combination is primary.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AdapThink, an RL post-training framework for reasoning language models that adaptively controls reflection-related transition words instead of imposing fixed length budgets. It introduces a group-relative reasoning preference reward (GRPR) based on model confidence and correctness-group-relative statistics of token length, completion markers, and 'Branch-Extension' words, and a diversity-aware sampling mechanism that oversamples and then entropy-selects a balanced group. Experiments on AIME2025, AIME2024, MATH500, and AMC with DeepSeek-R1-Distill-Qwen-1.5B report higher PASS@1 and lower average token counts than GRPO, TLB, LCPO, and CosFn under an 8K inference budget, together with ablations and an n-gram repetition analysis.","tokens_in":13335,"tokens_out":3197,"duration_ms":35554,"significance":"If the results hold, AdapThink is a useful contribution to the growing literature on efficient reasoning in RL-trained LLMs: it targets reasoning style rather than raw length, introduces diversity-aware group construction, and reports gains over several length-control baselines on public math benchmarks. The paper also includes multiple ablations (reward components, oversampling factor, curriculum training, word-type variants) that go beyond a single main comparison. However, the central empirical claim is not yet reproducible from the manuscript because the reward that is claimed to drive training is never connected to the GRPO objective, and no variance or seed information is given. The work would be strengthened by releasing code and exact training configurations, including the missing reward-combination rule.","major_comments":[{"comment":"The paper never defines how the GRPR reward r(x,G,θ) from Eq. (3) enters the GRPO update. Algorithm 1 line 15 says 'Update πθ using Equation 9 with GRPR reward (Equation 3) and accuracy reward,' but Eq. (9) is the standard GRPO objective whose advantage A_i in Eq. (10) is computed from the binary accuracy reward r_i = I(y_i = y) only. No equation, appendix, or hyperparameter table specifies whether the GRPR term is added to r_i with a coefficient, replaces the accuracy reward, or is incorporated into the group-relative normalization. Because every PASS@1 and token count in Table 1 is produced by this undefined update, the headline comparison against LCPO/TLB/CosFn is not reproducible from the paper. Please specify the exact combined advantage or reward formalism, including any coefficient for r(x,G,θ).","section":"§4.1, §5, Algorithm 1 (Eqs. 3, 9, 10)"},{"comment":"The claim of a '27% average performance improvement' is not supported by the numbers in Table 1. Averaging the relative PASS@1 gains across the four benchmarks gives approximately 31.6% in one simple aggregation, while other reasonable aggregations give around 20.7%; neither equals 27% unless an unreported weighting is used. Please state precisely how the 27% figure is computed, or remove it. In addition, no standard deviations, confidence intervals, or number of independent seeds are reported for any entry in Tables 1–3, so it is impossible to assess whether the observed differences are statistically meaningful rather than checkpoint noise.","section":"§6.1, Table 1"},{"comment":"The same hand-picked transition words ('wait', 'hold on', 'check', 'verify' for Pause-Validation; 'alternatively', 'however', 'another', 'instead' for Branch-Extension) are used both in the reward (Eq. 3 via λ_b) and in the evaluation metrics (n_p, n_b). This creates a self-referential loop: the model is rewarded for reducing words that are then counted as evidence of efficiency. The paper should provide a concrete test that the chosen word sets remain a valid proxy after training, e.g., by showing that reductions in these words correlate with human-judged reasoning quality or with held-out efficiency metrics that do not depend on the same lexicon, on datasets not seen during training.","section":"§3, §4.1, §5 (Evaluation Protocol)"},{"comment":"Several implementation details needed to reproduce the method are missing or inconsistent. Eq. (5) defines H_* over bins S but does not define how the bins are constructed for length and word counts beyond 'four equal intervals'; the text says α_l = α_b = α_d = 1 while Eq. (6) lists α_l, α_p, α_b; and Table 4 reports batch size 8 with gradient accumulation 8 while the text states a batch size of 32. Please clarify the binning, the exact weights used, and the effective batch size.","section":"§4.2, Eq. (5), Table 4"}],"minor_comments":[{"comment":"There are several typos and inconsistencies: 'Methdology' in the Section 4 heading, 'queations' in Section 5, 'Coressponding author' in the footnote, and inconsistent capitalization of 'DeepSeek' (e.g., 'Deepseek-distilled'). A careful proofread is needed.","section":"Throughout"},{"comment":"The rows for FastCuRL-1.5B-24K and DeepScaler-1.5B-24K are marked with a footnote saying they were originally trained with a 24K token limit and are evaluated here with an 8K limit. This is a legitimate reference comparison, but the paper should explicitly note that those models were not trained for the 8K budget and that the comparison may therefore disadvantage them.","section":"Table 1 / §6.1"},{"comment":"The example 'AdapThink Example (from AIME 2025)' reports 4,502 tokens and 19 reflection words, while the base model example reports 7,377 tokens and 46 reflection words. These single examples are illustrative, but the text should note that they are anecdotal and not a substitute for the aggregate statistics.","section":"§E.3, Appendix"},{"comment":"The n-gram repetition analysis is a useful reward-hacking check, but the table would be easier to interpret if it reported the standard deviation across test examples or a statistical test comparing AdapThink's 0.7% with GRPO's 0.3%, since both are very small numbers and the qualitative claim about 'consistently low' rates would benefit from uncertainty quantification.","section":"§7.2, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is timely and the experimental design is more thorough than many short RL-post-training papers, but the missing definition of the actual training objective is a serious reproducibility gap. I would encourage the editor to require the authors to provide the exact reward-combination rule and ideally release code or detailed configuration files before considering the paper further. The absence of seeds and variance is also a concern for a paper whose conclusions are comparative; this should be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"AdapThink is a candidate for a solid incremental contribution to efficient reasoning for small LLMs. The new piece is the confidence-gated, group-relative reward over a hand-picked set of transition words (\"wait\", \"check\", \"alternatively\", etc.) combined with an entropy-based diversity-aware sampler. That combination isn't in the cited baselines, and the paper does a reasonable job of motivating it with an observational analysis of token-length and transition-word distributions on a 1.5B DeepSeek-distilled model.\n\nWhat the paper does well: the method is clearly described (except for one significant gap I'll come to), the ablations show each reward component and the diversity sampler contribute, and the n-gram repetition analysis is a nice check against reward hacking. Results are consistent across four math benchmarks: AdapThink beats GRPO, LCPO, TLB, and CosFn on PASS@1 while using fewer tokens on average. That's not a huge win, but it's a real one.\n\nThe soft spots are mostly fixable. The big one: the GRPR reward from Eq. 3 is never shown to enter the GRPO objective. Algorithm 1 says \"Update πθ using Equation 9 with GRPR reward and accuracy reward,\" but Eq. 9 is just the standard GRPO objective with advantage from Eq. 10 defined only over the binary accuracy reward. No equation, appendix, or hyperparameter table says whether it's additive, a coefficient-weighted term, or a replacement. That's load-bearing: the 2K-token training results in Table 1 all come from an undefined update. A referee should require this to be spelled out before the numbers can be trusted.\n\nSecondary issues: no seeds or variance, several free parameters (φ_low, φ_high, T_min, F_min, α's) without sensitivity analysis, and the \"27% average performance improvement\" in Section 6.1 doesn't match the numbers in Table 1 (my calculation gives ~31.6% average relative gain on PASS@1, or ~8.7 percentage points; either way the 27% isn't substantiated). The transition-word selection is a bit self-referential—words chosen from the base model's behavior are both rewarded and measured—but the authors acknowledge this and the sequential-word control in the appendix is a good guard against picking arbitrary words.\n\nWho is this for? People working on efficient RL post-training for small reasoning models, especially length-control methods. The idea is worth engaging with; it just needs a clear objective and variance-aware evaluation. I'd send it to a serious reviewer.\n\nRecommendation: engage with it, ask for a rewrite that defines the reward combination and adds error bars.","headline":"AdapThink has a plausible idea and decent results, but the missing reward combination makes it unreproducible; fix that and it's a solid incremental contribution.","tokens_in":13893,"tokens_out":3236,"would_cite":true,"duration_ms":33116,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By training inside a 2K-token budget, AdapThink gets a small reasoning model to solve more math problems with fewer tokens than four length-control baselines.","keywords":["reinforcement learning","chain-of-thought efficiency","adaptive reasoning length","reward design","diversity-aware sampling","mathematical reasoning","overthinking","underthinking"],"falsifier":"Take a benchmark where correct solutions naturally require many branch-extension moves, such as combinatorial proof problems with multiple cases. Train two copies of a reasoning model with AdapThink's reward: one using the hand-picked word list and one using a semantic measure of branching (e.g., the number of distinct proof paths attempted). If the word-count version shows lower PASS@1 or loses its token advantage over the semantic version, the word proxy, rather than the adaptive control, is the limiting factor.","tokens_in":12800,"feed_emoji":"🧮","tokens_out":6128,"duration_ms":60058,"temperature":0.7,"pith_summary":"AdapThink is a post-training method for reasoning language models that tries to make the model think for the right amount of time: more reflection on hard problems, less on easy ones, without being told a fixed token budget. The paper's central claim is that a group-relative reward built from model confidence and counts of two kinds of transition words—'pause-validation' words and 'branch-extension' words—can steer a small distilled reasoning model to higher accuracy and lower token use than four length-control baselines. The authors train with only a 2K-token cap and evaluate at 8K tokens, reporting that AdapThink improves first-attempt accuracy (PASS@1) while cutting average response length. If true, this matters because it points to a lightweight, word-level lever for controlling reasoning depth that does not need a stronger external judge or hand-set length limits.","feed_headline":"Short-budget training yields faster, more accurate math reasoning","feed_subtitle":"AdapThink adapts reflection depth to model confidence, cutting tokens while improving PASS@1 on four benchmarks.","key_machinery":"The carrying mechanism is the group-relative reasoning preference reward (GRPR), Eq. 3, together with the confidence-constrained diversity sampling of Eq. 8. GRPR uses $\\varphi$, the fraction of correct answers in a group, to interpolate with a cosine function between two reward regimes: for $\\varphi \\le \\varphi_{\\mathrm{low}}$ it emphasizes completion and length control; for $\\varphi \\ge \\varphi_{\\mathrm{high}}$ it adds suppression of branch-extension words. The reward components are normalized deviations from the group mean of length, completion, and branch-extension counts, so the signal is relative to what the current model already produces rather than an absolute target. Diversity-aware sampling first oversamples by a factor $K$, then selects groups that maximize an entropy-weighted score over token length, pause-validation words, and branch-extension words, while enforcing minimum counts of correct and incorrect answers.","core_discovery":"The paper claims that reasoning efficiency is better controlled by regulating the preference for reflection-related transition words than by penalizing length directly. On a 1.5B distilled reasoning model, AdapThink's group-relative reasoning preference reward compares each response's token count, completion marker, and branch-extension word count with the average of the correct or incorrect group to which the response belongs, then weights those comparisons by the model's group-level confidence $\\varphi$. When the model is confident, the reward suppresses branch-extension words; when it is uncertain, the reward mainly encourages completing the answer while keeping length near the group mean. The training group is built by oversampling and then selecting samples to maximize an entropy-based diversity score over length, pause-validation words, and branch-extension words. Under a 2K-token training cap, the paper reports PASS@1 improvements over GRPO, LCPO, TLB, and CosFn on AIME2025, AIME2024, MATH500, and AMC, with fewer tokens on average, and low n-gram repetition rates that suggest the method does not fall into the reward hacking seen with strict length rewards.","pith_inferences":["The word-count proxy could be extended to semantic-level controls: if reflection quality is judged by content overlap rather than surface words, the same confidence-relative design might work for open-ended or non-mathematical reasoning.","Because the reward is defined relative to the model's own current output distribution, it may be portable across model scales without retuning the confidence thresholds, though the paper only demonstrates one 1.5B model.","A direct test would be to swap the hand-picked word lists for model-generated synonyms; if accuracy degrades, the specific vocabulary matters more than the underlying behavior.","The framework could be combined with learned difficulty estimates or separate confidence models to replace the group-accuracy $\\varphi$ with a per-sample uncertainty measure."],"forward_implications":["Training under a short 2K-token cap transfers to 8K-token evaluation, so budget pressure during training can teach efficient habits rather than truncation.","Controlling branch-extension words specifically, rather than all reflection words, is what keeps accuracy high while shortening responses.","The confidence-interpolated reward means the same rule automatically lets a weaker model think longer and a stronger model think shorter, without a per-question difficulty estimate.","Diversity-aware selection with oversampling improves both accuracy and token efficiency, suggesting that group-relative rewards need diverse groups to avoid collapsing to one pattern.","Low 40-gram repetition rates relative to LCPO indicate the reward does not encourage degenerate repetition, a known failure mode of hard length penalties."],"supporting_citations":[{"why":"the base reasoning model and the GRPO algorithm that AdapThink builds on and compares against as a baseline.","marker":"[3]"},{"why":"LCPO, a length-penalty baseline that must be beaten on accuracy and token efficiency.","marker":"[1]"},{"why":"TLB/DAST, the difficulty-adaptive token-budget baseline whose static budget motivates AdapThink's confidence-adaptive reward.","marker":"[21]"},{"why":"CosFn baselines and the N-gram repetition metric used to diagnose reward hacking.","marker":"[28]"},{"why":"the dynamic upsampling strategy that motivates the diversity-aware sampling stage.","marker":"[29]"},{"why":"the training dataset and curriculum schedule used for data and progressive token-limit training.","marker":"[10]"}],"fun_headline_variants":["Adaptive reasoning training saves tokens, lifts accuracy","Confidence-driven thinking: fewer tokens, better math","Adaptive reflection preferences cut compute, keep accuracy","Adapt thinking depth to model confidence for efficient reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole reward rests on the assumption that counts of a fixed set of transition words ('wait', 'hold on', 'check', 'verify' for pause-validation; 'alternatively', 'however', 'another', 'instead' for branch-extension) reliably mark wasteful versus useful reasoning, both before and after training.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive reasoning training saves tokens, lifts accuracy","Confidence-driven thinking: fewer tokens, better math","Adaptive reflection preferences cut compute, keep accuracy","Adapt thinking depth to model confidence for efficient reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000417,"raw_usage":{"total_tokens":2156,"prompt_tokens":959,"completion_tokens":1197,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":1137}},"tokens_in":575,"tokens_out":1197,"duration_ms":10446,"temperature":1.0,"reasoning_tokens":1137,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:23:25.278583+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a benchmark where correct solutions naturally require many branch-extension moves, such as combinatorial proof problems with multiple cases. Train two copies of a reasoning model with AdapThink's reward: one using the hand-picked word list and one using a semantic measure of branching (e.g., the number of distinct proof paths attempted). If the word-count version shows lower PASS@1 or loses its token advantage over the semantic version, the word proxy, rather than the adaptive control, is the limiting factor.","supporting_citations":[],"review_version":1}