{"id":"7559fda7-e583-43b1-97be-2fe2dc0b2e84","arxiv_id":"2505.13379","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A 1.5B model learns to decide when to use short versus long reasoning via control tokens and a decoupled GRPO objective, reducing thinking-mode usage by 50-90% on math benchmarks with minor accuracy loss.","lead":"Thinkless trains a small reasoning model to choose between short and long answers using two special tokens, cutting long-chain thinking by 50% to 90% on math benchmarks. The key contribution is DeGRPO, a reinforcement learning variant that stabilizes training and prevents the model from collapsing into one mode.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reward defines 'short' by the control token, not by actual response length, so the headline Think% may not reflect true reasoning usage; per-mode token statistics are needed to verify the central claim.","rationale":"The reader's gamma concern is real but secondary: γ sets the steepness of the accuracy/efficiency trade-off, yet even a correctly chosen γ would not establish that the model stops reasoning when it emits . The load-bearing gap is that the reward labels 'short' by the control token, not by response length, while the response-loss term continues to optimize all subsequent tokens. This permits the policy to collect the short-mode bonus while still reasoning at length, so the central behavioral claim is not secured by reported Think% and total token counts alone. A per-mode length audit from the released code would settle this. Separately, I found the DeGRPO derivation internally consistent and the U-shaped dynamics in Figure 3 plausible; the mode-collapse comparison with vanilla GRPO is credible. The unreported γ and the missing per-mode length statistics both warrant the reader's CONDITIONAL verdict, so no change in verdict is needed.","tokens_in":13191,"tokens_out":12816,"duration_ms":125083,"concrete_test":"Using the released checkpoint and evaluation config, decode MATH-500 and record response token counts separately for - and -initiated trajectories. Compare the median and 90th-percentile length of trajectories with (a) the Qwen2.5-Math-1.5B-Instruct short-teacher distribution and (b) the model's own distribution. If the median is not within about 2x of the short teacher, or more than roughly 10% of responses are longer than the median response, the model is gaming the control-token label and the '50-90% reduction' claim does not measure actual thinking. If the lengths are cleanly separated per mode, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 defines the reward as a function of the control token c and answer correctness only: a -labeled correct response earns 1.0 and a -labeled correct response earns 1−γ, regardless of the number of tokens actually generated. Because the response loss in Eq. (4) updates every token after the control token, nothing prevents the policy from emitting and then still producing a long chain-of-thought; that would collect the higher short-mode reward whenever the answer is correct. The paper's headline metric 'Think%' therefore counts the self-reported mode, not actual computation. Table 1's total #Tokens do drop, which is encouraging, but the paper never reports per-mode length statistics after RL, so we cannot tell whether the model learned 'when to think' or merely 'when to say '. The SFT warm-up initially ties to concise Qwen outputs, but RL can drift because the reward does not enforce conciseness except through the arbitrary preference margin γ, whose value is never stated in the paper. This is load-bearing for the central claim: if a substantial fraction of trajectories remain verbose, the 50–90% reduction in long-chain thinking is overstated and the accuracy/efficiency trade-off is misrepresented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Thinkless, a two-stage framework for training an LLM to decide between a concise response and a detailed response. The method first distills paired long/short responses into a base reasoning model via SFT, then applies a proposed Decoupled Group Relative Policy Optimization (DeGRPO) that separates the loss on the control token from the loss on response tokens. Experiments on AIME, Minerva Algebra, MATH-500, and GSM8K report Pass@1 close to the base reasoning model while reducing the fraction of queries by roughly 50–90% on the easier benchmarks. The paper also analyzes training dynamics, showing that vanilla GRPO collapses to a single mode while DeGRPO exhibits a U-shaped learning curve.","tokens_in":13426,"tokens_out":6756,"duration_ms":59422,"significance":"If the central claim is verified, the work is a practical and timely contribution to efficient reasoning: it shows that a small reasoning model can learn a difficulty- and capability-dependent routing policy without an external router, and the decoupled objective addresses a real mode-collapse problem in applying GRPO to hybrid responses. The release of code is a strength, and the U-shape finding plus the calibration case study provide falsifiable qualitative predictions. However, because the reward is defined over the control token rather than actual output length, the headline efficiency figures need additional verification before the significance can be fully assessed.","major_comments":[{"comment":"The reward function r(a,y*,c) depends on the control token c and answer correctness only, not on the actual number of tokens generated. Consequently, after RL the policy can emit and still generate a long chain-of-thought, receiving the higher reward 1.0 whenever the extracted answer is correct. The headline \"Think%\" therefore measures the self-reported mode, not actual computation. While the total #Tokens reported in Table 1 do decrease, per-mode token statistics (e.g., average token count for and trajectories) are not reported, so we cannot verify that short-mode responses remain concise after RL. If many trajectories remain verbose, the 50–90% reduction in long-chain thinking is overstated and the accuracy/efficiency trade-off is misrepresented. Please report per-mode lengths and consider rewarding actual brevity (e.g., a length term) to enforce conciseness.","section":"§3.2, Eq. (4) and reward definition"},{"comment":"The preference margin γ in the reward r(a,y*,c) = 1.0 (short, correct), 1−γ (think, correct), −1 (wrong) is never specified; the paper only states 1>γ>0. This parameter determines the relative pressure toward short responses and directly controls the trade-off between mode reduction and accuracy. Without its value and a sensitivity analysis, the reported results are not reproducible, and the stability of the method under different γ is unknown. Please state γ and include an ablation over γ.","section":"§3.2, reward design"},{"comment":"All results in Table 1 appear to be from a single run with no standard deviations, confidence intervals, or significance tests. Several comparisons are close (e.g., Minerva Algebra Pass@1: 0.9459 for Thinkless vs 0.9577 for the base; GSM8K: 0.8418 vs 0.8347), so without error bars we cannot be confident that the \"within a few points\" accuracy claim holds robustly. Additionally, evaluation is restricted to mathematics benchmarks; the central claim of \"learning when to think\" would be strengthened by results on other domains (e.g., science, code, agentic tasks). This is a moderate but important gap for the reported performance figures.","section":"§4.2, Table 1"},{"comment":"The statement \"we found that the RL will also compress the length of long responses\" is not supported by the described training objective. Since the reward is identical for any correct response regardless of its length, the policy gradient provides no direct incentive to shorten responses within the think mode; it only incentivizes selecting mode. If the authors intend this claim, they should provide evidence (e.g., per-mode token counts) and explain the mechanism.","section":"§4.2, paragraph after Table 1"}],"minor_comments":[{"comment":"Typo: \"direcytly\" should be \"directly\".","section":"§4.1"},{"comment":"The phrase \"the its difficulty\" should be \"its difficulty\".","section":"§2, last paragraph"},{"comment":"The reference to \"as illustrated in Figure 3\" appears to be a mistake; the relevant figure in the introduction seems to be Figure 1.","section":"§1, second paragraph"},{"comment":"The stated range \"50%–90%\" is slightly inconsistent with Table 1: on MATH-500 the Think% is 51.56%, corresponding to a reduction of about 48.4% from a hypothetical 100% baseline, which is marginally below 50%. Please clarify the exact calculation or adjust the range.","section":"§4.2, Finding 1"},{"comment":"The phrase \"mode-mode imbalance\" is confusing; it likely should be \"mode-short imbalance\" or \"think-short imbalance\" to match the later terminology.","section":"§3.2, first paragraph after Eq. (3)"},{"comment":"The legend entries \"= 0.001\" and \"= 0.5\" are ambiguous; they should explicitly refer to the control-token weight α and the axis should be labeled accordingly.","section":"Figure 4"},{"comment":"Reference [37] \"Aime problem set 1983-2024\" is not a standard citable source; a proper dataset reference or URL would be more appropriate.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main concerns are addressable with additional experiments and reporting (per-mode token statistics, γ value and ablation, error bars). I do not see fundamental flaws in the method, and the paper fits the journal's scope. The central claim is defensible but needs the proposed validation before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper has a real idea—decouple the control-token loss from the response loss in GRPO so a hybrid reasoning model can learn when to emit a vs. token—and it shows a clean mode-collapse failure in vanilla GRPO that the decoupling fixes. The U-shaped learning curve is a nice diagnostic. Worth engaging with.\n\nWhat's new: DeGRPO is a simple but effective modification. The reward is minimal: short correct = 1, long correct = 1−γ, wrong = −1. The authors report 50–90% reduction in usage with small accuracy loss on math benchmarks. Comparisons to merging, CoT-Valve, and router baselines are useful and broadly fair.\n\nSoft spots: (1) γ is never stated, and the reward for long correct answers depends on it. That's a real omission but easily fixed. (2) No error bars; single runs. Standard for the field, so the exact reduction numbers are uncertain. (3) The stress-test concern about control tokens: because the reward uses the control token rather than actual response length, a model could in principle say and still produce verbose reasoning. The paper doesn't report per-mode token statistics after RL. That said, overall #Tokens in Table 1 drop substantially, and warm-up distillation ties to concise Qwen outputs, so the concern is plausible but not damning. Read Think% as self-reported mode, not measured computation.\n\nThe central claim—that RL can learn to allocate thinking adaptively—holds up. The paper is limited to math and a 1.5B model, so generality is unproven, but the mechanism is clear.\n\nRecommendation: send to peer review. A good referee can ask for γ, error bars, and per-mode token lengths. Code is public. I'd bring it to a reading group only if someone cares about efficient reasoning; I'd cite it in work on token-efficient RL.","headline":"A real RL training fix for hybrid reasoning—worth refereeing, but the headline Think% metric needs per-mode token stats and a few missing hyperparameters.","tokens_in":13895,"tokens_out":1697,"would_cite":true,"duration_ms":15860,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reinforcement-learning policy can teach a reasoning language model when to think, cutting long-chain usage by 50–90%.","keywords":["reasoning language models","hybrid reasoning","reinforcement learning","GRPO","control tokens","inference efficiency","mode selection","chain-of-thought"],"falsifier":"Retrain the same pipeline with the short-correct reward margin $\\gamma$ set to two extremes, for example $0.01$ and $0.99$, and measure long-thinking usage and Pass@1 on MATH-500 and AIME 2024. If the 50–90% reduction and the near-baseline accuracy do not both survive across this sweep, the headline efficiency gain is tied to the unreported reward margin rather than to DeGRPO itself.","tokens_in":13009,"feed_emoji":"🧠","tokens_out":8917,"duration_ms":78856,"temperature":0.7,"pith_summary":"The paper asks whether a reasoning language model can decide, for each input, whether to think at length or answer immediately, instead of always emitting a long chain of thought. It proposes Thinkless, a two-stage scheme: first distill paired long and short responses so the model can produce either style, then optimize a decoupled reinforcement-learning objective, DeGRPO, that treats choosing the reasoning mode as a separate learning signal from improving answer accuracy. The central claim is that this lets a small reasoning model route easy questions to short answers, cutting long-chain thinking by 50–90% on standard math benchmarks while keeping the single-answer correctness (Pass@1) within a few points of the always-think baseline. If correct, this matters because the dominant cost of reasoning models is token generation; adaptive thinking makes that cost depend on the query instead of being fixed for every query.","feed_headline":"LLM learns when to think, cutting chain-of-thought use 50-90%","feed_subtitle":"Decoupled RL lets the model answer easy queries directly and reserve reasoning for hard ones, with little accuracy loss.","key_machinery":"The load-bearing object is the DeGRPO objective, a decoupled variant of group-relative policy optimization. The paper splits the sequence loss into a control-token term $\\alpha \\mathcal{L}_{i,0}$ and a response term $\\frac{1}{T_i}\\sum_{t=1}^{T_i}\\mathcal{L}_{i,t}$, removing the length normalization that makes vanilla GRPO under-update the token. The reward is deliberately simple: a correct short answer gets $1.0$, a correct long answer gets $1-\\gamma$, and any wrong answer gets $-1$; advantage is computed against the group mean, which is robust to mixed-difficulty batches. This decomposition is what lets the same model improve its short-form answer quality while learning which queries deserve long reasoning.","core_discovery":"The paper's central claim is that a reasoning LLM can learn when to think from reward feedback alone, without a hand-built router or difficulty heuristic. On MATH-500, Minerva Algebra, and GSM8K, the trained model reduces long-chain thinking usage by roughly 50–90% while keeping Pass@1 within a few points of the original reasoning model; on AIME 2024, where nearly every problem is hard, it uses thinking mode on all queries and retains essentially the same accuracy. The decision is carried by a control token emitted as the first output token— or —so switching modes is a single-token decision. DeGRPO normalizes the control-token and response-token losses separately, which prevents the mode signal from being drowned out by hundreds of response tokens and stops the collapse to a single mode that the paper observes in vanilla GRPO.","pith_inferences":["A direct follow-up is to sweep the unreported reward margin $\\gamma$ between correct short and correct long answers; the claimed 50–90% saving is one operating point on an accuracy-versus-token frontier, not a fixed property of the algorithm.","The same control-token decoupling should transfer to other binary output-mode choices beyond math—for example, whether to call a tool, cite sources, or answer in a formal style—wherever one mode is longer and more expensive than the other.","If the warm-up distillation were replaced by a weaker short-response teacher, the RL stage would have to improve response quality and mode selection simultaneously, which would likely move the reported trade-off; this makes the quality of the paired distillation a hidden precondition for the headline numbers."],"forward_implications":["Simple queries dominate math benchmarks such as GSM8K, so the same model can serve most requests with a fraction of its original tokens while accuracy stays close to the always-think policy.","DeGRPO removes the early collapse to the short mode that vanilla GRPO shows, so both response styles remain trainable enough for the policy to improve short answers instead of merely avoiding them.","The probability the model assigns to acts as a learned difficulty signal, rising on multi-condition problems and falling on routine arithmetic, which could be reused as a soft routability score.","On hard benchmarks like AIME, the model chooses thinking mode on all tested queries, so the efficiency savings do not come at the expense of the hardest questions."],"supporting_citations":[{"why":"Provides the long-chain reasoning model used as the base policy whose accuracy and token counts are the reference point for the reported savings.","marker":"[12]"},{"why":"Supplies the group-relative policy optimization objective and reward-driven training scheme that DeGRPO modifies.","marker":"[30]"},{"why":"Supplies the group-relative advantage estimate used in the surrogate loss.","marker":"[23]"},{"why":"Provides the roughly 40K labeled math problems used as the reinforcement-learning training set.","marker":"[25]"},{"why":"Supplies long-form reasoning traces used to build the paired long/short distillation data.","marker":"[10]"},{"why":"Supplies a larger, more diverse long-form dataset tested in the warm-up distillation comparison.","marker":"[35]"},{"why":"Is a length-controllable chain-of-thought baseline whose reasoning length must be tuned per dataset, the main efficiency comparison for adaptive thinking.","marker":"[26]"},{"why":"Is a model-merging baseline with a fixed reasoning-length trade-off, used to show the advantage of learning the mode decision.","marker":"[34]"},{"why":"Supplies the short-form expert responses used as distillation targets for the concise mode.","marker":"[41]"}],"fun_headline_variants":["Thinkless: LLM learns when to think, cuts reasoning cost 50-90%","LLM adapts: short answers for easy, deep thinking for hard","Reinforcement learning teaches LLM to control its thinking depth","LLM uses chain-of-thought only on hard problems, slashing compute","Learn to think: RL makes LLM skip unnecessary reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported savings rest on a reward that pays a correct short answer more than a correct long answer by a margin the paper never reports; if that margin is too large, the model may stop thinking too often and lose accuracy, and if it is too small, it may keep thinking and save little.","fun_headline_variants_meta":{"raw":{"variants":["Thinkless: LLM learns when to think, cuts reasoning cost 50-90%","LLM adapts: short answers for easy, deep thinking for hard","Reinforcement learning teaches LLM to control its thinking depth","LLM uses chain-of-thought only on hard problems, slashing compute","Learn to think: RL makes LLM skip unnecessary reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001126,"raw_usage":{"total_tokens":4704,"prompt_tokens":988,"completion_tokens":3716,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":3620}},"tokens_in":604,"tokens_out":3716,"duration_ms":25130,"temperature":1.0,"reasoning_tokens":3620,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:14:02.872930+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same pipeline with the short-correct reward margin $\\gamma$ set to two extremes, for example $0.01$ and $0.99$, and measure long-thinking usage and Pass@1 on MATH-500 and AIME 2024. If the 50–90% reduction and the near-baseline accuracy do not both survive across this sweep, the headline efficiency gain is tied to the unreported reward margin rather than to DeGRPO itself.","supporting_citations":[{"cited_title":"Deepscaler: Surpassing o1-preview with a 1.5 b model by scaling rl","cited_arxiv_id":null,"evidence_quote":"Provides the roughly 40K labeled math problems used as the reinforcement-learning training set."},{"cited_title":"Open thoughts, January 2025","cited_arxiv_id":null,"evidence_quote":"Supplies a larger, more diverse long-form dataset tested in the warm-up distillation comparison."}],"review_version":1}