{"id":"7427434e-fec8-40e7-80c9-a3925ea965d9","arxiv_id":"2505.20282","paper_version":4,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Minimizing a language model's token-level entropy on one hand-picked prompt for 10 steps is reported to rival thousands-of-example reinforcement learning.","lead":"A post-training technique called one-shot entropy minimization claims to match or beat rule-based RL for LLM reasoning using a single prompt and just 10 optimization steps. If the claim holds, post-training would become nearly free; the catch is that selecting that one prompt requires labeled answers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'unlabeled one-shot' claim is internally contradicted by Section 2.2: selecting the single prompt requires computing pass@k variance, which needs ground-truth labels, so the method is not label-free as claimed.","rationale":"The reader's verdict is REJECT, and the stress-test pass supports that verdict, so no change is recommended. The reader's stated weakest assumption was the correctness-entropy premise, but the more immediate and documented problem is the Section 2.2 data-selection protocol, which requires ground-truth correctness labels to compute pass@k variance. This directly contradicts the abstract's 'single unlabeled data' and Section 3.1's 'we do not need any data labels.' The reader did mention this contradiction in the rationale, so there is partial agreement. The concern is concrete, internal to the text, and testable by ablation or by inspecting the released code. The paper does have real strengths: it trains many models across seeds, reports seed variance, and identifies overconfidence and instability as limitations; those are honest and useful. But the central framing is not supported by the stated methodology. A random-prompt control would determine whether the effect survives without supervised selection, and a code audit would confirm the label use. If the effect does survive, the remaining claim would still need a matched comparison to RL baselines under identical base models and evaluation protocols, but the supervised-selection issue is the single most load-bearing concern.","tokens_in":11927,"tokens_out":2961,"duration_ms":33515,"concrete_test":"Run the EM protocol on Qwen2.5-Math-7B under three prompt-selection conditions while keeping the 10-step training and all hyperparameters fixed: (a) the variance-selected prompt from Section 2.2 using ground-truth labels; (b) a uniformly random prompt from the same pool; (c) a prompt selected without labels, e.g., by maximum predictive entropy or by self-consistency disagreement across samples. Compare average scores on MATH500, MinervaMath, OlympiadBench, and AMC23 over the same number of seeds. If (a) substantially outperforms (b) and (c), the headline result depends on supervised data selection and the 'single unlabeled data' claim fails. Additionally, audit the released code to confirm whether the data-selection stage invokes correctness labels; if it does, that settles the internal inconsistency.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that entropy minimization achieves RL-comparable gains from a single unlabeled example. However, Section 2.2 defines a data-selection objective x* = argmax Varpass@k(x), where Varpass@k is the variance of a binary correctness variable I[y(i) is correct] computed via execution or string match. Computing this variance requires knowing whether each sampled response is correct, i.e., ground-truth labels for the pool D. Section 3.1 then states 'Since it is an unsupervised method, we do not need any data labels,' and the abstract claims 'requires only a single unlabeled data.' These statements are incompatible: the pipeline uses labels to choose the one prompt, even if the EM loss itself does not consume labels. Because all headline results use the variance-selected prompt, the observed gains may come from supervised selection of an unusually informative example rather than from entropy minimization per se. This is load-bearing because the 'fully unsupervised' and 'single unlabeled data' framing is the paper's main novel selling point; a method that needs a labeled pool to pick its one prompt is a different, weaker claim. The reported gains are also best-of-16-seed numbers with acknowledged high seed variance, but the supervised-selection issue is more fundamental because it directly contradicts the stated contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes one-shot entropy minimization (EM) as a post-training method: minimize token-level entropy on a single prompt for only 10 optimization steps. The authors report that this achieves performance comparable to or better than rule-based RL baselines that use thousands of examples, on math, logic, and code benchmarks. The study is large-scale (13,440 trained models) and includes analyses of logit shifts, temperature sensitivity, and EM before/after RL.","tokens_in":12185,"tokens_out":5758,"duration_ms":57859,"significance":"If the claims held, this would be a striking result: a nearly free, fully unsupervised post-training boost that rivals RL. The paper's strengths are its scale, the release of code, and the attempt to characterize EM's behavior through logits and temperature analyses. However, the central claim is currently undermined by label-based data selection and by reporting best-of-seed results, so the headline contribution is not yet established.","major_comments":[{"comment":"The data-selection step is not label-free. Computing Var_pass@k(x) requires the indicator I[y^(i) is correct] via execution or string match, which requires ground-truth answers. This directly contradicts the abstract's \"single unlabeled data\" and Section 3.1's \"we do not need any data labels.\" Because all reported results use the variance-selected prompt, the observed gains may come from supervised selection of a high-variance example rather than from entropy minimization per se. Please report results with a randomly chosen prompt, or explicitly reframe the contribution as one-shot EM with oracle example selection.","section":"Section 2.2, Eq. (2)-(3)"},{"comment":"The reported results are the peak over 16 seeds, and Figure 3 states that average scores can differ by as much as a factor of two across seeds. Tables 1 and 2 report single numbers without mean/median, standard deviation, or confidence intervals, so the headline improvements may be selection artifacts. Furthermore, Section 3.5 says temperature 0.5 was chosen because it creates \"opportunities for higher peak performance,\" making the final result a best-of-temperature and best-of-seed selection.","section":"Section 3.5 and Section 3.8"},{"comment":"The choice of 10 training steps is made post hoc: the figure shows that benchmark performance peaks at step 10 while the EM loss continues to decrease. Thus \"converges within 10 steps\" is not a property of the method but an early-stopping rule selected using labeled evaluation data. Please report the full performance trajectory and either justify the stopping rule independently or acknowledge that 10 steps is a tuned hyperparameter.","section":"Section 3.4, Figure 2"}],"minor_comments":[{"comment":"The text says \"We present our experimental results in Table 3,\" but the corresponding table is Table 1.","section":"Section 3.2"},{"comment":"The phrase \"as depicted in Figure 3.4\" appears to refer to Figure 2; the reference is incorrect.","section":"Section 3.9"},{"comment":"There are typos: \"avaliable\" in the abstract, \"Acclerate\" in Section 3.1, and \"evalutating\" in Section 3.6.","section":"Abstract and Section 3.1"},{"comment":"The text's LLaMA-3.1-8B numbers (23.6% to 24.3%) and Qwen2.5-7B-Instruct numbers (43.12% to 44.5%) do not match Table 2, which shows 32.0 to 33.1 and 50.3 to 49.8, respectively. This inconsistency makes the cross-model discussion unreliable.","section":"Section 3.8 vs Table 2"},{"comment":"The claim of competitiveness with RL should be qualified: on KK and MBPP, the EM model scores 17.4 and 65.1, while SimpleRL-Zoo scores 27.2 and 78.3, so EM is not universally superior.","section":"Table 1"},{"comment":"The RL baselines may not be directly comparable because they use different base models and training data; please state the base model and data for each baseline clearly.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"This paper's headline is likely to attract attention, but the label-based data-selection step and peak-over-seeds reporting are serious load-bearing issues. If the authors cannot provide average-seed results and results with a random or label-free selected prompt, I would not support acceptance even after revision. The text/table mismatches in Section 3.8 also suggest that a careful data audit is needed before the paper can be trusted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is more honest in the details than in the abstract. The core experiment is cheap and clean: 13,440 training runs, a single entropy-minimization objective, 10 steps, code released. That scale is real and the temperature / seed analysis is useful. But the headline claim — \"requires only a single unlabeled data\" — is not supported by the protocol. Section 2.2 selects the one prompt by maximizing pass@k variance, and computing pass@k requires knowing whether sampled responses are correct via execution or string match. The pool D is implicitly labeled. Section 3.1 says no labels are needed; that is wrong on the paper's own description. This is not a nit: every headline number comes from the variance-selected prompt, so the reported gains could come from supervised selection of an informative example, not from entropy minimization per se.\n\nSecond soft spot: the reported results are peak over 16 seeds, and Section 3.5 admits up to 2x variance across seeds. That makes the headline comparisons to RL baselines shaky, though the broad pattern across models and temperatures is probably stable.\n\nWhat is genuinely new: the combination of one-shot entropy minimization with a 13,440-run sweep, the logits-shift analysis, and the EM-before/after-RL asymmetry. The related work is fair; they explicitly build on Agarwal et al. and Wang et al. rather than overclaiming priority. On weak bases (LLaMA-3.1-8B) the effect shrinks, which is a useful honesty check.\n\nI think the underlying effect is likely real — entropy minimization on high-variance prompts does reshape the distribution and can improve benchmark scores. But the paper's central framing is overstated. The fix is not impossible: rerun the main table with a randomly chosen prompt, or report both; then the 'one-shot' claim becomes legitimate and the supervised-selection variant becomes a clear ablation.\n\nMy recommendation: this deserves a serious referee — the empirical phenomenon is interesting and the analysis is broad — but not as-is. The authors should be required to separate supervised prompt selection from the unsupervised training step, report seed distributions rather than peaks, and soften the abstract. A desk reject would be defensible given the contradiction; a conditional accept after major revision is more useful.","headline":"The 13,440-run study is real and the EM effect is plausible, but the 'unlabeled one-shot' claim is contradicted by the label-based data selection in Section 2.2.","tokens_in":12749,"tokens_out":1976,"would_cite":false,"duration_ms":20665,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that minimizing a language model's token-level entropy on a single unlabeled prompt for ten optimization steps improves reasoning performance as much as, or more than, rule-based reinforcement learning trained on…","keywords":["entropy minimization","post-training","reinforcement learning","large language models","token reranking","logits shift","variance-based data selection","math reasoning"],"falsifier":"Take a prompt whose correct answer is a rare, low-probability token but whose incorrect answer is high-confidence, run one-shot EM on that prompt, and check whether held-out accuracy falls instead of rises; this would directly test whether low entropy is actually correlated with correctness. A simpler version is to correlate, across many prompts, the model's entropy on its own final answer with whether that answer is correct, and show whether low-entropy wrong answers occur often enough to break the method.","tokens_in":1592,"feed_emoji":"⚡","tokens_out":1875,"duration_ms":85501,"temperature":0.7,"pith_summary":"Minimizing the token-level entropy of a language model's own predictions on a single unlabeled prompt, for ten optimization steps, is claimed to improve reasoning performance as much as or more than rule-based reinforcement learning trained on thousands of labeled examples with hand-designed rewards. The authors trained 13,440 models to show the effect holds across seeds and report large accuracy jumps on math, logic, and code benchmarks. They argue that entropy minimization and RL both work by 'token reranking' that reshapes the model's output distribution, but EM shifts the logit distribution rightward (more confidence in high-probability tokens) while RL shifts it leftward. They conclude that EM is a distribution-shaping tool rather than a learning method, so it can act as a nearly free post-training boost. If this claim is right, much of the data collection and reward engineering behind RL post-training becomes unnecessary for improving reasoning.","feed_headline":"One unlabeled prompt plus 10 steps matches thousands of RL samples","feed_subtitle":"Entropy minimization on a single unlabeled prompt rivals rule-based RL on math and code reasoning benchmarks.","key_machinery":"The central object is the one-shot entropy minimization loss, the average token-level entropy over generated positions: $\\mathcal{L}_{EM}(x;\\theta)=\\frac{1}{|I|}\\sum_{t\\in I} H_t$, with $H_t=-\\sum_{v\\in V} p_\\theta(v|y_{<t},x)\\log p_\\theta(v|y_{<t},x)$. This loss is fully differentiable and needs no reward model, labels, or value baseline. The second load-bearing piece is variance-based data selection: from an unlabeled pool, pick the single prompt $x^*$ that maximizes the sample variance of pass@k correctness across multiple sampled responses, on the grounds that such prompts sit at the model's decision boundary and produce the largest entropy gradients. Together these turn entropy minimization into a ten-step, one-example procedure that the paper claims rivals RL.","core_discovery":"The central claim is that entropy minimization is a fully unsupervised post-training method that rivals or surpasses rule-based RL. The loss is the average conditional entropy over the generated tokens, and minimizing it makes the model more confident in its own token predictions. With a single unlabeled prompt chosen for high variance in the model's pass@k correctness, ten steps at learning rate 2e-5 and temperature 0.5 raised a 7-billion-parameter math base model from 53.0 to 78.8 on MATH500 and produced comparable jumps on other math benchmarks, a logic benchmark, and a code benchmark. The paper also reports that EM's training loss keeps decreasing past step 10 while benchmark scores decline, an 'overconfidence' effect, and that EM-trained models have strongly right-skewed logits whereas RL-trained models have left-skewed logits; it frames EM as confidence shaping that concentrates probability mass on high-probability reasoning paths, and finds that EM before RL helps while EM after RL hurts.","pith_inferences":["The paper does not independently test whether low-entropy predictions tend to be correct; if confidently wrong answers are common, one-shot EM could reinforce errors, and the observed gains would depend on the luck of the chosen prompt.","Because EM shifts logits rightward and prefers greedy decoding, its gains may be partly reproducible by temperature scaling alone; comparing one-shot EM against inference-time temperature reduction would isolate what the parameter updates actually add.","The paper's claim that EM is distribution shaping, not learning, implies EM cannot add knowledge; it can only expose latent ability, so one-shot EM could serve as a cheap diagnostic of a model's untapped reasoning quality before investing in RL.","Seed-level variance in results (scores can differ by as much as a factor of two across identical settings) means the headline numbers are best-case or seed-lucky points; reporting the full seed distribution would be needed to know how robust the single-prompt result is."],"forward_implications":["One unlabeled prompt and ten optimization steps can substitute for thousands of labeled examples and hand-designed rewards when improving reasoning performance.","Continuing EM beyond roughly ten steps degrades performance even as the loss falls, so early stopping is a built-in requirement, not optional.","EM-trained models perform best at low sampling temperatures (greedy decoding), while RL-trained models improve at higher temperatures, so the two methods have opposite inference-time behavior.","Applying EM before RL gives consistent gains, but applying EM after RL degrades performance, which constrains how post-training methods can be sequenced.","The ceiling of EM's gains is set by the base model's intrinsic reasoning strength, so weak base models gain little."],"supporting_citations":[{"why":"Establishes the one-training-example RL baseline and first observes that entropy loss alone improves reasoning, which this paper extends to a full one-shot method.","marker":"[23]"},{"why":"Prior study of entropy minimization in LLM post-training that the paper positions against and goes beyond.","marker":"[1]"},{"why":"Source of the unlabeled math prompts used for data selection and the single training exemplar.","marker":"[13]"},{"why":"MATH500 benchmark that supplies the headline accuracy numbers for the main result.","marker":"[14]"},{"why":"Rule-based RL baseline whose reported scores the paper compares against.","marker":"[4]"},{"why":"Rule-based RL baseline whose reported scores the paper compares against.","marker":"[28]"},{"why":"Rule-based RL baseline whose reported scores the paper compares against.","marker":"[11]"},{"why":"Rule-based RL baseline and the reference for the claim that both EM and RL unlock latent potential without adding knowledge.","marker":"[15]"}],"fun_headline_variants":["Single prompt entropy drop rivals thousands of RL samples","One unlabeled prompt, 10 steps: EM beats rule-based RL","Entropy min on one prompt outperforms RL on math code","10-step entropy minimization matches RL with no labels","Unsupervised EM: one example beats RL training"],"cache_read_input_tokens":14848,"weakest_assumption_plain":"The method assumes that a model's low-entropy token choices tend to be correct answers, so making the model more confident in its own predictions generally improves accuracy; if a confidently wrong answer has low entropy, EM will lock in that error.","fun_headline_variants_meta":{"raw":{"variants":["Single prompt entropy drop rivals thousands of RL samples","One unlabeled prompt, 10 steps: EM beats rule-based RL","Entropy min on one prompt outperforms RL on math code","10-step entropy minimization matches RL with no labels","Unsupervised EM: one example beats RL training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1258,"prompt_tokens":801,"completion_tokens":457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":417,"completion_tokens_details":{"reasoning_tokens":377}},"tokens_in":417,"tokens_out":457,"duration_ms":4998,"temperature":1.0,"reasoning_tokens":377,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:55:43.585996+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a prompt whose correct answer is a rare, low-probability token but whose incorrect answer is high-confidence, run one-shot EM on that prompt, and check whether held-out accuracy falls instead of rises; this would directly test whether low entropy is actually correlated with correctness. A simpler version is to correlate, across many prompts, the model's entropy on its own final answer with whether that answer is correct, and show whether low-entropy wrong answers occur often enough to break the method.","supporting_citations":[{"cited_title":"Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025","cited_arxiv_id":null,"evidence_quote":"Rule-based RL baseline whose reported scores the paper compares against."}],"review_version":1}