{"id":"039d5104-d96c-4b1f-bbd1-564e841078e1","arxiv_id":"2508.11800","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"GRPO, a common RL method, induces overconfident probability estimates for stochastic outcomes; removing group standard normalization restores calibration.","lead":"This paper tests whether reinforcement learning methods that work well on math problems also work when outcomes are random, like lab experiments. It reports that one popular method, GRPO, makes AI models overconfident about probabilities, while other methods stay well-calibrated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Causal attribution to group normalization is confounded by effective learning-rate changes; ablation with constant rescaling and a learning-rate control would resolve whether normalization per se causes overconfidence.","rationale":"The reader's weakest assumption already flags confounds in the empirical comparison, but my concern identifies a specific confound: the division by the group standard deviation changes the scale of the advantage, and therefore the effective learning rate. Removing normalization may fix miscalibration simply because updates become smaller, not because of any property of per-group standardization. This is a load-bearing concern because the paper's theoretical explanation is supposed to demonstrate why normalization causes overconfidence; if the effect is merely a scale effect, the mechanism is not unique. The abstract shows the authors performed an ablation (removing normalization), but we do not know whether they controlled for the average update magnitude. The proposed concrete test would settle this: compare per-group normalization, constant rescaling, and a learning-rate-adjusted baseline. If all variants with similar average update magnitude yield similar calibration, then the unique role of per-group normalization is not supported. Since the full text is unavailable, I cannot resolve this from the abstract alone, so the reader's UNVERDICTED verdict remains appropriate. I partially agree with the reader's weakest_assumption because they correctly identify the causal-mechanism assumption and confound risk, but my concern is more specific about the scaling confound.","tokens_in":577,"tokens_out":4430,"duration_ms":50077,"concrete_test":"Run the synthetic binary-outcome task with three GRPO variants: (i) standard group normalization; (ii) advantages divided by a constant c equal to the average across groups of the standard deviation (a global running estimate), removing per-group noise; (iii) standard normalization but with the learning rate scaled down so that the average update magnitude matches variant (ii). If (ii) and (iii) both fix calibration, the effect is not specific to per-group normalization; if only (ii) fixes calibration, per-group estimation matters. Report reliability diagrams with confidence intervals for each variant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that GRPO's group standard normalization causes overconfident probability predictions and that removing it fixes miscalibration. The load-bearing step is the causal mechanism. A key confound is that group standard normalization scales the advantage estimates: dividing by the group reward standard deviation changes the magnitude of policy updates relative to PPO/RLOO baselines. Removing normalization therefore also changes the effective step size. A smaller learning rate or a constant rescaling of advantages can trivially reduce overconfidence by slowing convergence, without any structural effect attributable to the per-group normalization itself. The proposed theoretical explanation must show that per-group noise in the standard-deviation estimate—not just its average scale—modulates calibration. If overconfidence is driven instead by a too-large overall update magnitude, the paper's mechanism is not uniquely identified. Because the full text is unavailable, this confound is underdetermined. A controlled ablation is needed before accepting the normalization-specific causal claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript (arXiv:2508.11800) examines whether current RL methods that work well for deterministic verifiable domains (e.g., mathematics) also work for stochastic verifiable domains such as scientific experiments. Based on apparently synthetic and real biological experiments, it claims that GRPO induces overconfident probability predictions for binary stochastic outcomes, whereas PPO and RLOO remain well-calibrated. The paper further claims that removing group standard normalization from GRPO fixes the miscalibration, and it offers a theoretical explanation for why normalization causes overconfidence. This review is based solely on the abstract, as no full text is available; the abstract itself makes a strong causal claim about the role of a specific algorithmic component.","tokens_in":745,"tokens_out":2419,"duration_ms":24541,"significance":"If the causal claim holds, the paper would provide actionable guidance for using RL to fine-tune language models in stochastic verifiable environments, an important emerging use case. The proposed intervention (removing group standard normalization) is concrete and falsifiable, and the combination of synthetic experiments, real biological experiments, and a theoretical explanation is a strong evidentiary structure in principle. The paper would also contribute to the broader debate about the effects of normalization in policy-gradient methods. However, the strength of the significance depends entirely on the full evidence, which is not available in this abstract-only review; the causal attribution to group standard normalization is exactly the kind of claim that requires careful confounding controls.","major_comments":[{"comment":"The abstract attributes overconfidence specifically to group standard normalization, but it does not address the well-known confound that removing normalization changes the effective scale of the advantage estimates and therefore the effective learning-rate magnitude. If the full text does not include an ablation with constant rescaling of advantages or a matched learning-rate control, the causal mechanism is not uniquely identified. The theoretical explanation must demonstrate that per-group noise in the standard-deviation estimate, not just its average scale, is what degrades calibration.","section":"Abstract (causal claim)"},{"comment":"The abstract reports only qualitative outcomes: GRPO is overconfident, while PPO and RLOO are well-calibrated, and removing normalization fixes the problem. No effect sizes, calibration metrics, confidence intervals, or experimental details are given. The full text must report quantitative calibration error (e.g., expected calibration error or reliability diagrams) before the claim that PPO and RLOO are 'well-calibrated' and GRPO is not can be audited.","section":"Abstract (empirical evidence)"},{"comment":"The abstract states that a theoretical explanation is provided but does not specify the assumptions of that theory. Without knowing the assumptions about group size, reward distributions, or the stochastic outcome model, it is unclear whether the theory genuinely explains the empirical result or simply restates that normalization introduces a bias. The full text should state the model assumptions explicitly and show that the predicted direction of the calibration error matches the experimental data.","section":"Abstract (theoretical explanation)"}],"minor_comments":[{"comment":"The term 'stochastic outcomes' is used without a precise definition; the paper should clarify whether the binary outcomes are Bernoulli draws with fixed but unknown probabilities, or whether the stochasticity arises from the environment or the policy.","section":"Abstract (terminology)"},{"comment":"The phrase 'new evidence against the use of standard normalization in GRPO' is broader than the binary-stochastic-outcome setting described; the paper should explicitly limit the recommendation to that setting unless additional evidence is provided.","section":"Abstract (scope of claim)"}],"recommendation":"uncertain","confidential_remarks":"This is an abstract-only review; the full text is necessary to evaluate the load-bearing causal claim. The skeptic's concern about the effective learning-rate confound is a genuine risk that the full text must address. If the paper includes the recommended ablation (constant rescaling or learning-rate control) and the theory indeed isolates per-group normalization noise, it could be a strong contribution. My uncertainty is due to missing information, not to a demonstrated flaw."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is a concrete, testable observation: GRPO, the default RL method for reasoning LMs, produces overconfident probability predictions when outcomes are stochastic, while PPO and RLOO stay well calibrated. That matters because anyone pushing RL into scientific domains—where outcomes are inherently noisy—needs to know this. The simple fix, dropping group standard normalization, gives practitioners something they can try immediately. Credit where it is due: comparing three methods on synthetic and biological data is the right design, and the theoretical explanation is a useful bonus even if it does not carry the paper alone. If the empirical pattern reproduces, this is a solid within-subfield contribution to the calibration literature.\n\nThe soft spot is the causal attribution. Group standard normalization does not just normalize; it rescales the advantage estimates, so removing it also changes the effective learning rate. A slower effective update can trivially reduce overconfidence without any structural effect from the normalization itself. The paper needs an ablation with constant rescaling or a learning-rate-matched control to show the per-group noise in the normalization, not just its average scale, is what drives the miscalibration. The stress-test note makes this point, and it is a fair one. It is not a fatal flaw—the empirical observation can still stand even if the mechanistic story needs refining—but it is the load-bearing part of the theoretical claim, and the abstract gives no indication that this control was run.\n\nOther than that, my only reservation is that I am reviewing an abstract. No code, no data, no derivations. The claims are specific enough that a full review can settle them quickly. The central phenomenon is plausible and the authors seem to be reporting what they saw rather than fitting a narrative.\n\nWho is this for? Anyone working on RL for reasoning models, especially in scientific or other stochastic domains. I would bring it to a reading group and cite it as related work. A serious referee should see the full version, because the claim is important and the experimental design is auditable. The main thing I would ask the reviewers to push on is the normalization versus learning-rate control.","headline":"Plausible and important claim that GRPO makes probabilistic predictions overconfident on stochastic outcomes; the causal story about normalization needs a controlled ablation before it fully lands.","tokens_in":1178,"tokens_out":1319,"would_cite":true,"duration_ms":16615,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GRPO causes overconfident probability predictions for stochastic binary outcomes, and removing its group standard normalization fixes the miscalibration.","keywords":["GRPO","reinforcement learning","calibration","overconfidence","stochastic outcomes","language models","group standard normalization","PPO"],"falsifier":"On a synthetic binary-stochastic task, train GRPO with and without group standard normalization under matched hyperparameters and compute calibration error on held-out outcomes; if both variants show the same calibration error, the normalization explanation fails. A complementary check would replace normalization with a fixed scaling factor: if that fixed scaling also causes overconfidence, the mechanism is not group standardization per se.","tokens_in":427,"feed_emoji":"🎯","tokens_out":2258,"duration_ms":25019,"temperature":0.7,"pith_summary":"This paper asks whether reinforcement learning methods that sharpen language models on deterministic tasks like mathematics also work when the verifiable outcome is stochastic, as in scientific experiments. It claims that Group Relative Policy Optimization (GRPO) drives models to report overconfident probabilities for binary stochastic events, while Proximal Policy Optimization (PPO) and REINFORCE Leave-One-Out (RLOO) stay well-calibrated. The authors locate the cause in GRPO's group standard normalization: removing that step restores calibration. If the claim is right, it offers a concrete design fix and a caution that group normalization can be harmful beyond deterministic domains.","feed_headline":"Removing one step fixes GRPO's overconfidence","feed_subtitle":"The popular RL method reports too-extreme odds on random outcomes; without group normalization it recalibrates.","key_machinery":"The central object is GRPO's group standard normalization: a step that centers and scales advantage values by the mean and standard deviation within a group of sampled outputs before the policy update. The paper claims this normalization is the mechanism that induces overconfidence, and that omitting it makes GRPO behave like the calibrated baselines.","core_discovery":"When language models are trained with GRPO on tasks where each output is scored against a stochastic ground truth, the resulting probability estimates are systematically too extreme relative to observed frequencies. The same training setup with PPO or RLOO yields well-calibrated predictions. The paper shows that the overconfidence disappears when group standard normalization is removed from GRPO, and it offers a theoretical explanation for why normalization produces overconfidence.","pith_inferences":["The overconfidence mechanism may generalize beyond binary outcomes to any stochastic reward where group-relative normalization exaggerates small sample differences; this is a testable extension the paper does not pursue.","A natural follow-up experiment would vary group size in GRPO while holding other factors fixed; the theoretical account implies overconfidence should grow as group variance shrinks.","The results raise a broader question about whether standardization steps in other RL variants, even when they do not fully break calibration here, still distort uncertainty in subtler settings such as multi-class or continuous prediction.","For real biological experiments, the finding suggests that calibration error should be reported alongside accuracy whenever RL is used to optimize scientific decision-making."],"forward_implications":["Practitioners who need trustworthy probability estimates from RL-tuned models on stochastic scientific outcomes should avoid group standard normalization in GRPO.","PPO and RLOO are safer defaults for verifiable domains with stochastic outcomes, because they preserve calibration.","Removing normalization is a minimal, targeted change that reportedly restores GRPO's calibration without replacing the whole algorithm.","The theoretical account predicts that other schemes relying on group-relative standardization of advantages should similarly distort probability estimates.","Reinforcement learning can be useful for scientific reasoning tasks only if calibration is explicitly checked and the normalization choice is made with the stochastic nature of the reward in mind."],"supporting_citations":[],"fun_headline_variants":["GRPO's normalization causes overconfidence","Why GRPO overpredicts on random outcomes","Drop normalization to fix GRPO's odds","GRPO's overconfidence traced to normalization","One tweak makes GRPO calibrated"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that group standard normalization is the actual cause of GRPO's overconfidence, rather than some other difference between GRPO and the calibrated baselines in the training setup.","fun_headline_variants_meta":{"raw":{"variants":["GRPO's normalization causes overconfidence","Why GRPO overpredicts on random outcomes","Drop normalization to fix GRPO's odds","GRPO's overconfidence traced to normalization","One tweak makes GRPO calibrated"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000108,"raw_usage":{"total_tokens":959,"prompt_tokens":769,"completion_tokens":190,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":385,"completion_tokens_details":{"reasoning_tokens":125}},"tokens_in":385,"tokens_out":190,"duration_ms":2262,"temperature":1.0,"reasoning_tokens":125,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:26:10.492442+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a synthetic binary-stochastic task, train GRPO with and without group standard normalization under matched hyperparameters and compute calibration error on held-out outcomes; if both variants show the same calibration error, the normalization explanation fails. A complementary check would replace normalization with a fixed scaling factor: if that fixed scaling also causes overconfidence, the mechanism is not group standardization per se.","supporting_citations":[],"review_version":1}