{"id":"cf1a86de-6984-476a-a9d0-1d2465417e4e","arxiv_id":"2603.18444","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"Discounted Beta-Bernoulli reward estimation reduces variance and variance collapse in group RLVR, improving GRPO Acc@8 on reasoning benchmarks at no extra cost.","lead":"Group-based RL for LLM reasoning often wastes samples by estimating rewards from a few rollouts, causing high variance and collapse. This work proposes Discounted Beta-Bernoulli reward estimation that reuses historical reward statistics to stabilize advantages and improve GRPO accuracy without extra compute.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Central claims remain uncheckable: supplied full text is a different paper (SR-Nav), so DBB theory and Acc@8 gains cannot be stress-tested beyond the abstract.","rationale":"The reader correctly flagged that only the abstract is available and that the supplied full text belongs to a different paper, forcing an UNVERDICTED / LOW-confidence stance. No deeper technical soft spot inside DBB can be isolated until the right manuscript is present; manufacturing a critique of equations that are not here would violate good-faith reading. The single load-bearing concern therefore remains the unverified non-stationarity assumption plus the missing empirical support. Once the correct paper is supplied, the concrete test above would settle whether that assumption holds. Until then the reader's verdict should stay UNVERDICTED.","tokens_in":18949,"tokens_out":530,"duration_ms":11940,"concrete_test":"Retrieve the correct PDF of arXiv:2603.18444. Locate the theorem/proposition claiming lower MSE and no estimated-variance collapse for DBB; re-derive the MSE comparison under a simple non-stationary schedule (e.g., two successive Bernoulli means p1 then p2 with discount γ). If the inequality fails for realistic γ and group sizes used in GRPO, or if the Acc@8 tables lack multi-seed error bars that still show positive gains, the central claim weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim (GRPO+DBB Acc@8 gains of 3.22/2.42 ID and 12.49/6.92 OOD on 1.7B/8B, plus lower MSE and theoretical avoidance of estimated variance collapse) rests on two uninspectable pillars: (1) a formal argument that a discounted Beta-Bernoulli posterior over historical binary rewards yields lower MSE than group point estimates under a non-stationary, policy-induced reward distribution, and (2) the experimental tables that report those Acc@8 numbers. The CACHEABLE manuscript is SR-Nav (object-goal navigation, arXiv 2603.18443), not the DBB paper. Consequently neither the bias-variance derivation nor the GRPO ablations, seeds, or error bars can be examined. The reader's weakest assumption—that discounted historical Beta statistics remain informative as the policy changes—is exactly the condition that would have to be verified in the missing theory section; without that text the claim is unsupported rather than refuted.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The abstract claims that group-based RLVR (e.g., GRPO) is sample-inefficient because it uses high-variance point estimates of binary rewards from few rollouts, which can collapse estimated variance and waste generated responses. The authors reframe advantage estimation as finite-sample estimation of a policy-induced reward distribution and propose Discounted Beta-Bernoulli (DBB) estimation, which folds discounted historical reward statistics into a Beta-Bernoulli posterior. They assert that the resulting biased estimator has lower and more stable variance, theoretically avoids estimated variance collapse, and attains lower MSE than point estimation, and that GRPO+DBB yields average Acc@8 gains of 3.22/2.42 (in-distribution) and 12.49/6.92 (out-of-distribution) on 1.7B/8B models with no extra compute or memory. The supplied full manuscript body, however, is an unrelated paper (SR-Nav on zero-shot object-goal navigation), so the DBB theory, algorithms, and experimental tables cannot be inspected.","tokens_in":19159,"tokens_out":877,"duration_ms":69104,"significance":"If the abstract claims hold, the work would be a practically useful and theoretically motivated fix for a known pain point in RLVR post-training of reasoning LLMs: cheap, drop-in reward estimation that improves both ID and especially OOD Acc@8 without extra rollouts, memory, or compute. A clean bias–variance analysis showing that a discounted Beta-Bernoulli posterior avoids variance collapse under non-stationary policy-induced rewards would also be a reusable statistical contribution for group-based RL methods. Those strengths cannot be credited from the materials provided, because the full text does not contain the DBB derivation, proofs, or GRPO experiments.","major_comments":[{"comment":"Manuscript mismatch: the CACHEABLE full text is SR-Nav (object-goal navigation; arXiv-style header 2603.18443), not the DBB/RLVR paper named in the abstract and paper_id 2603.18444. No DBB estimator, no Beta-Bernoulli update, no GRPO advantage formula, and no Acc@8 tables appear in the body. The central claims are therefore unverifiable from the submitted materials.","section":null},{"comment":"Abstract theory claim (bias, stable variance, avoidance of estimated variance collapse, lower MSE vs point estimation) cannot be checked: there is no equation defining the discounted historical sufficient statistics, no statement of the non-stationary reward model, and no theorem or proof sketch. The load-bearing assumption that discounted history remains informative under a changing policy is unsupported in the provided text.","section":null},{"comment":"Abstract experimental claim (Acc@8 +3.22/+2.42 ID and +12.49/+6.92 OOD on 1.7B/8B; six ID and three OOD benchmarks; no extra cost) cannot be audited: no tables, seeds, group sizes, baselines beyond “naive GRPO,” ablations of the discount/prior, or error bars are present. Reported gains must be treated as unconfirmed.","section":null}],"minor_comments":[{"comment":"Even at abstract level, free parameters (historical discount, Beta prior hyperparameters, group size) are not specified; a camera-ready version should state defaults and sensitivity.","section":null},{"comment":"Clarify whether “verifiable rewards” are strictly binary (0/1) so that Beta-Bernoulli is conjugate, or whether multi-valued rewards are reduced to Bernoulli.","section":null}],"recommendation":"uncertain","confidential_remarks":"The review packet is broken: title/abstract describe a cs.LG RLVR estimator paper, while the full manuscript is an unrelated cs.CV navigation paper (SR-Nav). I cannot produce a content-level accept/reject on DBB. Please re-supply the correct PDF/source for 2603.18444 before a technical decision. If the intent was to review SR-Nav, that should be stated explicitly; under the stated paper_id I must mark uncertain."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline first: the packet does not contain the DBB manuscript. The full text is SR-Nav (object-goal navigation). Everything below is abstract-only for 2603.18444, so confidence has to stay low.\n\nWhat looks new from the abstract is a clean statistical reframing of group RLVR. They treat verifiable rewards as Bernoulli draws from a policy-induced distribution and cast advantage computation as finite-sample estimation of that distribution. Discounted Beta-Bernoulli then folds historical reward counts into a conjugate posterior with a discount for non-stationarity. Beta-Bernoulli and discounting are classical; the contribution is applying them as a drop-in replacement for GRPO’s small-group point estimates, targeting high variance, variance collapse, and wasted rollouts. Claimed properties are reduced/stable variance, theoretical avoidance of estimated variance collapse, lower MSE than point estimation despite bias, and Acc@8 gains of about 3.2/2.4 ID and 12.5/6.9 OOD on 1.7B/8B with no extra compute or memory. If those numbers and proofs hold, this is useful practice-level work for GRPO-style pipelines.\n\nSoft spots, in proportion: the load-bearing assumption is that discounted historical Beta statistics remain informative as the policy changes. That is exactly what the missing theory section must justify; without it the claim is unsupported, not refuted. Free parameters (discount, prior α/β, group size) are not characterized in the abstract. No seeds, error bars, or ablations are visible. None of that is a reason to dismiss the idea—only a reason not to trust the deltas yet.\n\nWho it is for: people running verifiable-reward post-training who care about sample efficiency. The abstract is coherent and non-tautological enough that a serious editor should send the real paper to referees rather than desk-reject. I would not redesign a training stack or cite this until the correct PDF, proofs, and tables are in hand. Bring it to reading group only after we have the right manuscript; until then, park it.","headline":"We only have the abstract for DBB; the supplied full text is a different paper (SR-Nav), so the theory and Acc@8 claims cannot be checked.","tokens_in":19850,"tokens_out":531,"would_cite":false,"duration_ms":15359,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A discounted Beta-Bernoulli estimator turns noisy group rewards into stable advantages, making verifiable RL for LLM reasoning far more sample-efficient.","keywords":["reinforcement learning with verifiable rewards","sample efficiency","Beta-Bernoulli estimation","advantage estimation","GRPO","LLM reasoning","variance collapse"],"falsifier":"Run the identical GRPO training loop on the same model and data with and without DBB; if the Acc@8 gap on the six in-distribution and three out-of-distribution reasoning suites disappears (or reverses) while the empirical variance of the advantage estimates does not collapse under the naïve estimator, the central claim fails.","tokens_in":19742,"feed_emoji":"📈","tokens_out":865,"duration_ms":16732,"temperature":0.7,"pith_summary":"Group-based reinforcement learning with verifiable rewards (RLVR) for large language models wastes most of its generated responses because it treats a handful of rollouts as a point estimate of reward, producing high-variance or collapsed advantages. This paper recasts the problem as statistical estimation of a policy-induced reward distribution and introduces Discounted Beta-Bernoulli (DBB) estimation, which folds historical success/failure counts into a discounted Beta posterior. The resulting estimator is biased yet has lower and more stable variance, theoretically avoids estimated-variance collapse, and yields lower mean-squared error than the usual point estimate. When plugged into GRPO it raises Acc@8 by several points on in-distribution reasoning tasks and by larger margins out-of-distribution, all without extra compute or memory. A reader who cares about post-training reasoning models therefore gains a drop-in way to extract more learning signal from the same number of rollouts.","feed_headline":"Discounted Beta-Bernoulli rewards cut RL waste for LLM reasoning","feed_subtitle":"Same rollouts, lower variance, higher Acc@8 in and out of distribution, zero extra cost","key_machinery":"Discounted Beta-Bernoulli (DBB) reward estimation: a biased posterior mean that mixes the current group’s binary rewards with exponentially discounted historical success/failure counts, thereby regularizing the reward distribution estimate used for advantage computation.","core_discovery":"Modeling binary verifiable rewards as draws from a non-stationary Bernoulli whose success probability is induced by the current policy, then estimating that probability with a discounted Beta-Bernoulli posterior that re-uses historical counts, produces advantage estimates whose variance stays controlled, never collapses to zero, and has lower MSE than the empirical mean of a small group of rollouts; substituting these advantages into GRPO therefore improves sample efficiency and final accuracy on reasoning benchmarks.","pith_inferences":["Because the method only needs running success/failure counts, it can be layered on top of any online RL algorithm that already stores recent rewards, not just GRPO.","If the discount factor is annealed more aggressively as the policy stabilizes, the bias–variance trade-off might improve further late in training.","The same Beta-Bernoulli construction could stabilize multi-objective or preference-based RL where reward signals are similarly sparse and binary."],"forward_implications":["Existing group-based RLVR pipelines can replace their point-estimate advantage step with DBB and obtain higher final accuracy for free.","Advantage estimates remain usable even when a group is all-correct or all-incorrect, eliminating the variance-collapse failure mode that wastes whole batches.","Out-of-distribution reasoning gains are larger than in-distribution gains, suggesting the estimator improves generalization of the learned policy.","The same statistical view can be applied to other binary or low-cardinality verifiable reward signals beyond the GRPO setting."],"fun_headline_variants":["Discounted Beta-Bernoulli cuts sample waste in RL for LLM reasoning","Beta-Bernoulli posterior reuses history for lower-variance RLVR advantages","DBB estimation prevents variance collapse in group-based RLVR","Historical counts stabilize rewards for efficient GRPO training","Discounted Beta-Bernoulli yields lower-MSE advantages in RLVR"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"Historical reward counts, even after discounting, remain informative enough about the current policy’s reward distribution that the resulting biased estimator still has lower mean-squared error than an unbiased point estimate of a tiny group.","fun_headline_variants_meta":{"raw":{"variants":["Discounted Beta-Bernoulli cuts sample waste in RL for LLM reasoning","Beta-Bernoulli posterior reuses history for lower-variance RLVR advantages","DBB estimation prevents variance collapse in group-based RLVR","Historical counts stabilize rewards for efficient GRPO training","Discounted Beta-Bernoulli yields lower-MSE advantages in RLVR"]},"model":"grok-4.5","effort":"low","cost_usd":0.00589,"raw_usage":{"total_tokens":1585,"prompt_tokens":809,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":58900000,"prompt_tokens_details":{"text_tokens":809,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":706,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":809,"tokens_out":70,"duration_ms":6331,"temperature":1.0,"reasoning_tokens":706,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T22:35:50.896716+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the identical GRPO training loop on the same model and data with and without DBB; if the Acc@8 gap on the six in-distribution and three out-of-distribution reasoning suites disappears (or reverses) while the empirical variance of the advantage estimates does not collapse under the naïve estimator, the central claim fails.","supporting_citations":[],"review_version":1}