{"id":"a56afbe9-347f-440e-b29d-ad57eaea9cc0","arxiv_id":"2608.09324","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"CoRE replaces majority-vote rewards in test-time RL with a graph-based consensus extracted by replicator dynamics, improving accuracy and convergence speed on math and science benchmarks.","lead":"Test-time reinforcement learning for language models usually rewards responses that match the majority answer among a handful of samples, which can ignore correct minority answers. This paper replaces the vote with a graph-based consensus that grades each response by how strongly it is supported by its peers, improving accuracy and cutting training time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 3's multiplicative threshold reduction is unproven: Eq. (7) drops the within-class variance term from Eq. (6), and a constructed 8-vs-24 case reverses the predicted recovery.","rationale":"The reader's weakest assumption was the empirical confidence ordering (bar_w_* > bar_w_w). My concern is more specific and more damaging: even when that ordering holds, Proposition 3's claimed multiplicative threshold is not derived correctly. Eq. (6) is the quadratic score at the uniform vector over a clique, but with nonconstant confidence weights the within-class replicator dynamics do not keep the conditional distribution uniform, so the score entering the threshold is not Eq. (6). Moreover, substituting Eq. (6) into the extraction inequality yields the exact threshold with -nu_bar_S/|S| terms, and Eq. (7) drops them. The dropped terms are largest for small n_*, exactly the correct-minority regime CoRE targets, and they depend on variance rather than only on the mean ordering. My 8-vs-24 example has bar_w_* > bar_w_w, yet the exact threshold is 3.64 while Eq. (7) gives 3.16 and the uncalibrated threshold is 3.29, so confidence calibration can flip a recoverable case into an unrecoverable one. This does not disprove the empirical gains, and the Lemmas 1-3 and the voting special case appear sound, but the headline theoretical guarantee is false as stated and the post-hoc recovery-region analysis in Sec. 6 uses the incorrect formula. The proper response is conditional acceptance requiring either a corrected variance-aware threshold, a restricted Proposition 3 for constant within-class weights, or empirical recovery-region evidence computed with the exact score. I set verdict_should_be to CONDITIONAL to reflect this required correction while acknowledging that the empirical study still merits scrutiny rather than outright rejection.","tokens_in":19301,"tokens_out":23708,"duration_ms":255748,"concrete_test":"Run the exact two-class replicator dynamics (Eq. 3) for the model of Props. 2-3 with n_*=8, m=24, alpha=0, c_*=3.3 c_w, correct weights {1 x4, 0.0004 x4}, and wrong weights {0.25 x24}. Eq. (7) predicts the correct class is extracted; the exact uniform-score threshold (~3.64) predicts the wrong class. Sweep c_*/c_w across the uncalibrated threshold 3.29 and the exact threshold 3.64, and record which class the converged x* supports. Then repeat with the -nu_bar_S/|S| terms included and excluded, and with nonuniform within-class weights, to see whether the actual extraction boundary follows Eq. (7), the exact uniform-score formula, or the full within-class replicator equilibrium. If it does not follow Eq. (7), re-derive the recovery-region boundary used in Sec. 6 and re-check whether the measured cells in Fig. 4 remain on the correct side of the corrected boundary.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Proposition 3's central claim, that confidence calibration multiplicatively lowers the extraction threshold whenever correct clusters are more confident, does not follow from the proof. Two gaps. (i) The exact identity (6) evaluates the internal score at the uniform distribution on clique S, but with heterogeneous weights w_i the uniform distribution is not a fixed point of the within-class replicator flow in Lemma 5, Eq. (8). The score that actually enters the mass-competition threshold of Prop. 2 is the limit of that flow, not the uniform-vector value in Eq. (6). (ii) Even taking Eq. (6) as the score, substituting it into n_* mu'_* > m mu'_w gives the exact threshold c_*/c_w > [m mu_bar_w^2 - nu_bar_w]/[n_* mu_bar_*^2 - nu_bar_*], not Eq. (7). Eq. (7) drops the -nu_bar_S/|S| terms, which are largest for the small correct-minority cliques the paper targets and depend on within-class confidence variance, not just means. Concrete failure in the alpha=0 setting of Props. 2-3: take n_*=8, m=24, c_*=3.3 c_w; correct-clique weights are four at 1 and four at about 0.0004, so mu_bar_*^2=0.26 > 0.25=mu_bar_w^2, while wrong-clique weights are constant 0.25. Then Prop. 2's uncalibrated threshold is 23/7=3.29 and Eq. (7) gives 3.16, so both predict correct recovery; the exact uniform-score threshold is about 3.64, so the wrong plurality actually wins. The claimed exp(-delta/tau) reduction and the '8-vs-24 threshold falls below c_w' prediction are therefore not guaranteed, and the recovery-region map in Fig. 4, fitted with Eq. (7), can mislabel cells. This is an internal mathematical gap, not a disagreement with consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoRE, a test-time reinforcement learning reward that replaces majority voting over N rollouts with dominant-set extraction on a graph whose edges combine answer agreement, lexical reasoning similarity, and generation confidence. The method produces a refined pseudo-label, graded per-rollout rewards, and a question-level cohesiveness gate, and plugs into GRPO. The theoretical section claims that majority voting is a special case (Prop. 1), gives a threshold for a correct minority to overturn a wrong plurality under two-class uniform-clique assumptions (Prop. 2), and claims that confidence calibration multiplicatively lowers that threshold (Prop. 3). The evaluation covers seven backbones and five benchmarks (42 model-benchmark cells, three seeds each) and reports average gains of +21.7 points over the untrained base versus +20.4 for majority-vote TTRL, plus faster convergence to the vote's plateau accuracy.","tokens_in":19573,"tokens_out":18260,"duration_ms":178535,"significance":"If the main claims held, CoRE would be a useful and well-motivated contribution: it upgrades the consensus function at no extra rollout cost, includes a clean reduction to voting at kappa=1, and provides an extensive benchmark comparison with offline pseudo-label evidence that the gains are not purely optimization noise. The monotone ascent and constant-shift lemmas are sound, and the authors are explicit about competence-floor and saturation regimes. However, the headline theoretical claim, Proposition 3 and the associated exponential threshold reduction, is not established, and the counterexample in this report shows it can fail under the paper's own confidence premise. The empirical method may still be valuable, but the paper currently overclaims its theoretical support.","major_comments":[{"comment":"Proposition 3 as stated is not established. The proof evaluates the calibrated quadratic form at the uniform vector on S, but with heterogeneous w_i the uniform vector is not a fixed point of the within-class replicator flow (Lemma 5, Eq. (8)); the score that enters the mass-competition threshold of Prop. 2 is the limit of that flow, not the uniform-vector value in Eq. (6). Even taking Eq. (6) as the score, substituting it into n_* mu'_* > m mu'_w gives the exact threshold c_*/c_w > [m bar_mu_w^2 - bar_nu_w] / [n_* bar_mu_*^2 - bar_nu_*], not Eq. (7), which drops the -bar_nu_S/|S| terms. Concrete counterexample in the alpha=0 setting of Props. 2-3: n_*=8, m=24, c_*=3.3 c_w; correct-clique weights are four at 1 and four at about 0.0004, wrong-clique weights constant 0.25. Then bar_mu_*^2 = 0.2601 > 0.25 = bar_mu_w^2, so the paper's confidence premise holds; Prop. 2's threshold is 23/7 ≈ 3.29 and Eq. (7) gives ≈ 3.16, both predicting recovery, but the exact inequality requires c_*/c_w > ≈ 3.64 (and if the flow concentrates on the four high-confidence nodes the required ratio is even larger, ≈ 3.83). The wrong plurality wins, so the claimed e^{-delta/tau} multiplicative reduction and the '8-vs-24 threshold falls below c_w' statement in §3.3 are not guaranteed. This directly contradicts the abstract's claim that confidence calibration provably lowers the threshold multiplicatively.","section":"Appendix G / §3.3, Eqs. (6)–(7)"},{"comment":"The recovery-region analysis applies the two-class, alpha=0 threshold of Prop. 2 to experiments that use alpha=0.1 and many answer classes per question. The only support for this transfer is the sentence in Appendix F stating that the threshold was 'verified to four decimal places on the exact pipeline'; no verification details, diagnostics, or code are provided. Because the headline claim that CoRE's gains and failures follow the predicted regimes depends on this transfer, the paper should either provide the numerical verification or a robustness bound, or explicitly re-label Fig. 4 and the surrounding discussion as heuristic.","section":"§6 / Fig. 4"}],"minor_comments":[{"comment":"The sentence 'CoRE improves the no-RL base by +25.0 points on average' conflicts with Table 1's reported CoRE mean Δ of +21.6; please correct the number or clarify which subset it refers to.","section":"§5 / Table 1"},{"comment":"The claim that CoRE is the strongest RL method in every model family should be qualified: in several individual cells CISC or Majority is higher (e.g., Qwen2.5-Math-7B on MATH-500, Qwen3-8B on MATH-500, Llama-3.1-8B on AIME), so 'best arm on every model' overstates the tables.","section":"§5"},{"comment":"The reproducibility claim cannot be checked because the public repository is listed as 'TBA'; please provide the actual URL or a statement of when it will be available.","section":"Appendix A"},{"comment":"The description of CISC as 'pre-registered' is asserted without evidence; either provide the registration document or soften the wording to 'fixed in advance'.","section":"§2"},{"comment":"Example A includes a diagram that is omitted from the manuscript, making the geometry question impossible to verify; please include the diagram or remove the example.","section":"Appendix H"}],"recommendation":"major_revision","confidential_remarks":"The central problem is the false Proposition 3. The authors should either correct the theorem (for example, by stating the exact threshold with the variance terms and the conditions under which confidence actually lowers it) or downgrade the claim to a heuristic, and then adjust the abstract, Section 3.3, Section 6, and the conclusion accordingly. The empirical comparison is extensive and the method is plausible, so I do not think rejection is warranted, but the theoretical overclaim is central to the paper's framing. Code availability is also currently TBA, which should be resolved before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CoRE is a clean idea with a real empirical payoff, but its headline theoretical result — that confidence calibration multiplicatively lowers the recovery threshold — does not survive close reading. Proposition 3 says the internal score obeys the exact identity (6), which is true for the uniform vector, but then (7) drops the finite-size variance term. The stress-test counterexample (8 vs 24, with heterogeneous correct-clique weights) is correct: the exact threshold from (6) is about 3.64, not 3.16, so the predicted recovery flips. Also, Lemma 5 shows the within-class replicator flow does not stay at the uniform distribution when weights are heterogeneous, so using the uniform-vector score in the threshold is unjustified. This is a genuine gap in the proof, not a disagreement with consensus.\n\nWhat is actually new: applying dominant-set extraction via replicator dynamics to reward construction in test-time RL, with a graph that combines answer agreement, reasoning similarity, and confidence. That is a sensible contribution, and the empirical evaluation is broad: 42 model–benchmark cells, three seeds each, with consistent gains over majority-vote TTRL and the ablations. The comparison against CISC and EC is well-designed, and the sample-efficiency figure is a nice bonus. The paper is honest about its limitations, including the confidence-ordering assumption.\n\nThe soft spots beyond the proof gap: code is TBA, so nothing is reproducible yet. The headline +21.7 vs +20.4 is not tested for significance; only one subset gets proper tests. The 'operating regimes' analysis in §6 is post hoc, and the recovery-region map is fitted to the same data it is supposed to explain. These are addressable, but they should be fixed before the paper is relied on.\n\nBottom line: the method deserves a serious referee, and the empirical claims are likely to hold up in broad strokes, but the theory as stated is not established. A revision that either proves the threshold with the variance terms or honestly re-scopes Prop. 3 to an approximation would be much stronger.","headline":"A promising consensus-reward method with broad empirical results, but the headline theory of confidence lowering the recovery threshold does not hold as proven.","tokens_in":20313,"tokens_out":3875,"would_cite":false,"duration_ms":34959,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CoRE replaces majority voting in test-time reinforcement learning with an equilibrium-based consensus over a graph of sampled roll-outs, yielding a refined pseudo-label, graded rewards, and a cohesiveness gate.","keywords":["test-time reinforcement learning","consensus rewards","dominant sets","replicator dynamics","majority voting","confidence calibration","self-supervised reward","language model reasoning"],"falsifier":"Take a benchmark where the model's wrong answers are engineered to be more confident than its correct ones (for example, by adding confident-looking plausible distractors) and run CoRE and majority-vote TTRL under the same GRPO loop; if the mean-confidence gap $\\bar w_w - \\bar w_*$ is consistently positive, the theory predicts CoRE's accuracy should fall below the vote's, and the measured gap should correlate with the size of the loss.","tokens_in":18930,"feed_emoji":"🗳️","tokens_out":9952,"duration_ms":88999,"temperature":0.7,"pith_summary":"Test-time reinforcement learning has to invent its own reward on unlabeled questions, and the standard choice—majority voting over sampled roll-outs—throws away the model's confidence and the structure of its reasoning, rewarding every majority-matching roll-out identically and discarding correct minorities. CoRE replaces the vote with dominant-set extraction on a graph whose edges combine answer agreement, reasoning similarity, and generation confidence, producing a refined pseudo-label, a graded per-roll-out reward, and a per-question cohesiveness gate. The paper claims this strictly generalizes majority voting, gives a sharp threshold for when a correct minority can overturn a larger wrong plurality, and shows that confidence calibration lowers that threshold multiplicatively. Across seven backbones and five benchmarks, CoRE improves the untrained base by +21.7 points on average versus +20.4 for majority-vote TTRL, and reaches the vote's plateau accuracy in 54–70% fewer steps.","feed_headline":"Majority vote is a special case: graph equilibrium lifts test-time RL","feed_subtitle":"Same roll-outs, better supervision: 54-70% fewer steps to vote-level accuracy","key_machinery":"The machinery is the roll-out graph plus dominant-set extraction. Roll-outs are nodes; an edge exists only between roll-outs with the same extracted answer, weighted by a reasoning-similarity kernel (TF-IDF n-gram cosine) with a floor $\\kappa$, and the affinity matrix is then confidence-calibrated by $A' = D^{1/2} A D^{1/2}$, where node weights are exponentials of relative mean token log-probability. A dominant set—the weighted-graph generalization of a maximal clique that appears as the support of a strict local maximizer of $x^\\top A' x$ on the simplex—is found by iterating replicator dynamics from the simplex barycenter, so initial mass is proportional to answer-class size. The equilibrium mass $x^*$ gives the pseudo-label (answer with greatest mass), graded rewards $R_i = (A x^*)_i / \\max_j (A x^*)_j$, and a cohesiveness gate $w_q = x^{*\\top} A x^*$ that scales the GRPO loss. Block-diagonality of the affinity over answer classes lets the analysis decompose the dynamics into within-class refinement and between-class mass competition, which is what yields the extraction threshold.","core_discovery":"The central claim is that consensus should be found, not counted: the $N$ roll-outs for a question form a weighted graph, and the equilibrium of replicator dynamics on that graph selects a dominant set of mutually supporting trajectories. With answer-only affinity and uniform confidence, the selected class is exactly the plurality, so majority voting is a special case (Prop. 1). For a correct clique of size $n^*$ and affinity $c^*$ facing a wrong clique of size $m > n^*$ and affinity $c_w$, barycenter-initialized replicator dynamics recover the correct class iff $c^* > \\frac{m-1}{n^*-1} c_w$ (Prop. 2). Confidence calibration rescales each class's internal score by the mean of its square-root weights, multiplying the recovery threshold by the ratio of the wrong-to-correct mean weights; when correct roll-outs are more confident, the threshold drops multiplicatively, turning otherwise infeasible minority recoveries into feasible ones (Prop. 3). The paper tests these predictions across seven backbones and five benchmarks, reporting average gains of +21.7 points over the untrained base versus +20.4 for majority-vote TTRL, wins of up to +7.5 points where agreement is contested, and 54–70% faster convergence to the vote's plateau.","pith_inferences":["If the confidence premise is weak in a target domain, a practitioner could test the sign of $\\bar w_* - \\bar w_w$ on a held-out set before committing to CoRE; this is a direct, cheap check the paper's theory implies.","The same equilibrium-consensus read-out could be reused outside GRPO, for example to build verifier-free rerankers or to select which unlabeled questions are informative enough to train on at all.","Replacing the TF-IDF reasoning kernel with hidden-state similarity might extend CoRE's recovery to semantically fluent but lexically dissimilar wrong answers, a direction the paper flags as future work.","The threshold formula suggests CoRE's gains should transfer to new models and benchmarks only when the measured operating point (size and affinity ratios, confidence gap) lies inside the recovery region; reporting those four quantities alongside accuracy would make the method's claims easier to test."],"forward_implications":["Test-time RL can use CoRE's graded rewards and cohesiveness gate with the same $N=64$ roll-outs and the same GRPO loop, so the better supervision costs no extra sampling.","Where a coherent, confident correct minority exists, CoRE recovers questions the vote gets wrong; where roll-outs are nearly unanimous, CoRE reduces to voting and does not hurt.","The extraction threshold gives a diagnostic for when to trust the pseudo-label: compare $n^*$, $c^*$, $m$, $c_w$, and the confidence gap before adapting on a question.","Because CoRE reaches the vote's plateau in 54–70% fewer steps, it also reduces the compute budget of test-time adaptation.","Confidence-weighted voting alone (CISC) and graph structure alone (EC) each perform near the vote; the multiplicative combination is what creates the recovery regime."],"supporting_citations":[{"why":"Defines test-time reinforcement learning and the majority-vote consensus reward that CoRE is designed to replace; supplies the training-loop baseline.","marker":"Zuo et al., 2026"},{"why":"GRPO group-relative policy optimization is the update rule CoRE integrates with, including the loss the cohesiveness gate scales.","marker":"Shao et al., 2024a"},{"why":"Provides dominant-set theory and the replicator-dynamics extraction used to find the consensus; its Theorem 1 connects fixed-point supports to dominant sets.","marker":"Pavan and Pelillo, 2007"},{"why":"Relates maximum cliques to quadratic optimization over the simplex, the theoretical root of the dominant-set objective.","marker":"Motzkin and Straus, 1965"},{"why":"The Baum–Eagon inequality underpins the monotone ascent lemma that guarantees replicator dynamics converge to dominant sets.","marker":"Baum, 1967"},{"why":"Confidence-weighted self-consistency (CISC) is the pre-registered control arm and the confidence-only ablation against which CoRE's graph is isolated.","marker":"Taubenfeld et al., 2025"},{"why":"Self-consistency majority voting is the aggregation scheme being generalized and the inference-time baseline for consensus.","marker":"Wang et al., 2022"},{"why":"Supports the use of mean token log-probability as the generation-confidence signal that calibrates the affinity edges.","marker":"Kadavath et al., 2022"},{"why":"TF-IDF term weighting underlies the lexical reasoning-similarity kernel that builds the within-answer edges.","marker":"Salton and Buckley, 1988"}],"fun_headline_variants":["Consensus beats counting: graph equilibrium refines test-time RL","Majority vote is a special case: CoRE finds consensus via equilibrium","Graph equilibrium lifts test-time RL beyond majority vote","CoRE: graded consensus rewards improve test-time RL by 21.7 points","Find consensus, don't count it: CoRE rewires test-time RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The recovery threshold in Proposition 3 assumes that correct answer clusters are on average more confident than incorrect clusters; if wrong answers are systematically overconfident, confidence weighting amplifies the wrong cluster and CoRE can underperform majority-vote TTRL.","fun_headline_variants_meta":{"raw":{"variants":["Consensus beats counting: graph equilibrium refines test-time RL","Majority vote is a special case: CoRE finds consensus via equilibrium","Graph equilibrium lifts test-time RL beyond majority vote","CoRE: graded consensus rewards improve test-time RL by 21.7 points","Find consensus, don't count it: CoRE rewires test-time RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000364,"raw_usage":{"total_tokens":2046,"prompt_tokens":1114,"completion_tokens":932,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":730,"completion_tokens_details":{"reasoning_tokens":840}},"tokens_in":730,"tokens_out":932,"duration_ms":8328,"temperature":1.0,"reasoning_tokens":840,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:18:28.757171+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a benchmark where the model's wrong answers are engineered to be more confident than its correct ones (for example, by adding confident-looking plausible distractors) and run CoRE and majority-vote TTRL under the same GRPO loop; if the mean-confidence gap $\\bar w_w - \\bar w_*$ is consistently positive, the theory predicts CoRE's accuracy should fall below the vote's, and the measured gap should correlate with the size of the loss.","supporting_citations":[],"review_version":1}