Pith. sign in

REVIEW 3 major objections 7 minor 14 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Two rival models grade each other's reasoning, doubling accuracy on hard math

2026-07-09 02:08 UTC pith:TWMJNYEY

load-bearing objection Agon's competitive cross-model RL paradigm is genuinely novel; the headline 15pp competition delta rests on single training runs with post-hoc cascade selection, but the pattern is internally consistent enough to warrant a serious referee. the 3 major comments →

arxiv 2607.07690 v1 pith:TWMJNYEY submitted 2026-07-08 cs.LG cs.AIcs.CL

Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning

Vladislav Beliaev This is my paper
classification cs.LG cs.AIcs.CL
keywords modelsmodelotherreasoningrivalagondraftsgrpo
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard reinforcement learning for reasoning models rewards only the final answer, never the chain of thought that produced it. On hard problems this creates a perverse incentive: models learn to write longer traces with more hedging and backtracking, buying more chances to stumble onto the right answer rather than learning to think more efficiently per token. The paper proposes a fix called Agon. Instead of one model grading its own output, two distinct models — instantiated as two low-rank adapters over a single frozen base — are trained head-to-head. On each problem, one model drafts a solution and the other reads that solution's summary (with the final answer withheld) and tries to out-solve it. Roles alternate every step so both models learn both roles. The reward is not just correctness but a conversion bonus: solving a problem that your opponent got wrong is worth more than solving one they also got right. This means a reasoning step that the opponent exploits is implicitly penalized, while a step that leads to winning is reinforced — all without any human labels for good reasoning and without a separate learned reward model. The paper tests two claims under a matched generation budget. The exchange claim is that cross-model information exchange (one model reading another's work) beats single-model self-refinement. The competition claim is that adversarial reward (rewarding out-solving the peer) beats cooperative reward (rewarding correctness alone, with the peer only in context). On a hard math benchmark with a 0.6B-parameter model, the full Agon method reaches 61% pass@1 versus 30% for vanilla GRPO, with the cooperative-exchange ablation at 46% and single-model self-refinement at 32%. The ordering replicates on competitive programming and across two other model families. Notably, the trained cascade produces shorter final-stage traces (3.5k versus 8.1k tokens), an emergent byproduct rather than an explicit length penalty.

Core claim

The central mechanism is the conversion bonus: when the challenger model solves a problem the drafter got wrong, it receives a reward of 3 instead of 2 (the base correctness weight). This bonus multiplies the action-dependent correctness term, so unlike a naive margin reward (which subtracts the opponent's outcome and contributes zero expected gradient under group normalization), it shifts the gradient itself toward harder opponent contexts. The within-group variance in opponent difficulty — each challenger rollout sees a different opponent draft — is what makes the competitive term trainable. Without that variance (e.g., a shared opponent draft for the entire group), the bonus normalizes to

What carries the argument

Two LoRA adapters over one frozen base model; draft-and-challenge rollout where the challenger conditions on the drafter's post-reasoning summary (answer withheld); competitive reward = correctness + conversion bonus c(b_i)(1-c(a_i)); group-relative GRPO advantage computed within each adapter's own rollout group; role rotation every optimizer step so both adapters train in both streams; inference deploys as the same two-stage cascade (one drafts, the other answers after reading).

Load-bearing premise

The two LoRA adapters, initialized from the same frozen base with only noise-level differences, are assumed to develop genuinely complementary failure modes during training. The paper acknowledges that complementarity is not measured directly and that the two adapters start nearly identical; if both converge to similar blind spots, the competitive signal reduces to difficulty-weighted reward shaping rather than genuine cross-model grading, and the paper's own experiments do

What would settle it

If the two adapters' error profiles are measured after training and found to be highly correlated (same problems solved, same problems missed), then the conversion bonus is not supplying cross-model grading but merely upweighting hard contexts — an effect achievable without a second model at all.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the competitive signal genuinely grades reasoning quality rather than acting as difficulty-weighted reward shaping, the same mechanism could replace expensive process reward models in domains where verifiers exist but step-level labels do not.
  • The 0.6B model trained with Agon (61%) outperforms a 7x larger 4B model trained with vanilla GRPO (59%), suggesting that improving signal quality per token may be more efficient than scaling parameters for hard-reasoning tasks.
  • The emergent trace shortening (3.5k vs 8.1k tokens without any length penalty) implies that giving a model a candidate solution to critique naturally reduces exploration bloat, which could reduce inference cost as a side benefit of the training method.
  • The method requires only a verifier and two divergent policies, so it extends to any domain with programmatic verification — the paper demonstrates this on competitive programming with unit-test rewards.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces Agon, a competitive cross-model RL method where two LoRA adapters over a shared frozen base are trained head-to-head: one drafts solutions, the other reads the draft's solution summary and is rewarded for out-solving the rival. The conversion bonus reward (correctness plus a bonus for solving where the opponent failed) is derived from first-principles analysis of group-relative normalization. A 2×2 experimental design (competition × information exchange) tests two claims: (1) cross-model information exchange improves over vanilla GRPO, and (2) adversarial reward improves over cooperative reward. On DeepMath-hard with Qwen3-0.6B, Agon achieves 61% pass@1 versus 30% for GRPO, with the exchange claim (46 vs 30) and competition claim (61 vs 46) both supported. The ordering replicates on CodeContests and across model families.

Significance. The core idea of using a competing peer model as an implicit process-level grader during RL is well-motivated and addresses a genuine gap in outcome-only RL. The theoretical analysis of reward forms under group-relative normalization (Section 4.4) is a clean contribution: the proof that a naive margin reward is gradient-free under standardization, while the multiplicative conversion bonus is trainable, is non-obvious and useful. The 2×2 ablation design is thoughtful, and the paper is commendably transparent about its limitations, including the single-run variance issue, the post-hoc cascade direction selection, and the alternative interpretation of the bonus as difficulty-weighted reward shaping. The method runs on a stock GRPO trainer with minimal overhead (~2% parameters for the second adapter), which is practical. The falsifiable prediction that within-group opponent-difficulty variance is necessary for the competitive gradient (confirmed by the shared-opponent ablation at 32 vs 61) is a good example of theory-guided experimentation.

major comments (3)
  1. §5.1, Table 3 and §4.5: The central competition claim (61 vs 46, a 15pp delta) rests on single training runs per condition. The Clopper-Pearson intervals (Table 4, ±5.5pp) cover only binomial sampling over 300 held-out problems, not run-to-run training variance, which the paper acknowledges is unquantified. RL with LoRA on 3,000 problems for one epoch can exhibit substantial seed-to-seed variation. Without at least 2-3 seeds for the two load-bearing conditions (cooperative exchange and Agon), the magnitude of the competition delta is unverifiable. This is the single most important issue: the paper's headline claim is the 15pp gap, and it could be substantially smaller or even reverse under different seeds. The paper is transparent about this, but transparency does not substitute for evidence. At minimum, 2-3 seeds for the cooperative and adversarial conditions on the primary benchmark (Q
  2. §4.5, last paragraph: Both cascade directions (A→B and B→A) are evaluated and the better is reported, selected post hoc on the held-out set. This is a form of optimistic selection on the evaluation set. The gap between the two directions is not reported, so the inflation is unquantified. If the two directions differ by, say, 5-10pp, the reported 61 could be an upper bound rather than a typical value. The paper should report both directions' pass@1 values, and ideally select the direction on a separate validation split rather than the held-out test set. This issue compounds with the single-run variance problem above.
  3. §4.2 and §4.4: The paper's mechanism story (competitive cross-model grading via complementary blind spots) is acknowledged to be indistinguishable from an alternative interpretation: the conversion bonus acts as difficulty-weighted reward shaping that upweights correct completions in hard contexts (§4.4, final paragraph). The paper states: 'Our experiments do not distinguish these two interpretations.' This is honest, but it means the conceptual contribution (rival grading of reasoning) is not empirically distinguished from a simpler reward-shaping account. The shared-opponent ablation (32 vs 61) shows that within-group variance is necessary, but this is consistent with both interpretations. A control that distinguishes them — for example, measuring whether the two adapters develop genuinely different error profiles (complementarity) versus simply benefiting from difficulty-weighted re —
minor comments (7)
  1. §4.4, Eqs. (2)-(3): The format weight λ=0.5 is described as '25% of the correctness term,' but the correctness coefficient is 2, so λ=0.5 is 25% of 2. This is correct but could be stated more clearly to avoid confusion.
  2. Table 3 footnote: The distinction between single-pass and two-pass methods is important but buried in a footnote. Consider making this more prominent, as it affects the fairness of comparisons.
  3. §5.5: The trace length comparison (3.5k vs 8.1k) is per-stage only; the drafter stage length is not reported. The paper acknowledges this, but a full token-cost accounting would strengthen the compute-parity claim.
  4. Figure 6: The example is hand-constructed. While useful for illustration, it would be strengthened by showing a real exchange from the trained models.
  5. §5.1: The drafter standalone accuracy (46) is attributed to hypothesized transfer of challenger-stream skills, but no control isolates this. The paper acknowledges this, but the attribution remains speculative.
  6. Table 5: The Qwen3.5 and Gemma 4 references point to 2026 technical reports; these should be verified as citable sources.
  7. §4.4: The self-citation to Beliaev 2025 (Esperantix) is for future work on latent-space exchange. This is appropriate but should be clearly marked as the author's own prior work.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive report. All three major comments identify real gaps in evidence that we acknowledge. We agree that (1) single-run training variance is the most serious concern, (2) post-hoc cascade direction selection inflates reported numbers, and (3) the mechanism story is not empirically distinguished from reward shaping. We commit to addressing all three in revision: we will add multi-seed runs for the two load-bearing conditions, report both cascade directions with validation-split selection, and add an error-profile complementarity analysis. The theoretical contribution (Section 4.4) and the 2x2 design are not contested.

read point-by-point responses

  1. Referee: Single training runs for the two load-bearing conditions (cooperative exchange and Agon) mean the 15pp competition delta is unverifiable. Clopper-Pearson intervals cover only binomial sampling, not run-to-run training variance. Need 2-3 seeds minimum.

    Authors: The referee is correct that this is the most important issue. The 15pp competition delta rests on single training runs, and our Clopper-Pearson intervals cover only evaluation sampling, not seed-to-seed training variance. We acknowledge this gap explicitly in the manuscript (Section 5.1, Limitations), but the referee is right that transparency does not substitute for evidence. We will run 3 seeds for the cooperative exchange and Agon conditions on Qwen3-0.6B on DeepMath-hard and report mean and standard deviation. If the delta shrinks or becomes unreliable under multi-seed measurement, we will report that honestly and adjust the claim accordingly. We note that the training curves in Figure 7b show consistent separation between cooperative and adversarial conditions across evaluation checkpoints within the single run, which provides some evidence the effect is not purely seed noise, but this is not a substitute for multi-seed confirmation. revision: yes

  2. Referee: Both cascade directions evaluated, better reported post hoc on held-out set. Optimistic selection on evaluation set. Gap between directions not reported. Should report both directions and select on separate validation split.

    Authors: The referee is correct on both points. Selecting the better cascade direction on the held-out test set is optimistic selection, and not reporting the gap between directions obscures the magnitude of inflation. We will: (1) report both cascade directions (A->B and B->A) pass@1 values for all pair-trained conditions, (2) split the current 300-problem held-out set into a validation portion (for direction selection) and a test portion (for final reporting), and (3) report the gap between directions so readers can judge the inflation. We expect the gap to be modest given that role rotation trains both adapters in both roles, but this is currently an expectation, not a measured fact. This change compounds with the multi-seed runs above: final reported numbers will be mean over seeds, with direction selected per-seed on the validation split. revision: yes

  3. Referee: The mechanism story (competitive cross-model grading via complementary blind spots) is indistinguishable from difficulty-weighted reward shaping. The shared-opponent ablation (32 vs 61) is consistent with both interpretations. Need a control that distinguishes them, e.g., measuring whether adapters develop genuinely different error profiles.

    Authors: The referee correctly identifies that the paper states it cannot distinguish the two interpretations and that the shared-opponent ablation is consistent with both. We agree this weakens the conceptual contribution. We will add an analysis measuring whether the two trained adapters develop genuinely different error profiles: specifically, we will compute per-problem correctness correlations between the two adapters' standalone outputs on the held-out set and compare to the correlation between two independently seeded GRPO runs. If the adapters have lower error correlation than independently trained single models, this supports the complementary-blind-spots mechanism; if correlations are similar, the reward-shaping interpretation is favored. We want to be honest about expectations: our instantiation starts from near-identical adapters (differing only by noise-level initialization), so it is possible that complementarity is weak and the reward-shaping account is the primary driver. If the analysis shows this, we will say so and reframe the contribution accordingly. We cannot promise that the experiment will cleanly favor the competitive-grading interpretation; we can promise to run it and report the result. revision: partial

Circularity Check

0 steps flagged

No circularity found: the derivation chain is self-contained, analytically derived, and tested against external benchmarks.

full rationale

The paper's central derivation chain is self-contained. (1) The conversion bonus reward (Eq. 3) is derived from first-principles analysis of group-relative normalization in Section 4.4: the paper shows mathematically that a naïve margin reward c(b_i)−c(a_i) reduces to the cooperative gradient because c(a_i) is action-independent (acts as a per-sample baseline), while the conversion bonus c(b_i)(1−c(a_i)) modulates the action-dependent correctness weight by opponent difficulty. This is an analytical argument, not a fit to data. (2) The prediction that the 'competition without exchange' cell is inert by construction is derived mathematically (group-constant c(a) normalizes away under GRPO standardization) and presented transparently as a consistency check, not as a novel prediction: 'this cell is theoretically equivalent to vanilla GRPO and serves as a consistency check rather than a true isolation of competition.' The empirical result (32 vs 30) confirms the theory. (3) The margin-vs-conversion-bonus comparison (Table 7: 49 vs 61) is a theoretical prediction (margin should behave like cooperative) confirmed by experiment. (4) The sole self-citation (Beliaev 2025, 'Esperantix') appears only in the future-work discussion of latent-space exchange (Sections 2, 6) and is explicitly not load-bearing for any current claim. (5) All headline claims are evaluated on external benchmarks (DeepMath-hard, CodeContests, GSM8K, MATH-500) with standard verifiers. The paper's acknowledged limitations—single training runs, post-hoc cascade direction selection, and the indistinguishability of 'competitive pressure' from 'difficulty-weighted reward shaping'—are statistical reliability and mechanism-interpretation concerns, not circularity. No step in the derivation chain reduces to its inputs by definition or by self-citation.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

No new entities are postulated. The method uses existing components: LoRA adapters, GRPO, standard verifiers, and text-based communication between models.

free parameters (5)
  • λ (format weight) = 0.5
    Weight on the format reward term φ. Standard choice, identical across all ablation cells so it introduces no bias between conditions.
  • γ (length tiebreak weight) = 0.5
    Used only in the auxiliary density study (Section 5.6), not in the headline recipe. Controls trace length compression.
  • N (group size) = 8
    GRPO group size for both drafter and challenger streams. Standard value.
  • learning rate = 5e-5
    Linear decay with 10% warmup. Standard for LoRA fine-tuning.
  • LoRA rank = 16
    Rank for both adapters. Standard choice.
axioms (4)
  • standard math GRPO group-relative normalization provides a valid policy gradient signal
    The entire method builds on GRPO (Equation 1). This is standard in the literature (Shao et al., 2024).
  • domain assumption The verifier r(x,y) ∈ {0,1} reliably checks correctness
    Used throughout for both math (DeepMath) and code (CodeContests unit tests). Standard for verifiable-reward RL.
  • ad hoc to paper Two LoRA adapters over a shared base develop sufficiently different blind spots during training
    Section 4.2: divergence is maintained heuristically via different initialization and rotated update streams. The paper states 'different blind spots are therefore a hypothesis about the trained pair, not a measured property.' This is the structural premise that makes cross-model grading useful.
  • domain assumption The DeepMath-hard held-out split (difficulty 8, binary-answer removed, 300 problems) is representative
    Section 5 Setup. The paper notes 'reported magnitudes come from a hard, decontaminated subset and should not be read as DeepMath-wide pass@1.'

pith-pipeline@v1.1.0-glm · 17090 in / 4132 out tokens · 310311 ms · 2026-07-09T02:08:08.582164+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning from verifiable rewards (e.g. GRPO) is the engine behind today's reasoning models, yet it grades only the final answer. On hard problems this trains models to write more rather than to think better, since the trace itself is never graded and no label for good thinking exists. We introduce Agon, which makes two competing models each other's graders. Both attempt the same problem; in alternating roles, one drafts a solution and the other reads it while solving, and each is rewarded for out-solving the other. To win, a model must out-reason a rival that has seen its work, so reasoning is judged implicitly during training, with no process labels and no reward model. Because both models are optimized, each faces a progressively stronger rival, which single-model RL cannot provide. The two need only be comparably strong and behaviorally different. At inference the pair deploys as it trains, a two-stage cascade in which one model drafts and the other answers after reading the draft. On the hard split of DeepMath with Qwen3, this doubles GRPO's pass@1, roughly eight times the gain of an untrained Mixture-of-Agents pass over the same base. The ordering replicates on competitive-programming code and across model families (Qwen3.5, Gemma 4). For now the models talk in text; the next step is to let them reason together in latent space.

Figures

Figures reproduced from arXiv: 2607.07690 by Vladislav Beliaev.

Figure 1
Figure 1. Figure 1: From answer-grading to rival-grading. On a hard-math held-out set, vanilla GRPO lifts a Qwen3 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: One Agon step. The drafter A produces N solutions from the plain prompt and receives a vanilla GRPO update on them (its standalone stream); the challenger B reads each opponent’s post-reasoning solution summary in-context (final answer withheld), produces one paired rollout per opponent (N total), and receives the competitive reward, which uses the paired draft’s outcome for the conversion bonus. Both adap… view at source ↗
Figure 3
Figure 3. Figure 3: Why a second model. Left: a single policy optimized on its own rollouts re-audits with the same biases that caused its errors, so self-play saturates [Huang et al., 2024]. Right: Agon pairs two divergent policies; each grades the other with different blind spots, each is rewarded for winning, and because both are optimized, the grader improves with the graded. in behavior, and we make the interaction compe… view at source ↗
Figure 4
Figure 4. Figure 4: Designed divergence (schematic of the target regime, not a measurement). Two copies of one model [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Peek and outscore. A conceptual view aggregated over one rotation cycle: within a single step [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Illustrative exchange (hand-constructed for clarity). The drafter drops a sign, distrusts its own [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: (a) Agon sits in the top-left, higher accuracy at shorter traces (lengths are final-stage comple￾tions), while GRPO buys length for little accuracy and the untrained MoA adds only a small bump. The arrow is illustrative. (b) Held-out pass@1 at evaluation checkpoints over training. Each point is a single sampled evaluation of a single training run (binomial noise ∼ ± 5.5 pp at 300 problems); curves are not … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages · 13 internal anchors

  1. [1]

    DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning

    Z. He et al. DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset. arXiv:2504.11456,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning

    P. Aggarwal, S. Welleck. L1: Controlling How Long a Reasoning Model Thinks with Reinforcement Learning. arXiv:2503.04697,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Let's Verify Step by Step

    H. Lightman et al. Let’s Verify Step by Step. arXiv:2305.20050,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Mixture-of-Agents Enhances Large Language Model Capabilities

    J. Wang et al. Mixture-of-Agents Enhances Large Language Model Capabilities. arXiv:2406.04692,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    W. Li, Y. Lin, M. Xia, C. Jin. Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial? arXiv:2502.00674,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs

    X. Chen et al. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs. arXiv:2412.21187,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    14 Q. Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Absolute Zero: Reinforced Self-play Reasoning with Zero Data

    A. Zhao et al. Absolute Zero: Reinforced Self-Play Reasoning with Zero Data. arXiv:2505.03335,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    R-Zero: Self-Evolving Reasoning LLM from Zero Data

    C. Huang et al. R-Zero: Self-Evolving Reasoning LLM from Zero Data. arXiv:2508.05004,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Improving Factuality and Reasoning in Language Models through Multiagent Debate

    Y. Du et al. Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv:2305.14325,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    AI safety via debate

    G. Irving, P. Christiano, D. Amodei. AI Safety via Debate. arXiv:1805.00899,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    J. H. Kirchner et al. Prover-Verifier Games Improve Legibility of LLM Outputs. arXiv:2407.13692,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    The 2026 AI Index Report, Chapter 2: Technical Performance

    Stanford HAI. The 2026 AI Index Report, Chapter 2: Technical Performance. Stanford University,

    work page 2026

  14. [14]

    Training Verifiers to Solve Math Word Problems

    K. Cobbe et al. Training Verifiers to Solve Math Word Problems (GSM8K). arXiv:2110.14168,

    work page internal anchor Pith review Pith/arXiv arXiv