{"id":"ef7fb690-a606-49ff-ab58-68758630a872","arxiv_id":"2607.07690","paper_version":1,"verdict":"CONDITIONAL","confidence":"UNKNOWN","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":5,"one_line_summary":"Training two LoRA adapters competitively against each other, where each reads the other's solution summary and is rewarded for out-solving it, doubles GRPO's pass@1 on hard math while shortening reasoning traces.","lead":"The paper introduces Agon, a method that trains two language models to compete against each other at math and code problems, where each model reads the other's solution and is rewarded for out-solving it. This implicitly grades reasoning quality without needing step-by-step labels, roughly doubling accuracy gains over standard RL training on hard math problems.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The 15pp competition delta (61 vs 46) rests on single training runs with post-hoc cascade direction selection; without multi-seed confirmation the central claim's magnitude is unverifiable.","rationale":"The reader identified the right general area — the paper's central claims rest on uncertain foundations — but framed the weakest assumption as the complementarity/mechanism question (whether the adapters develop different blind spots, or whether the gain is difficulty-weighted reward shaping). I think this is a real but secondary concern: the paper itself acknowledges the two interpretations are indistinguishable in its experiments (Section 4.4), and the method works regardless of which interpretation is correct. The more immediately load-bearing concern is whether the 15pp competition delta is real at all, given that it comes from single training runs with post-hoc cascade direction selection on the evaluation set. If training variance turns out to be large, the mechanism question becomes moot. The reader's rationale does mention single-run variance and post-hoc selection, so we are looking at the same evidence; I would prioritize the reliability concern over the mechanism concern. The paper's transparency about these limitations is commendable and the CONDITIONAL verdict is appropriate. The ablation matrix is well-designed (the shared-opponent inert cell is a nice theoretical check), the reward analysis in Section 4.4 is careful and correct about when the conversion bonus contributes gradient, and the cross-domain/cross-family replications provide some evidence of generality. But none of this substitutes for multi-seed confirmation of the headline delta. I recommend UNCHANGED because the reader already arrived at CONDITIONAL with the right concerns noted; my stress test sharpens the priority but does not change the verdict. The concrete test (multi-seed runs with both cascade directions reported) is the single check that would settle whether the 15pp delta is stable enough to support the paper's claims.","tokens_in":15045,"tokens_out":6534,"duration_ms":437791,"concrete_test":"Run 3 additional seeds (total n=4) of both cooperative and adversarial conditions on Qwen3-0.6B with identical hyperparameters. For each seed, report pass@1 for both cascade directions (A→B and B→A), not just the better one. Compute the mean and standard deviation of the adversarial-vs-cooperative delta across seeds. If the 95% CI of the mean delta excludes zero and the per-direction gap is <5pp, the competition claim is stable. If any seed shows adv ≤ coop, or if the delta's standard deviation exceeds 5pp, the headline magnitude is unreliable and the claim should be softened to directional only.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim is the competition claim: adversarial reward (61) beats cooperative reward (46), a 15pp delta on Qwen3-0.6B. Every number in Table 3 comes from a single training run per condition. The Clopper-Pearson intervals (Table 4, ±5.5pp) cover only binomial sampling over 300 held-out problems, not run-to-run training variance, which the paper acknowledges is unquantified. RL with LoRA on 3,000 problems for one epoch can easily exhibit ±5-10pp seed-to-seed variation. If training variance is at the higher end, the 15pp delta could shrink to a few points or even reverse in some seeds. Additionally, the paper evaluates both cascade directions (A→B and B→A) and reports the better one, selected post hoc on the held-out set (Section 4.5, Limitations). This is a form of optimistic selection on the evaluation set; the gap between directions is not reported, so the inflation is unquantified. The paper is transparent about both issues, but the headline claims — and the CONDITIONAL verdict — rest on this single-run, post-hoc-selected 15pp delta. The reader's weakest_assumption focuses on the mechanism (complementarity vs. difficulty-weighted reward shaping), which the paper itself flags as indistinguishable. I see the more immediately load-bearing concern as the reliability of the delta itself: even if the mechanism is difficulty-weighted reward shaping (a valid and useful signal), we cannot confirm the 15pp gap is stable without multiple seeds. The cross-domain and cross-family replications (Tables 5, 6) provide some evidence against pure noise, but each is also a single run and could share the same systematic bias. The ablation pattern (Table 7: conversion bonus 61 > margin 49 > coop 46) is internally consistent, which strengthens the case, but consistency of pattern across single runs does not substitute for variance estimates on the headline delta.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper introduces Agon, a competitive cross-model RL method where two LoRA adapters over a shared frozen base are trained head-to-head: one drafts solutions, the other reads the draft's solution summary and is rewarded for out-solving the rival. The conversion bonus reward (correctness plus a bonus for solving where the opponent failed) is derived from first-principles analysis of group-relative normalization. A 2×2 experimental design (competition × information exchange) tests two claims: (1) cross-model information exchange improves over vanilla GRPO, and (2) adversarial reward improves over cooperative reward. On DeepMath-hard with Qwen3-0.6B, Agon achieves 61% pass@1 versus 30% for GRPO, with the exchange claim (46 vs 30) and competition claim (61 vs 46) both supported. The ordering replicates on CodeContests and across model families.","tokens_in":15382,"tokens_out":1441,"duration_ms":121026,"significance":"The core idea of using a competing peer model as an implicit process-level grader during RL is well-motivated and addresses a genuine gap in outcome-only RL. The theoretical analysis of reward forms under group-relative normalization (Section 4.4) is a clean contribution: the proof that a naive margin reward is gradient-free under standardization, while the multiplicative conversion bonus is trainable, is non-obvious and useful. The 2×2 ablation design is thoughtful, and the paper is commendably transparent about its limitations, including the single-run variance issue, the post-hoc cascade direction selection, and the alternative interpretation of the bonus as difficulty-weighted reward shaping. The method runs on a stock GRPO trainer with minimal overhead (~2% parameters for the second adapter), which is practical. The falsifiable prediction that within-group opponent-difficulty variance is necessary for the competitive gradient (confirmed by the shared-opponent ablation at 32 vs 61) is a good example of theory-guided experimentation.","major_comments":[{"comment":"§5.1, Table 3 and §4.5: The central competition claim (61 vs 46, a 15pp delta) rests on single training runs per condition. The Clopper-Pearson intervals (Table 4, ±5.5pp) cover only binomial sampling over 300 held-out problems, not run-to-run training variance, which the paper acknowledges is unquantified. RL with LoRA on 3,000 problems for one epoch can exhibit substantial seed-to-seed variation. Without at least 2-3 seeds for the two load-bearing conditions (cooperative exchange and Agon), the magnitude of the competition delta is unverifiable. This is the single most important issue: the paper's headline claim is the 15pp gap, and it could be substantially smaller or even reverse under different seeds. The paper is transparent about this, but transparency does not substitute for evidence. At minimum, 2-3 seeds for the cooperative and adversarial conditions on the primary benchmark (Q","section":null},{"comment":"§4.5, last paragraph: Both cascade directions (A→B and B→A) are evaluated and the better is reported, selected post hoc on the held-out set. This is a form of optimistic selection on the evaluation set. The gap between the two directions is not reported, so the inflation is unquantified. If the two directions differ by, say, 5-10pp, the reported 61 could be an upper bound rather than a typical value. The paper should report both directions' pass@1 values, and ideally select the direction on a separate validation split rather than the held-out test set. This issue compounds with the single-run variance problem above.","section":null},{"comment":"§4.2 and §4.4: The paper's mechanism story (competitive cross-model grading via complementary blind spots) is acknowledged to be indistinguishable from an alternative interpretation: the conversion bonus acts as difficulty-weighted reward shaping that upweights correct completions in hard contexts (§4.4, final paragraph). The paper states: 'Our experiments do not distinguish these two interpretations.' This is honest, but it means the conceptual contribution (rival grading of reasoning) is not empirically distinguished from a simpler reward-shaping account. The shared-opponent ablation (32 vs 61) shows that within-group variance is necessary, but this is consistent with both interpretations. A control that distinguishes them — for example, measuring whether the two adapters develop genuinely different error profiles (complementarity) versus simply benefiting from difficulty-weighted re —","section":null}],"minor_comments":[{"comment":"§4.4, Eqs. (2)-(3): The format weight λ=0.5 is described as '25% of the correctness term,' but the correctness coefficient is 2, so λ=0.5 is 25% of 2. This is correct but could be stated more clearly to avoid confusion.","section":null},{"comment":"Table 3 footnote: The distinction between single-pass and two-pass methods is important but buried in a footnote. Consider making this more prominent, as it affects the fairness of comparisons.","section":null},{"comment":"§5.5: The trace length comparison (3.5k vs 8.1k) is per-stage only; the drafter stage length is not reported. The paper acknowledges this, but a full token-cost accounting would strengthen the compute-parity claim.","section":null},{"comment":"Figure 6: The example is hand-constructed. While useful for illustration, it would be strengthened by showing a real exchange from the trained models.","section":null},{"comment":"§5.1: The drafter standalone accuracy (46) is attributed to hypothesized transfer of challenger-stream skills, but no control isolates this. The paper acknowledges this, but the attribution remains speculative.","section":null},{"comment":"Table 5: The Qwen3.5 and Gemma 4 references point to 2026 technical reports; these should be verified as citable sources.","section":null},{"comment":"§4.4: The self-citation to Beliaev 2025 (Esperantix) is for future work on latent-space exchange. This is appropriate but should be clearly marked as the author's own prior work.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and the core idea is interesting. The theoretical analysis of reward forms is genuinely useful. However, the central empirical claim (15pp competition delta) is unverifiable without multi-seed runs, and the post-hoc cascade direction selection is a methodological concern. Both issues are acknowledged in the Limitations, but they are load-bearing for the headline claims. I would recommend major revision with a request for 2-3 seeds on the primary conditions and reporting both cascade directions. If the delta survives multi-seed confirmation, this is a solid contribution. The alternative interpretation (difficulty-weighted reward shaping) is not necessarily a problem — it is still a valid and useful signal — but the paper's framing as 'competitive cross-model grading' should be softened if the mechanism cannot be distinguished from the simpler account."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive report. All three major comments identify real gaps in evidence that we acknowledge. We agree that (1) single-run training variance is the most serious concern, (2) post-hoc cascade direction selection inflates reported numbers, and (3) the mechanism story is not empirically distinguished from reward shaping. We commit to addressing all three in revision: we will add multi-seed runs for the two load-bearing conditions, report both cascade directions with validation-split selection, and add an error-profile complementarity analysis. The theoretical contribution (Section 4.4) and the 2x2 design are not contested.","responses":[{"response":"The referee is correct that this is the most important issue. The 15pp competition delta rests on single training runs, and our Clopper-Pearson intervals cover only evaluation sampling, not seed-to-seed training variance. We acknowledge this gap explicitly in the manuscript (Section 5.1, Limitations), but the referee is right that transparency does not substitute for evidence. We will run 3 seeds for the cooperative exchange and Agon conditions on Qwen3-0.6B on DeepMath-hard and report mean and standard deviation. If the delta shrinks or becomes unreliable under multi-seed measurement, we will report that honestly and adjust the claim accordingly. We note that the training curves in Figure 7b show consistent separation between cooperative and adversarial conditions across evaluation checkpoints within the single run, which provides some evidence the effect is not purely seed noise, but this is not a substitute for multi-seed confirmation.","revision_made":"yes","referee_comment":"Single training runs for the two load-bearing conditions (cooperative exchange and Agon) mean the 15pp competition delta is unverifiable. Clopper-Pearson intervals cover only binomial sampling, not run-to-run training variance. Need 2-3 seeds minimum."},{"response":"The referee is correct on both points. Selecting the better cascade direction on the held-out test set is optimistic selection, and not reporting the gap between directions obscures the magnitude of inflation. We will: (1) report both cascade directions (A->B and B->A) pass@1 values for all pair-trained conditions, (2) split the current 300-problem held-out set into a validation portion (for direction selection) and a test portion (for final reporting), and (3) report the gap between directions so readers can judge the inflation. We expect the gap to be modest given that role rotation trains both adapters in both roles, but this is currently an expectation, not a measured fact. This change compounds with the multi-seed runs above: final reported numbers will be mean over seeds, with direction selected per-seed on the validation split.","revision_made":"yes","referee_comment":"Both cascade directions evaluated, better reported post hoc on held-out set. Optimistic selection on evaluation set. Gap between directions not reported. Should report both directions and select on separate validation split."},{"response":"The referee correctly identifies that the paper states it cannot distinguish the two interpretations and that the shared-opponent ablation is consistent with both. We agree this weakens the conceptual contribution. We will add an analysis measuring whether the two trained adapters develop genuinely different error profiles: specifically, we will compute per-problem correctness correlations between the two adapters' standalone outputs on the held-out set and compare to the correlation between two independently seeded GRPO runs. If the adapters have lower error correlation than independently trained single models, this supports the complementary-blind-spots mechanism; if correlations are similar, the reward-shaping interpretation is favored. We want to be honest about expectations: our instantiation starts from near-identical adapters (differing only by noise-level initialization), so it is possible that complementarity is weak and the reward-shaping account is the primary driver. If the analysis shows this, we will say so and reframe the contribution accordingly. We cannot promise that the experiment will cleanly favor the competitive-grading interpretation; we can promise to run it and report the result.","revision_made":"partial","referee_comment":"The mechanism story (competitive cross-model grading via complementary blind spots) is indistinguishable from difficulty-weighted reward shaping. The shared-opponent ablation (32 vs 61) is consistent with both interpretations. Need a control that distinguishes them, e.g., measuring whether adapters develop genuinely different error profiles."}],"tokens_in":15156,"tokens_out":959,"duration_ms":76005,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The core idea here is new and worth taking seriously: two LoRA adapters over a frozen base are trained head-to-head, each rewarded for out-solving a rival that has seen its work. This is structurally distinct from self-play (one model), cooperative ensembling (consensus), and prover-verifier games (fixed roles). The conversion bonus reward — which upweights correctness exactly when the opponent failed — is derived from first-principles analysis of group-relative normalization, and the paper proves that a naive margin reward collapses to the cooperative gradient under standardization. That theoretical observation is clean and the ablation confirms it empirically (margin: 49, conversion bonus: 61, cooperative: 46). The 2×2 experimental design is thoughtful, and the cross-domain replication on CodeContests plus cross-family replication on Qwen3.5 and Gemma 4 provide real evidence that the effect isn't Qwen-specific. The emergent trace shortening (8.1k → 3.5k tokens) without any length penalty is a genuine and useful side effect. Credit is due for transparency: the paper explicitly flags that complementarity between adapters is a hypothesis, not a measured property, and acknowledges that the adversarial gain might be difficulty-weighted reward shaping rather than game-theoretic competition. That honesty matters. The soft spots are real but proportionate. Every number comes from a single training run. The Clopper-Pearson intervals cover binomial sampling over 300 problems, not seed-to-seed training variance, which for LoRA RL on 3,000 problems could easily be ±5-10pp. The 15pp competition delta (61 vs 46) is large enough that I doubt it would fully reverse, but without multi-seed confirmation the magnitude is unverified. The post-hoc selection of the better cascade direction on the held-out set is a second source of optimistic bias, and the gap between directions is not reported. The stress-test note flags both issues correctly; I agree they are the load-bearing concerns. The reader's weakest-assumption point about complementarity vs. reward shaping is interesting but secondary — the paper itself says it cannot distinguish the two, and either way the method works. This paper is for researchers working on reasoning RL who want a practical method that doesn't require process labels or reward models. It deserves a serious referee who can push for multi-seed runs and the cascade direction gap before acceptance.","headline":"Agon's competitive cross-model RL paradigm is genuinely novel; the headline 15pp competition delta rests on single training runs with post-hoc cascade selection, but the pattern is internally consistent enough to warrant a serious referee.","tokens_in":15933,"tokens_out":581,"would_cite":true,"duration_ms":106536,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Two rival models grade each other's reasoning, doubling accuracy on hard math","keywords":[],"falsifier":"If the two adapters' error profiles are measured after training and found to be highly correlated (same problems solved, same problems missed), then the conversion bonus is not supplying cross-model grading but merely upweighting hard contexts — an effect achievable without a second model at all.","tokens_in":15350,"feed_emoji":"🤺","tokens_out":1084,"duration_ms":361916,"temperature":0.7,"pith_summary":"Standard reinforcement learning for reasoning models rewards only the final answer, never the chain of thought that produced it. On hard problems this creates a perverse incentive: models learn to write longer traces with more hedging and backtracking, buying more chances to stumble onto the right answer rather than learning to think more efficiently per token. The paper proposes a fix called Agon. Instead of one model grading its own output, two distinct models — instantiated as two low-rank adapters over a single frozen base — are trained head-to-head. On each problem, one model drafts a solution and the other reads that solution's summary (with the final answer withheld) and tries to out-solve it. Roles alternate every step so both models learn both roles. The reward is not just correctness but a conversion bonus: solving a problem that your opponent got wrong is worth more than solving one they also got right. This means a reasoning step that the opponent exploits is implicitly penalized, while a step that leads to winning is reinforced — all without any human labels for good reasoning and without a separate learned reward model. The paper tests two claims under a matched generation budget. The exchange claim is that cross-model information exchange (one model reading another's work) beats single-model self-refinement. The competition claim is that adversarial reward (rewarding out-solving the peer) beats cooperative reward (rewarding correctness alone, with the peer only in context). On a hard math benchmark with a 0.6B-parameter model, the full Agon method reaches 61% pass@1 versus 30% for vanilla GRPO, with the cooperative-exchange ablation at 46% and single-model self-refinement at 32%. The ordering replicates on competitive programming and across two other model families. Notably, the trained cascade produces shorter final-stage traces (3.5k versus 8.1k tokens), an emergent byproduct rather than an explicit length penalty.","feed_headline":"Two rival models grade each other's reasoning, doubling accuracy on hard math","feed_subtitle":"Competitive cross-model training replaces expensive process labels with an implicit signal: out-reason a peer who has seen your work.","key_machinery":"Two LoRA adapters over one frozen base model; draft-and-challenge rollout where the challenger conditions on the drafter's post-reasoning summary (answer withheld); competitive reward = correctness + conversion bonus c(b_i)(1-c(a_i)); group-relative GRPO advantage computed within each adapter's own rollout group; role rotation every optimizer step so both adapters train in both streams; inference deploys as the same two-stage cascade (one drafts, the other answers after reading).","core_discovery":"The central mechanism is the conversion bonus: when the challenger model solves a problem the drafter got wrong, it receives a reward of 3 instead of 2 (the base correctness weight). This bonus multiplies the action-dependent correctness term, so unlike a naive margin reward (which subtracts the opponent's outcome and contributes zero expected gradient under group normalization), it shifts the gradient itself toward harder opponent contexts. The within-group variance in opponent difficulty — each challenger rollout sees a different opponent draft — is what makes the competitive term trainable. Without that variance (e.g., a shared opponent draft for the entire group), the bonus normalizes to","pith_inferences":[],"forward_implications":["If the competitive signal genuinely grades reasoning quality rather than acting as difficulty-weighted reward shaping, the same mechanism could replace expensive process reward models in domains where verifiers exist but step-level labels do not.","The 0.6B model trained with Agon (61%) outperforms a 7x larger 4B model trained with vanilla GRPO (59%), suggesting that improving signal quality per token may be more efficient than scaling parameters for hard-reasoning tasks.","The emergent trace shortening (3.5k vs 8.1k tokens without any length penalty) implies that giving a model a candidate solution to critique naturally reduces exploration bloat, which could reduce inference cost as a side benefit of the training method.","The method requires only a verifier and two divergent policies, so it extends to any domain with programmatic verification — the paper demonstrates this on competitive programming with unit-test rewards."],"fun_headline_variants":["Competing models grade each other's reasoning to double math accuracy","Cross-model rivalry doubles reasoning accuracy without process labels","Two-model competition rewards out-reasoning a rival who saw your work","Implicit rival grading trains reasoning with no process labels","Models grade rivals' reasoning to double hard math pass rate"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The two LoRA adapters, initialized from the same frozen base with only noise-level differences, are assumed to develop genuinely complementary failure modes during training. The paper acknowledges that complementarity is not measured directly and that the two adapters start nearly identical; if both converge to similar blind spots, the competitive signal reduces to difficulty-weighted reward shaping rather than genuine cross-model grading, and the paper's own experiments do","fun_headline_variants_meta":{"raw":{"variants":["Competing models grade each other's reasoning to double math accuracy","Cross-model rivalry doubles reasoning accuracy without process labels","Two-model competition rewards out-reasoning a rival who saw your work","Implicit rival grading trains reasoning with no process labels","Models grade rivals' reasoning to double hard math pass rate","Competitive cross-model training doubles GRPO accuracy on hard math","Rival models judge each other's reasoning during RL training","No process labels needed as rival models grade reasoning implicitly","Conversion bonus drives competitive reasoning training between models","Paired models trade drafts to learn reasoning without reward models"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1259,"prompt_tokens":585,"completion_tokens":674,"prompt_tokens_details":null},"tokens_in":585,"tokens_out":674,"duration_ms":43802,"temperature":1.0,"reasoning_tokens":603,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T02:08:08.582164+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If the two adapters' error profiles are measured after training and found to be highly correlated (same problems solved, same problems missed), then the conversion bonus is not supplying cross-model grading but merely upweighting hard contexts — an effect achievable without a second model at all.","supporting_citations":[],"review_version":1}