{"id":"d308ca4f-e6e7-4066-b9c4-e9ef64848497","arxiv_id":"2504.19162","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An adversarial self-play game between a sneaky generator and a step critic lets a critic model evolve to detect LLM reasoning errors without fresh human labels, improving math reasoning on benchmarks.","lead":"This paper trains a \"critic\" AI that checks the steps of a language model's math reasoning, by having it play an adversarial game against a \"sneaky generator\" that tries to create tricky wrong steps. The critic improves from 70.8% to 77.7% accuracy on a reasoning-step benchmark and can guide other models to solve more math problems correctly.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The self-play critic's RL labels come from a solver-success proxy, not human step correctness; without measuring proxy-human agreement, the evolution claim may be an artifact of that proxy.","rationale":"The reader's weakest assumption—that the solver-success criterion in §3.2 is a faithful proxy for human step correctness—is exactly the load-bearing point. The entire adversarial game is built on this proxy: the sneaky generator is rewarded for producing steps that drop solver success to 0%, and the critic is rewarded for flagging those steps. If the proxy diverges from human labels, the self-play curriculum trains the wrong target, and the ProcessBench/PRM800K/DeltaBench results cannot be read as evidence of improved step-level error detection. The paper provides no direct calibration between the proxy and human judgments, so this is an unvalidated empirical assumption rather than an internal inconsistency. The concern is concrete and testable: a human-agreement study on generated sneaky steps would settle whether the proxy labels are trustworthy. I do not think this warrants rejection, because the method is otherwise coherent, the ablations support the design choices, and the proxy may well be a reasonable heuristic; it warrants a conditional acceptance with the requested validation. The secondary issues noted by the reader—the overclaim about eliminating manual annotation and the modified evaluation protocols—are also real but are presentation/scoping problems rather than the central scientific risk. The use of PRM800K labels for initialization and filtering weakens the 'no manual annotation' statement but does not invalidate the self-play improvement itself. The changed ProcessBench/DeltaBench protocol is applied uniformly to baselines, so internal comparisons remain meaningful, though scores should not be quoted as leaderboard-SOTA. My read is therefore that the reader's CONDITIONAL verdict is correct and no verdict change is needed; the condition should explicitly include the human-agreement check on the proxy labels.","tokens_in":19443,"tokens_out":4282,"duration_ms":48965,"concrete_test":"Sample 200–300 generated sneaky steps from round 1 and round 2 self-play data, paired with their original correct steps. Have two independent human annotators with mathematical expertise label each transformed step as correct/incorrect under the PRM800K/ProcessBench rubric. Compute human-human agreement and human-vs-proxy agreement (the §3.2 criterion: original success ≥75%, sneaky success 0%). If human-vs-proxy balanced accuracy or Cohen's kappa is not close to human-human agreement, the RL labels in §3.5 are not reliable step-error labels, and the reported benchmark gains cannot be attributed to learning human-defined step correctness. A useful secondary check: evaluate the round-2 critic separately on human-labeled and proxy-labeled subsets of the held-out generated steps; a large accuracy gap would confirm that the critic exploits proxy-specific cues rather than general step errors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The self-play training signal in §3.2/§3.5 depends entirely on the automated validity check: a transformed step is labeled 'incorrect' only when completing from the original step succeeds ≥75% and completing from the transformed step succeeds 0%. The critic's RL reward is therefore a function of solver success, not of human step-level correctness, yet the paper's headline claims are about detecting human-annotated reasoning errors on ProcessBench, PRM800K, and DeltaBench. The paper reports no human-agreement measurement on the generated sneaky steps used in rounds 1 and 2 (6.4K + 6.8K critic samples, §C.2). If the proxy is confounded—for example, a transformed step can be mathematically valid but systematically misdirect the particular solver, or a human-annotated error may not collapse that solver's success rate to 0%—then the critic is optimized to detect 'solver-deceiving' steps rather than human-defined step errors. Improvements on human-labeled benchmarks would then reflect partial distributional overlap between the proxy and human labels, not genuine evolution of step-error detection. The paper's claim that 'no manual step-level annotation' is needed is also inaccurate as stated: critic SFT uses PRM800K labels to filter 21.8K critiques, and sneaky-generator SFT is built from PRM800K correct/incorrect pairs, so the initial policies already encode human labels.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes SPC, an adversarial self-play framework for training a step-level critic of LLM mathematical reasoning. Two Qwen2.5-7B-Instruct copies are initialized by SFT: a 'sneaky generator' that rewrites correct reasoning steps into subtle erroneous steps, and a 'critic' that classifies the current step and writes a critique. The generator is trained on PRM800K-derived step transformations; the critic is trained on 21.8K PRM800K-derived critiques. In self-play, the generator proposes transformed steps, a solver-success criterion (original step success ≥75% and transformed step success 0%) labels them incorrect, and the critic attempts to detect them; rewards from Eq. (2) and (3) drive offline RL updates via Eq. (4). Two rounds of self-play improve average recall on ProcessBench from 70.8% to 77.7%, on PRM800K from 71.0% to 75.8%, and on DeltaBench from 54.9% to 60.5%. The critic is also used to verify steps during test-time search, improving MATH500 and AIME2024 accuracy for Llama-3.1-8B, Qwen2.5-32B, and DeepSeek-R1-Distill-Qwen-7B.","tokens_in":19692,"tokens_out":6086,"duration_ms":56026,"significance":"If the results are reproducible and the proxy-validation concern is addressed, the framework is a useful contribution: it demonstrates that an adversarial game anchored to solver outcomes can produce iterative gains on human-labeled step benchmarks, with a coherent ablation showing the importance of balanced games and solver-derived rewards. The paper provides transparent experimental details, uses three human-annotated benchmarks for evaluation, and includes ablations on game-balancing strategies. However, the current evidence does not fully establish that the critic is learning human-relevant step correctness, because the self-play labels are generated by a solver-success proxy, and the evaluation protocol in Tables 1 and 2 is modified from the official benchmarks. The claim of eliminating manual annotation is also stronger than the method supports, since both initializations use PRM800K human labels.","major_comments":[{"comment":"The automated validity criterion (original step success ≥75% and transformed step success 0%) is the sole source of RL labels for positive and negative sneaky steps, and yet the paper reports no measurement of agreement between this proxy and human step-incorrectness judgments on the generated sneaky steps. Because the evaluation benchmarks (ProcessBench, PRM800K, DeltaBench) are human-annotated, the headline improvements depend on the proxy being a faithful surrogate; if it instead rewards steps that merely mislead the particular solver, the critic's evolution could be an artifact of the proxy. I request an explicit human-agreement study on a random sample of the 6.4K and 6.8K self-play examples (or on PRM800K validation pairs), reporting confusion rates, and a discussion of how label noise from the proxy would affect the RL training in Eq. (4).","section":"§3.2, §3.5, §C.2"},{"comment":"The evaluation protocol in §4.1 changes the ProcessBench and DeltaBench tasks from locating the first (or all) errors in complete solutions to predicting the correctness of a single current step on a constructed 1:1 balanced subset. This makes the reported 'accuracy' and 'recall' numbers not comparable with published ProcessBench and DeltaBench leaderboard scores, because the official metrics are positional error-detection scores on the original class distribution. Since the abstract claims to 'surpass strong baselines' and 'state-of-the-art process reward models', the authors should either report results under the original official metrics or explicitly constrain all comparative claims to the modified current-step protocol, and should explain how the rebalancing affects each baseline.","section":"§4.1, Tables 1–2"},{"comment":"The claim that SPC 'eliminates the need for manual step-level annotation' is inaccurate as stated. The critic initialization uses 21.8K critiques filtered by PRM800K human step labels (§3.3, §C.2), and the sneaky-generator initialization uses correct–incorrect step pairs from PRM800K (§3.2). The contribution is better described as removing the need for new human annotation during the self-play phase, while still relying on existing human-labeled resources. The abstract and conclusion should be rephrased accordingly.","section":"Abstract, §3.2–3.3, §C.2"},{"comment":"The abstract's claim that SPC-based search 'surpasses' state-of-the-art PRMs is not uniformly supported by Table 3. On AIME2024 with DeepSeek-R1-Distill-Qwen-7B, Self-Consistency + SPC and Self-Consistency + Qwen2.5-Math-7B-PRM800K both achieve 73.3%, and on Llama-3.1-8B with AIME2024 both achieve 6.67%. Either report statistically meaningful margins or soften the claim to 'matches or improves' on the evaluated settings.","section":"Abstract, Table 3"}],"minor_comments":[{"comment":"The sentence 'We therefore extract a 1:1 ratio of correct and erroneous steps' should specify whether the 1:1 sampling is stratified per source dataset and per solver, since PRM800K and ProcessBench have different step distributions.","section":"§4.1"},{"comment":"There are typos in the prompt texts: 'converse the last step' should be 'convert the last step', 'ImcompleteCondition' should be 'IncompleteCondition', and 'input acritique' should be 'input a critique'.","section":"Appendix B"},{"comment":"The abbreviation 'HarMean' is not defined in the table caption; it should be defined as the harmonic mean of the correct-step and error-step recalls.","section":"Table 2"},{"comment":"For the MATH500 and AIME2024 runs without self-consistency, the paper says results are averaged over at least three runs, but no standard deviations are reported, making it impossible to assess whether the small differences in Table 3 (e.g., 4.27 vs 5.63 for Llama) are significant.","section":"§4.2, Table 3"},{"comment":"The limitation statement mentions extending to more general domains, but it does not mention the proxy-validation caveat, which is the most important limitation; add a sentence acknowledging that the self-play labels depend on the solver-success criterion.","section":"§5, Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is likely to be acceptable after a major revision; the proxy-validation request is the key gate. The authors should be encouraged to run the original official metrics as well, because the current protocol change may be seen as cherry-picking by the broader community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real method paper, not a throwaway. The idea is to train a 'sneaky generator' and a step critic against each other, with the generator rewarded for producing steps that (a) tank an LLM solver's completion success and (b) fool the critic. The critic improves from 70.8% to 77.7% average recall on ProcessBench over two self-play rounds, and the test-time search gains on MATH500/AIME2024 look solid. The asymmetric balancing (S1 vs C0 in round 2) is a sensible practical choice.\n\nWhat's genuinely new: most self-play work for reasoning has been at the solution level; this tackles step-level critique and uses an external anchor (solver success) to keep the game from collapsing. The ablations are informative, especially the 'w/o Reward from Solver' condition, which shows the solver anchor matters. The paper also evaluates on three human-labeled benchmarks and beats strong baselines under the same protocol.\n\nSoft spots: the RL labels are defined by the solver-success proxy, not by human correctness. A step is 'incorrect' only if the original completion succeeds >=75% and the sneaky-step completion succeeds 0%. That is a reasonable operational definition, but the paper never measures how well it agrees with human step labels on the generated sneaky steps. So the critic might be learning to detect 'solver-deceiving' steps, and the gains on human benchmarks could reflect distributional overlap rather than true error detection. This is the main thing I'd want a referee to push on.\n\nSecond, the 'eliminating the need for manual step-level annotation' claim in the abstract is too strong. The critic's SFT uses PRM800K human labels to filter 21.8K critiques, and the generator's SFT is built from PRM800K correct/incorrect pairs. The self-play phase itself needs no new human labels, but the initialization is human-dependent. Easy fix in wording.\n\nThird, the evaluation protocol for ProcessBench and DeltaBench is changed from first-error detection to current-step recall. That's a legitimate choice for their use case, but it means the headline SOTA comparison to published PRM numbers is not apples-to-apples. The internal baselines make the comparisons fair, but the 'state-of-the-art' language should be softened.\n\nNone of these are disqualifying. The central mechanism works, the experiments are reasonably thorough, and the limitations are addressable in revision. I'd send this to peer review. A good referee will ask for a human-agreement sample on the proxy and a cleanup of the no-annotation claim.","headline":"A genuinely interesting self-play scheme for training step-level critics, held back by an uncalibrated solver-success proxy and an over-strong 'no annotation' claim; send it to reviewers.","tokens_in":20277,"tokens_out":4164,"would_cite":true,"duration_ms":41735,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An adversarial self-play game between a 'sneaky' generator that corrupts reasoning steps and a critic that must catch the corruption can train a step-level critic without human step annotations, and using that critic to guide search…","keywords":["self-play critic","step-level error detection","process reward model","LLM reasoning","chain-of-thought verification","adversarial training","test-time search","mathematical reasoning"],"falsifier":"Take the step pairs where the automatic 75%-to-0% rule and human annotators disagree, and check whether SPC's judgments follow the human labels or the automatic rule; if the critic tracks the automatic rule on those disagreements, the reported benchmark gains would reflect the proxy rather than genuine detection of human-annotated errors.","tokens_in":19223,"feed_emoji":"🧮","tokens_out":7071,"duration_ms":69295,"temperature":0.7,"pith_summary":"The paper tries to show that a step-level critic for checking LLM mathematical reasoning can be trained without step-by-step human annotations. It sets up an adversarial game in which one model rewrites a correct reasoning step into a subtly wrong one and a critic must catch the error; both are updated by reinforcement learning from win/loss rewards. Across three human-annotated process benchmarks, the critic's accuracy rises over self-play rounds and surpasses same-size baselines, including a distilled R1 model. The same critic can be used during inference to flag and regenerate bad steps, improving math accuracy on MATH500 and AIME2024. If the claim holds, step verification no longer needs to wait for expensive new human annotations each time LLMs change.","feed_headline":"Adversarial self-play trains a step critic without human labels","feed_subtitle":"A 'sneaky' generator and a critic sharpen each other, boosting error detection and math accuracy.","key_machinery":"The central mechanism is the adversarial self-play game between a sneaky generator and a step critic. The sneaky generator converts correct reasoning steps into subtle wrong steps meant to both reduce solver success and fool the critic; the critic must classify each step and produce a brief critique. The load-bearing machinery is the automated validation rule replacing human annotation: a step counts as wrong only if the original step yields at least 75% solver success while the sneaky step yields 0%. This rule, combined with paired positive-negative samples and an asymmetric opponent-selection strategy, keeps the reinforcement learning signal informative and self-renewing across rounds.","core_discovery":"SPC claims that a step-level critic's ability to detect errors in mathematical reasoning can evolve through self-play rather than through static annotation. A sneaky generator, initialized by supervised fine-tuning on correct-incorrect step pairs, edits correct steps into difficult wrong steps using predefined error types, while a critic, initialized on standardized critiques distilled from long chain-of-thought reasoning, judges each step's correctness. The paper defines a valid sneaky step by its effect on an LLM solver's success rate: the original step must let the solver succeed at least 75% of the time, and the sneaky step must bring that rate to 0%. With this automated label, the game rewards the winner with +1 and the loser with -1, and offline reinforcement learning updates both roles. A second round of evolution uses an asymmetric matchup, the round-1 sneaky generator against the round-0 critic, to keep the game balanced. The paper reports that this continuous evolution raises average accuracy from 70.8% to 77.7% on ProcessBench and from 71.0% to 75.8% on PRM800K, and that the resulting critic improves test-time search for multiple LLM solvers on MATH500 and AIME2024.","pith_inferences":["A natural next test is to use the automatic 75%-to-0% rule as a data engine for training reasoners directly, not just critics; if that works, the same game could supply process supervision at scale for other tasks.","The adversarial game could be extended beyond mathematics by defining analogous automated success-rate proxies for code generation or multi-step tool use, where step correctness also affects downstream outcomes.","The paper's paired-sample trick suggests a testable design for other self-play setups: collect both correct and incorrect model outputs on identical inputs so the reinforcement learning signal explicitly contrasts the two.","Because the paper limits its experiments to mathematical reasoning, the strongest extension would be to measure whether the same self-play critic improves step verification in scientific or formal domains where step-level labels are even scarcer."],"forward_implications":["Step-level error detection for LLM reasoning can keep pace with new LLMs because the game generates fresh training steps from current solvers without new human annotations.","The trained critic can be used at test time to prune or regenerate erroneous steps, improving math accuracy on MATH500 and AIME2024, and it combines with self-consistency for further gains.","A critic trained only on short chain-of-thought data can generalize to long chain-of-thought reasoning models like R1 and QwQ on DeltaBench, where prior process reward models show large performance drops.","Balanced or asymmetric self-play is necessary for continued improvement, since an overly unbalanced game leads to performance degradation in later rounds.","The sneaky generator itself becomes more effective over rounds, meaning the self-play loop generates increasingly hard training examples rather than a static dataset."],"supporting_citations":[{"why":"Supplies the prover-verifier game idea and the balancing practice that SPC adapts for step-level critique.","marker":"[19]"},{"why":"Provides the distillation-based initial critique style for the critic and serves as the same-size baseline SPC surpasses.","marker":"[21]"},{"why":"Supplies the correct-incorrect step pairs used to initialize the sneaky generator and the human-annotated test set used to evaluate critic recall.","marker":"[23]"},{"why":"Math-Shepherd is the automated-annotation process reward model baseline that SPC compares against on step scoring and search.","marker":"[26]"},{"why":"ProcessBench is the main step-correctness evaluation benchmark and the source of the Qwen2.5-Math-7B-PRM800K baseline.","marker":"[27]"},{"why":"DeltaBench tests whether the critic generalizes to long chain-of-thought reasoning steps from R1 and QwQ.","marker":"[35]"},{"why":"Self-consistency is combined with SPC at test time to produce the reported search gains on MATH500 and AIME2024.","marker":"[2]"},{"why":"Qwen2.5-7B-Instruct is the base model fine-tuned into both the sneaky generator and the critic.","marker":"[12]"},{"why":"An earlier adversarial language game between attacker and defender that motivates the self-play reward design.","marker":"[47]"}],"fun_headline_variants":["Self-play games let a critic learn to spot reasoning errors","No step labels? SPC trains a critic via adversarial play","Sneaky generator vs critic: a game that sharpens math checks","Adversarial self-play evolves a step critic for LLM reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatic rule, where a step counts as wrong only when the original step yields at least 75% solver success and the sneaky step yields 0%, is a faithful proxy for human-annotated step correctness.","fun_headline_variants_meta":{"raw":{"variants":["Self-play games let a critic learn to spot reasoning errors","No step labels? SPC trains a critic via adversarial play","Sneaky generator vs critic: a game that sharpens math checks","Adversarial self-play evolves a step critic for LLM reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000803,"raw_usage":{"total_tokens":3597,"prompt_tokens":1079,"completion_tokens":2518,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":2445}},"tokens_in":695,"tokens_out":2518,"duration_ms":17478,"temperature":1.0,"reasoning_tokens":2445,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:59:19.761172+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the step pairs where the automatic 75%-to-0% rule and human annotators disagree, and check whether SPC's judgments follow the human labels or the automatic rule; if the critic tracks the automatic rule on those disagreements, the reported benchmark gains would reflect the proxy rather than genuine detection of human-annotated errors.","supporting_citations":[{"cited_title":"Can large language models detect errors in long chain-of-thought reasoning?, 2025","cited_arxiv_id":null,"evidence_quote":"DeltaBench tests whether the critic generalizes to long chain-of-thought reasoning steps from R1 and QwQ."},{"cited_title":"Self-playing adversarial language game enhances llm reasoning","cited_arxiv_id":null,"evidence_quote":"An earlier adversarial language game between attacker and defender that motivates the self-play reward design."}],"review_version":1}