{"id":"64cf3d6e-5be0-4a71-a3e2-28e47e6d0553","arxiv_id":"2610.18760","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An RL-trained agentic verifier that emits checkable counterexamples improves verdict accuracy, code selection, and refinement, beating online-judge feedback under the same submission budget.","lead":"Researchers trained a 35-billion-parameter system that reads a competitive programming problem and a candidate solution, predicts whether the solution is correct, and produces a test case that actually breaks the code when it is wrong. The system improved its success at breaking buggy programs from roughly 41% to 63%, and its feedback helped a much larger language model exceed the gold medal threshold in an IOI 2026 case study.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The load-bearing assumption is that ARBITER's own judge re-host is a faithful oracle: one validator or checker bug would inflate training reward and every headline metric at once, and no independent audit of the hand-written validators is reported.","rationale":"The central claim is that execution-grounded RL training produces a verifier whose checkable counterexamples improve both verdict accuracy and downstream refinement. For that claim to hold, the judge-side checks that produce the reward and the evaluation labels must faithfully implement the contest's judging semantics. This is the weakest link because the same in-house oracle sits on both sides: it validates generated inputs, runs the candidate and reference solution, compares outputs, and even re-labels officially Accepted candidates when a counterexample succeeds. A bug in any of those components would not just add noise; it would systematically reward and measure the same false signal, so the reported deltas could survive internal consistency checks while failing to transfer to official judging. The paper reports careful end-to-end verification of validators and checkers, an external QOJ score for the IOI case study, and a human audit, all of which are real support, but none independently re-audits the USACO-JUDGE oracle on model-generated counterexamples. I do not claim the oracle is in fact buggy; I claim it is the assumption that must be independently checked before the headline numbers can be read as evidence about real contest semantics. Since the reader already identified this as the weakest assumption and built a CONDITIONAL verdict on it, my read does not change the verdict: it stays conditional pending independent oracle validation.","tokens_in":32670,"tokens_out":21530,"duration_ms":145057,"concrete_test":"Have two independent competitive programmers, or the USACO problem setters, re-implement the 39 input validators and six custom checkers from the official problem statements without seeing the authors' code, then replay every successful ARBITER counterexample in USACO-JUDGE, including the Appendix D.6 hacks of officially Accepted submissions. Recompute hack success and the 71.41/87.41 verdict accuracy using only hacks that pass the independent validator and fail under the independent checker; any mismatch with the in-house UOJ results quantifies the inflation, while a clean replay would settle the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every training reward and evaluation label flows through the paper's UOJ re-host, and the oracle is not independent of the method. For USACO-JUDGE, input validators for all 39 problems and output checkers for six multi-output problems are hand-written by the authors (Appendix B.3). The reward in Figure 3 pays +0.75 for an input that breaks the candidate and +0.10 for overturning an officially Accepted verdict; the same in-house check decides whether an Accepted candidate is re-labelled unaccepted in Section 6.1. A validator that admits illegal inputs, a reference solution with a bug, or a custom checker that misjudges an output would simultaneously inflate the training signal and the headline 40.61% to 62.67% and 71.41% to 87.41% numbers. The claim that validators and checkers are 'verified end to end' is an assertion, not an audit; reproducing a candidate's original score on official tests does not validate a checker on novel generated inputs. The external QOJ realization (390.20) and the 400-case human audit are real mitigating evidence, but they do not re-check the USACO-JUDGE hack protocol. The reader's Table 1 column-order concern does not land: the advertised 71.41/87.41 figures reproduce exactly as candidate-weighted averages of the USACO verdict columns (85.56/65.34 and 94.20/84.50 over 270/630 candidates), so the oracle question is the one that controls the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper defines a single-candidate verification task in which an agentic judge receives a problem statement and a candidate program and must return a verdict, a natural-language diagnosis, and, for suspected non-compilation failures, a deterministic generator of a counterexample input; the counterexample is verified externally by an input validator, a reference solution, and an output checker. The authors construct USACO-JUDGE (900 C++17 candidates from five LLMs and five human programmers over 39 held-out USACO 2026 problems, at a 1:2 accepted/unaccepted ratio), re-host the UOJ-BENCH hacking splits, and train ARBITER, a 35B MoE verifier, with GRPO in a generic execution sandbox, gating most of the reward on a counterexample that passes validation and breaks the candidate. On the Qwen3.6-35B-A3B backbone they report overall counterexample success rising from 16.10% (no tools) to 40.61% (agentic scaffold) to 62.67% (ARBITER), and USACO-JUDGE binary verdict accuracy rising from 71.41% to 87.41%. They further report gains in code selection relative to published baselines and an IOI 2026 case study in which ARBITER-guided refinement gives GPT-OSS-120B an expected 385.46 points under the official 50-submission cap, above the 361.12 gold threshold, versus 334.6 expected points with online-judge feedback.","tokens_in":32942,"tokens_out":30556,"duration_ms":191369,"significance":"The paper is careful and unusually honest about its own failure modes: the majority-class-guessing diagnostic (Appendix G.1), length-control side effects (G.3), the false-positive penalty trade-off (G.2), and the expected-versus-realized IOI accounting (F.2) are all disclosed and analyzed. The evaluation apparatus is a strength: USACO-JUDGE labels are rejudged with reproduced original scores, UOJ-BENCH is re-hosted and reproduces original outcomes, the verification funnel (D.3) localizes where attacks succeed or die, and the 400-diagnosis human audit reached unanimous labels. The headline verdict-accuracy figures are internally consistent: 71.41 and 87.41 reproduce exactly as candidate-weighted averages of the split-level USACO-JUDGE columns once the Table 1 column labels are read correctly, and the overall hack-success figures are consistent with the split-level funnel data in expectation. The QOJ 390.20 realization of the IOI case study provides an independent check of the absolute claim.","major_comments":[{"comment":"The headline counterexample-success numbers (40.61% to 62.67%) and the Figure 3 training reward both flow through the same in-house oracle: hand-written validators for all 39 USACO problems, custom output checkers for six, and reference solutions. The statement that validators and checkers are 'verified end to end' means they reproduce official verdicts on the official test data, which is not an audit of their behavior on novel generated inputs. A validator that admits illegal inputs, a reference solution with a bug, or a custom checker that misjudges a valid output would inflate training reward and evaluation labels at the same time, and reinforcement learning could in principle exploit such a loose oracle beyond what a single-shot baseline would do. The verdict-accuracy claims (71.41% to 87.41%) are far less exposed because they rest on rejudged official test data whose original scores are reproduced, so the residual risk is specific to the hack protocol. The mitigating evidence is real: UOJ-BENCH re-hosting reproduces original outcomes, the 400-case human audit was unanimous, the QOJ 390.20 realization is independent of the authors' oracle, and Appendix D.6 documents two hand-verifiable hacks of officially accepted submissions. None of this, however, re-checks a random sample of successful USACO-JUDGE counterexamples against an independent implementation. I request a manual audit of a random sample of successful hacks (input legality against the problem statement and genuineness of the induced failure under the intended judging semantics), plus an explicit statement of this residual risk in Section 5.2.","section":"§5.2, Appendix B.3, Appendix A, Figure 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The 385.46 versus 334.6 contrast in Figure 1 and the Introduction is attributed to verifier feedback versus online-judge feedback, but the two policies differ in exploration as well: the ARBITER condition starts 50 internal trajectories with free verifier reviews, while the judge condition runs 10 trajectories with at most five judge queries each, both under the same 50-submission cap. The appendix discloses the trajectory counts, which I credit, but no matched-trajectory control is reported and the abstract and Figure 1 framing do not carry the caveat. The absolute claim is well supported: 385.46 is above the 361.12 gold threshold and is bracketed by the realized 390.20 on QOJ (Appendix F.2). The fix is therefore local: either add a judge-condition arm with 50 trajectories and one submission each (or an ARBITER arm with 10 trajectories), or reframe Figure 1 as a policy-level comparison rather than a feedback-type ablation.","section":"§6.2, Appendix F.2, Figure 1","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The two USACO-JUDGE column groups appear to be swapped relative to their labels: the advertised 71.41/87.41 verdict-accuracy averages and the 65.34/84.50 hard-split values quoted in Section 6.1 reproduce only when the columns labelled 'Hack Succ.' and 'Verdict Acc.' are exchanged for the USACO half of the table. The underlying data are consistent; the labels must be fixed.","section":"Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The 'Overall Hack Succ.' column is not recoverable from the split-level values in Table 1 and the funnel in Table 7: candidate-weighted averages of the corrected split rates give approximately 42.5 and 65.4 for the agentic Qwen baseline and ARBITER, versus the reported 40.61 and 62.67. Please state explicitly how the overall is pooled, including how officially Accepted candidates that are hacked enter the numerator and denominator.","section":"Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Please state explicitly whether 'exclude all evaluation problems' covers the UOJ-BENCH problem set; since the training rollouts are online-judge-graded across 6,145 problems and UOJ-BENCH is UOJ-based, problem-level overlap with the training pool would inflate the UOJ-BENCH results even though the released candidates are excluded.","section":"§4.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The headline verification figures are point estimates without dispersion; the selection study reports ranges over repeats, and the same treatment (or confidence intervals) should be given to the verdict-accuracy and hack-success numbers and to the expected IOI scores.","section":"§6.1, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Given the 1:2 accepted/unaccepted ratio of USACO-JUDGE, an always-unaccepted baseline achieves 66.7% verdict accuracy; stating this floor next to the 71.41/87.41 figures would put the reported gain in context.","section":"§6.1, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Comparisons against published baselines use regenerated pools with unmatched inference compute, as the paper acknowledges; because pool difficulty can differ even when aggregate uniform-selection rates match, the same-pool comparison against the perfect-selection bound is the strongest evidence, and the cross-pool baseline comparisons should be labeled as suggestive.","section":"§6.3, Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The caption and reward annotations are garbled in the submitted text ('+0.05runs +0.10legal +0.750breakssurvives' and similar); the reward decomposition needs a clean, legible rendering.","section":"Figure 3","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Short version: this is one of the more careful empirical papers I have seen in this area. The new thing is the integrated task—verdict plus diagnosis plus deterministic replayable counterexample for a single candidate—trained end-to-end with execution-grounded rewards, and evaluated on USACO-JUDGE, a balanced benchmark with accepted and unaccepted programs. The training gains are real on their own terms: counterexample success on incorrect programs goes from 40.61% to 62.67%, and the candidate-weighted USACO verdict accuracy from 71.41% to 87.41%. I checked the reader's column-order worry: it does not land. Those figures reproduce from Table 1 as weighted averages of the USACO verdict columns, so the abstract is not misreporting the table. The evaluation design earns real credit: rejudged labels, validators and custom checkers exercised end to end, a verification funnel, a 400-case human audit with unanimous labels, and honest diagnostics of the majority-class shortcut and length-control side effects. The IOI case study is presented as an expected score, not a realized run, and the appendix discloses the trajectory-count asymmetry; the separate QOJ realization of 390.20 partly mitigates the worry.\n\nThe soft spot that matters is oracle fidelity. Every training reward and every headline metric flows through the authors' own UOJ re-host, with hand-written validators for all 39 USACO problems and custom checkers for six. Reproducing a candidate's original score verifies that the re-host agrees with official judging on existing tests; it does not independently validate behavior on novel generated inputs. A bug in a validator or checker would inflate training signal and evaluation numbers at the same time. The human audit and the QOJ run are mitigating evidence, but they do not re-audit the USACO-JUDGE hack protocol. This is not a fatal flaw, but absolute numbers will only be credible after independent validation of a sample of the validators or third-party judging of counterexamples. Two smaller issues: the selection comparison uses published baselines on regenerated pools with unmatched inference compute, and no error bars accompany the headline deltas. Both are disclosed, but they make the precise cross-system gaps weaker than the tables suggest. The central direction—that executable-reward RL on an agentic judge improves both selection and refinement feedback—holds up. This paper deserves a serious referee. If you work on code verifiers or test-time scaling, it is worth your time and worth citing after the oracle question gets an independent check.","headline":"Serious empirical paper on agentic code verification with checkable counterexamples; the central training claim holds up, and the real risk is the self-built judge oracle, not the table readings.","tokens_in":33652,"tokens_out":4798,"duration_ms":36103},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"A 35B verifier trained on executable counterexamples can judge a solution against hidden tests, explain its failure with a replayable input, and use that feedback to guide code selection and repair without spending official judge…","keywords":["agentic verifier","counterexample generation","code judging","reinforcement learning","competitive programming","test-time scaling","code selection","code refinement"],"falsifier":"Re-run a random sample of ARBITER's successful counterexamples and of the 900 USACO-JUDGE candidates through a second, independently implemented judge—fresh validators and checkers written from the problem statements alone, with no shared code—and compare acceptance labels and hack outcomes pairwise; if more than a few percent of counterexamples stop validating or breaking candidates, or the accept/reject labels diverge, the oracle assumption collapses. A cheaper spot check: submit the two counterexamples from Appendix D.6, which broke officially accepted USACO submissions, to the actual USACO grading system and see whether those failures reproduce.","tokens_in":33979,"feed_emoji":"🐛","tokens_out":19401,"duration_ms":114713,"temperature":0.7,"pith_summary":"This paper tries to establish that an LLM can be trained to act as a judge for competitive-programming solutions: given only the problem statement and a candidate program, the model must predict whether the program survives the hidden official tests, say why it fails when it does, and return a concrete failing input that anyone can run and check. That matters because official judges spend a finite submission budget and return only a verdict, while test-time scaling—generating many candidates or revising a faulty attempt—needs cheap feedback that does not consume that budget. The authors train ARBITER, a 35B mixture-of-experts verifier, with reinforcement learning in an execution sandbox, rewarding it mainly for counterexamples that pass an input validator and actually break the candidate. Training raises counterexample success on incorrect programs from 40.61% to 62.67% and mean binary accept/reject accuracy on their new USACO-JUDGE benchmark from 71.41% to 87.41%; in an IOI 2026 case study, ARBITER-guided refinement lifts an open 120B solver to an expected 385.46 points, above the 361.12 gold threshold, under the official 50-submission cap. If the claims hold, a comparatively small verifier can supply the evidence needed for both choosing among solutions and repairing them, without spending scarce official submissions.","feed_headline":"Counterexamples teach a 35B verifier to judge programs at 87.4%","feed_subtitle":"Execution-checked counterexamples guide selection and repair without spending official judge submissions.","key_machinery":"The load-bearing object is the episode interface and the evidence gate built around it. The verifier's final call is \\texttt{submit}($v, d, g$): a verdict among Accepted, Wrong Answer, Time Limit Exceeded, Runtime Error, Memory Limit Exceeded, and Compile Error; a free-text diagnosis; and, for any non-Accepted verdict other than Compile Error, a deterministic Python 3 generator whose stdout is exactly one input. The judge withholds the hidden tests, reference solution, validator, checker, and even the result of checking the verifier's own counterexample, so the model must build its evidence inside the sandbox, by compiling, running, differential-testing, and stress-testing the candidate. The reward design carries the argument: agreeing with an unaccepted reference verdict earns only 0.05 of the 1.00 available, running the generator adds 0.05, a legal input 0.10, a validated input that breaks the candidate 0.75, and matching the failure type the final 0.05, with an extra 0.10 for overturning an officially accepted verdict—a deliberately asymmetric gate, because a counterexample can prove failure while the absence of one cannot prove acceptance. The paper reports that this proxy is meaningful: counterexamples succeed for 71.6% of human-rated correct diagnoses and only 10.3% of incorrect or vacuous ones, and an alternating adaptive length penalty keeps the trained policy from drifting into unboundedly long rollouts.","core_discovery":"The central claim is that verification with a replayable counterexample is a trainable task, and that a model trained to reward only executable evidence becomes a useful substitute for a large share of official judge feedback. ARBITER takes a problem statement $P$ and a candidate program $C$, runs Python and shell commands in a sandbox to test $C$ on inputs it constructs itself, then ends its episode with a triple $(v, d, g)$: a verdict, a natural-language diagnosis, and a deterministic Python generator for one input. An external judge then runs $C$ on the official tests for the reference verdict $v^\\star$; if $g$ is supplied, it validates the generated input against the problem constraints, runs a reference solution, and applies the output checker, and only a legal input that breaks $C$ counts as a verified failure. Reinforcing this with reinforcement learning—a small reward (0.05 of 1.00) for agreeing on an unaccepted verdict but the bulk of the credit (0.75) reserved for a breaking counterexample—raises counterexample success from 40.61% to 62.67% overall and mean binary verdict accuracy on USACO-JUDGE from 71.41% to 87.41%, and on the hard splits brings the 35B model past non-agentic verifiers up to 1.6T parameters. The same verifier outperforms published baselines in selecting among regenerated candidate pools and, in the IOI 2026 case study, guides an open 120B solver to an expected 385.46 points versus 334.6 with online-judge feedback; the appendix discloses that the two comparison arms differ in the number of internal refinement trajectories and that ARBITER's reviews use no submission slots, an asymmetry the abstract's headline does not state.","pith_inferences":["Editorial inference: if the oracle-fidelity assumption holds, the same evidence-gated recipe should transfer to any setting where a candidate artifact can be executed against a checkable oracle and authoritative evaluations are rationed—unit-test generation, proof scripts checked by a trusted kernel, or data-pipeline validation—because the mechanism is not specific to competitive programming.","Editorial inference: the reward asymmetry the authors build in (a counterexample proves failure, its absence proves nothing) suggests learned verifiers will always be better at pruning bad candidates than at certifying good ones, so downstream systems should treat an Accepted verdict as a ranking signal rather than a guarantee; the paper's fallback to uniform selection when nothing is Accepted is ","Editorial inference: a testable extension is to measure whether counterexample-grounded reinforcement learning continues to scale with backbone size and sampling budget, since the paper's curves show one ARBITER sample beating eight samples of the untrained backbone on every split, leaving open the question of where the remaining gap to frontier verifiers goes.","Editorial inference: because ARBITER found bugs inside officially accepted submissions, running it over the accepted history of a fixed contest would quantify how often official acceptance is fragile, a cheap audit the paper does not perform."],"forward_implications":["A trained verifier of this kind can supply the feedback that test-time scaling needs—picking among generated candidates and revising failed attempts—without consuming any official submission budget, since ARBITER's reviews use no judge slots.","The evidence gate converts diagnosis quality into a mechanically checkable training signal, so a verifier can be improved at explaining bugs without large-scale human labels.","In code selection, one ARBITER call per candidate beats the strongest published selection baseline in all eight reported settings (for example 82.6 versus 73.7 pass@1 for a 30B solver on LiveCodeBench with pools of 64), and eight calls move closer to the perfect-selection upper bound.","Officially accepted programs are not beyond suspicion: ARBITER broke USACO submissions that scored 100 on official tests, one through a quadratic cleanup loop that times out on a legal 200,000-element input and one through a sentinel collision that yields a wrong answer.","In refinement, the verdict-plus-diagnosis condition beats verdict-plus-counterexample in ten of twelve settings, indicating that the natural-language diagnosis, not the counterexample source, carries most of the repair signal."],"supporting_citations":[{"why":"Supplies the UOJ-BENCH Easy/Hard hacking splits used to measure counterexample discovery on overt and covert faults; the paper re-hosts its candidates on its own judge.","marker":"(Xu et al., 2026)"},{"why":"Provides the agentic-verifier selection baseline and the evaluation protocol (regenerated candidate pools, LiveCodeBench v6 and OJBench, pass@1) that ARBITER must beat.","marker":"(Ma et al., 2026)"},{"why":"The Qwen3.6-35B-A3B backbone that becomes ARBITER; its no-tools 16.10% and agentic-scaffold 40.61% counterexample success anchor the reported training gain.","marker":"(Qwen Team, 2026)"},{"why":"Supplies the group-relative policy optimization (GRPO) algorithm used to train the verifier.","marker":"(Shao et al., 2024)"},{"why":"Inspires the adaptive length control that keeps rollout length and abort rates in check during training.","marker":"(Kimi Team, 2026)"},{"why":"Provides LiveCodeBench v6, one of the two code-selection benchmarks whose candidate pools ARBITER ranks.","marker":"(Jain et al., 2025)"},{"why":"Provides OJBench, the second code-selection benchmark used in the selection experiments.","marker":"(Wang et al., 2025)"},{"why":"Describes GPT-OSS-120B, the open 120B solver whose solutions are verified and guided in the refinement study and the IOI 2026 case study.","marker":"(OpenAI, 2025)"}],"fun_headline_variants":["A 35B verifier learns code judging from executable counterexamples","Counterexample training raises verifier accuracy to 87.4% on USACO-JUDGE","ARBITER uses generated counterexamples to judge, select, and repair code","Executable counterexamples help a 35B verifier outperform 1.6T non-agentic baselines","IOI case study: counterexample-guided repair lifts 120B solver above gold-medal score"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The entire argument assumes the authors' re-hosted online judge faithfully reproduces official contest judging: a validator-approved input that breaks a candidate under their hand-written validators and custom checkers is taken to certify the candidate's failure, and that same oracle computes both every training reward and the headline metrics, so a single buggy validator or checker would inflate all reported numbers at once.","fun_headline_variants_meta":{"raw":{"variants":["A 35B verifier learns code judging from executable counterexamples","Counterexample training raises verifier accuracy to 87.4% on USACO-JUDGE","ARBITER uses generated counterexamples to judge, select, and repair code","Executable counterexamples help a 35B verifier outperform 1.6T non-agentic baselines","IOI case study: counterexample-guided repair lifts 120B solver above gold-medal score"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000317,"raw_usage":{"cost":0.0003174626,"is_byok":false,"cost_details":{"upstream_inference_cost":0.0003174626,"upstream_inference_prompt_cost":3.22686e-05,"upstream_inference_completions_cost":0.000285194},"total_tokens":2832,"prompt_tokens":1265,"completion_tokens":1567,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":1138,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":1465}},"tokens_in":1265,"tokens_out":1567,"cost_source":"openrouter_usage","duration_ms":9792,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":1465,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-14T07:31:34.925226+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Re-run a random sample of ARBITER's successful counterexamples and of the 900 USACO-JUDGE candidates through a second, independently implemented judge—fresh validators and checkers written from the problem statements alone, with no shared code—and compare acceptance labels and hack outcomes pairwise; if more than a few percent of counterexamples stop validating or breaking candidates, or the accept/reject labels diverge, the oracle assumption collapses. A cheaper spot check: submit the two counterexamples from Appendix D.6, which broke officially accepted USACO submissions, to the actual USACO grading system and see whether those failures reproduce.","supporting_citations":[{"cited_title":"Qwen3.6-35B-A3B : Agentic coding power, now open to all, April 2026","cited_arxiv_id":null,"evidence_quote":"The Qwen3.6-35B-A3B backbone that becomes ARBITER; its no-tools 16.10% and agentic-scaffold 40.61% counterexample success anchor the reported training gain."},{"cited_title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","cited_arxiv_id":"2402.03300","evidence_quote":"Supplies the group-relative policy optimization (GRPO) algorithm used to train the verifier."},{"cited_title":"Kimi K2.5: Visual Agentic Intelligence","cited_arxiv_id":"2602.02276","evidence_quote":"Inspires the adaptive length control that keeps rollout length and abort rates in check during training."},{"cited_title":"gpt-oss-120b & gpt-oss-20b Model Card","cited_arxiv_id":"2508.10925","evidence_quote":"Describes GPT-OSS-120B, the open 120B solver whose solutions are verified and guided in the refinement study and the IOI 2026 case study."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}