Artificial Intelligence · arXiv:2610.18760 v1 · submitted 13 Oct 2026 · 69 references
Beyond the Verdict: Agentic Code Judging with Verifiable Feedback
The paper says
“Test-time scaling for large language models (LLMs) can involve generating multiple candidate solutions for selection or revising an earlier attempt. To guide selection and revision, we need a verifier that can estimate candidate correctness and provide evidence of failure.”
more of the abstract
In competitive programming, however, official judging systems provide authoritative verdicts under finite submission budgets, without revealing failure explanations or failing inputs. To provide informative feedback without consuming official submissions, we formulate a task in which a model judges and diagnoses a candidate program using only the problem statement and source code, with generated counterexamples serving as checkable evidence. To evaluate this task, we build USACO-JUDGE from the USACO 2026 contests, using accepted and unaccepted programs written by human experts and multiple LLMs. We then train ARBITER, a 35B mixture-of-experts (MoE) agentic verifier, with reinforcement learning in a generic execution sandbox. Training raises counterexample success on incorrect programs from 40.61% to 62.67% and mean binary verdict accuracy on USACO-JUDGE from 71.41% to 87.41%, approaching frontier verification performance. Beyond verification, ARBITER outperforms baselines in both code selection and refinement. In an IOI 2026 case study, ARBITER-guided refinement raises GPT-OSS-120B's expected score above the gold-medal threshold under the official submission limit.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 14 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Serious empirical paper on agentic code verification with checkable counterexamples; the central training claim holds up, and the real risk is the self-built judge oracle, not the table readings. This line is wrong
2 major objections · 7 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 69 references say.
The paper says, at §5.2, Appendix B.3, Appendix A, Figure 3
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The headline counterexample-success numbers (40.61% to 62.67%) and the Figure 3 training reward both flow through the same in-house oracle: hand-written validators for all 39 USACO problems, custom output checkers for six, and reference solutions. The statement that validators and checkers are 'verified end to end' means they reproduce official verdicts on the official test data, which is not an audit of their behavior on novel generated inputs.
more of objection 1
A validator that admits illegal inputs, a reference solution with a bug, or a custom checker that misjudges a valid output would inflate training reward and evaluation labels at the same time, and reinforcement learning could in principle exploit such a loose oracle beyond what a single-shot baseline would do. The verdict-accuracy claims (71.41% to 87.41%) are far less exposed because they rest on rejudged official test data whose original scores are reproduced, so the residual risk is specific to the hack protocol. The mitigating evidence is real: UOJ-BENCH re-hosting reproduces original outcomes, the 400-case human audit was unanimous, the QOJ 390.20 realization is independent of the authors' oracle, and Appendix D.6 documents two hand-verifiable hacks of officially accepted submissions. None of this, however, re-checks a random sample of successful USACO-JUDGE counterexamples against an independent implementation. I request a manual audit of a random sample of successful hacks (input legality against the problem statement and genuineness of the induced failure under the intended judging semantics), plus an explicit statement of this residual risk in Section 5.2.
The paper says, at §6.2, Appendix F.2, Figure 1
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The 385.46 versus 334.6 contrast in Figure 1 and the Introduction is attributed to verifier feedback versus online-judge feedback, but the two policies differ in exploration as well: the ARBITER condition starts 50 internal trajectories with free verifier reviews, while the judge condition runs 10 trajectories with at most five judge queries each, both under the same 50-submission cap.
more of objection 2
The appendix discloses the trajectory counts, which I credit, but no matched-trajectory control is reported and the abstract and Figure 1 framing do not carry the caveat. The absolute claim is well supported: 385.46 is above the 361.12 gold threshold and is bracketed by the realized 390.20 on QOJ (Appendix F.2). The fix is therefore local: either add a judge-condition arm with 50 trajectories and one submission each (or an ARBITER arm with 10 trajectories), or reframe Figure 1 as a policy-level comparison rather than a feedback-type ablation.
7 minor objections
The paper says, at Table 1
Pith says: minor objection 1 of 7
The two USACO-JUDGE column groups appear to be swapped relative to their labels: the advertised 71.41/87.41 verdict-accuracy averages and the 65.34/84.50 hard-split values quoted in Section 6.1 reproduce only when the columns labelled 'Hack Succ.' and 'Verdict Acc.' are exchanged for the USACO half of the table. The underlying data are consistent; the labels must be fixed.
The paper says, at Table 1
Pith says: minor objection 2 of 7
The 'Overall Hack Succ.' column is not recoverable from the split-level values in Table 1 and the funnel in Table 7: candidate-weighted averages of the corrected split rates give approximately 42.5 and 65.4 for the agentic Qwen baseline and ARBITER, versus the reported 40.61 and 62.67. Please state explicitly how the overall is pooled, including how officially Accepted candidates that are hacked enter the numerator and denominator.
show 5 more
The paper says, at §4.1
Pith says: minor objection 3 of 7
Please state explicitly whether 'exclude all evaluation problems' covers the UOJ-BENCH problem set; since the training rollouts are online-judge-graded across 6,145 problems and UOJ-BENCH is UOJ-based, problem-level overlap with the training pool would inflate the UOJ-BENCH results even though the released candidates are excluded.
The paper says, at §6.1, Table 1
Pith says: minor objection 4 of 7
The headline verification figures are point estimates without dispersion; the selection study reports ranges over repeats, and the same treatment (or confidence intervals) should be given to the verdict-accuracy and hack-success numbers and to the expected IOI scores.
The paper says, at §6.1, Table 1
Pith says: minor objection 5 of 7
Given the 1:2 accepted/unaccepted ratio of USACO-JUDGE, an always-unaccepted baseline achieves 66.7% verdict accuracy; stating this floor next to the 71.41/87.41 figures would put the reported gain in context.
The paper says, at §6.3, Table 3
Pith says: minor objection 6 of 7
Comparisons against published baselines use regenerated pools with unmatched inference compute, as the paper acknowledges; because pool difficulty can differ even when aggregate uniform-selection rates match, the same-pool comparison against the perfect-selection bound is the strongest evidence, and the cross-pool baseline comparisons should be labeled as suggestive.
The paper says, at Figure 3
Pith says: minor objection 7 of 7
The caption and reward annotations are garbled in the submitted text ('+0.05runs +0.10legal +0.750breakssurvives' and similar); the reward decomposition needs a clean, legible rendering.
What would settle it
Have two independent competitive programmers, or the USACO problem setters, re-implement the 39 input validators and six custom checkers from the official problem statements without seeing the authors' code, then replay every successful ARBITER counterexample in USACO-JUDGE, including the Appendix D.6 hacks of officially Accepted submissions. Recompute hack success and the 71.41/87.41 verdict accuracy using only hacks that pass the independent validator and fail under the independent checker; any mismatch with the in-house UOJ results quantifies the inflation, while a clean replay would settle the concern.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 14 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 7 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Short version: this is one of the more careful empirical papers I have seen in this area. The new thing is the integrated task—verdict plus diagnosis plus deterministic replayable counterexample for a single candidate—trained end-to-end with execution-grounded rewards, and evaluated on USACO-JUDGE, a balanced benchmark with accepted and unaccepted programs. The training gains are real on their own terms: counterexample success on incorrect programs goes from 40.61% to 62.67%, and the candidate-weighted USACO verdict accuracy from 71.41% to 87.41%. I checked the reader's column-order worry: it does not land. Those figures reproduce from Table 1 as weighted averages of the USACO verdict columns, so the abstract is not misreporting the table. The evaluation design earns real credit: rejudged labels, validators and custom checkers exercised end to end, a verification funnel, a 400-case human audit with unanimous labels, and honest diagnostics of the majority-class shortcut and length-control side effects. The IOI case study is presented as an expected score, not a realized run, and the appendix discloses the trajectory-count asymmetry; the separate QOJ realization of 390.20 partly mitigates the worry.
The soft spot that matters is oracle fidelity. Every training reward and every headline metric flows through the authors' own UOJ re-host, with hand-written validators for all 39 USACO problems and custom checkers for six. Reproducing a candidate's original score verifies that the re-host agrees with official judging on existing tests; it does not independently validate behavior on novel generated inputs. A bug in a validator or checker would inflate training signal and evaluation numbers at the same time. The human audit and the QOJ run are mitigating evidence, but they do not re-audit the USACO-JUDGE hack protocol. This is not a fatal flaw, but absolute numbers will only be credible after independent validation of a sample of the validators or third-party judging of counterexamples. Two smaller issues: the selection comparison uses published baselines on regenerated pools with unmatched inference compute, and no error bars accompany the headline deltas. Both are disclosed, but they make the precise cross-system gaps weaker than the tables suggest. The central direction—that executable-reward RL on an agentic judge improves both selection and refinement feedback—holds up. This paper deserves a serious referee. If you work on code verifiers or test-time scaling, it is worth your time and worth citing after the oracle question gets an independent check.
The strongest objection
Every training reward and evaluation label flows through the paper's UOJ re-host, and the oracle is not independent of the method. For USACO-JUDGE, input validators for all 39 problems and output checkers for six multi-output problems are hand-written by the authors (Appendix B.3). The reward in Figure 3 pays +0.75 for an input that breaks the candidate and +0.10 for overturning an officially Accepted verdict; the same in-house check decides whether an Accepted candidate is re-labelled unaccepted in Section 6.1. A validator that admits illegal inputs, a reference solution with a bug, or a custom checker that misjudges an output would simultaneously inflate the training signal and the headline 40.61% to 62.67% and 71.41% to 87.41% numbers. The claim that validators and checkers are 'verified end to end' is an assertion, not an audit; reproducing a candidate's original score on official tests does not validate a checker on novel generated inputs. The external QOJ realization (390.20) and the 400-case human audit are real mitigating evidence, but they do not re-check the USACO-JUDGE hack protocol. The reader's Table 1 column-order concern does not land: the advertised 71.41/87.41 figures reproduce exactly as candidate-weighted averages of the USACO verdict columns (85.56/65.34 and 94.20/84.50 over 270/630 candidates), so the oracle question is the one that controls the central claim.
The central claim is that execution-grounded RL training produces a verifier whose checkable counterexamples improve both verdict accuracy and downstream refinement. For that claim to hold, the judge-side checks that produce the reward and the evaluation labels must faithfully implement the contest's judging semantics. This is the weakest link because the same in-house oracle sits on both sides: it validates generated inputs, runs the candidate and reference solution, compares outputs, and even re-labels officially Accepted candidates when a counterexample succeeds. A bug in any of those components would not just add noise; it would systematically reward and measure the same false signal, so the reported deltas could survive internal consistency checks while failing to transfer to official judging. The paper reports careful end-to-end verification of validators and checkers, an external QOJ score for the IOI case study, and a human audit, all of which are real support, but none independently re-audits the USACO-JUDGE oracle on model-generated counterexamples. I do not claim the oracle is in fact buggy; I claim it is the assumption that must be independently checked before the headline numbers can be read as evidence about real contest semantics. Since the reader already identified this as the weakest assumption and built a CONDITIONAL verdict on it, my read does not change the verdict: it stays conditional pending independent oracle validation.
The referee's report
The paper is careful and unusually honest about its own failure modes: the majority-class-guessing diagnostic (Appendix G.1), length-control side effects (G.3), the false-positive penalty trade-off (G.2), and the expected-versus-realized IOI accounting (F.2) are all disclosed and analyzed. The evaluation apparatus is a strength: USACO-JUDGE labels are rejudged with reproduced original scores, UOJ-BENCH is re-hosted and reproduces original outcomes, the verification funnel (D.3) localizes where attacks succeed or die, and the 400-diagnosis human audit reached unanimous labels. The headline verdict-accuracy figures are internally consistent: 71.41 and 87.41 reproduce exactly as candidate-weighted averages of the split-level USACO-JUDGE columns once the Table 1 column labels are read correctly, and the overall hack-success figures are consistent with the split-level funnel data in expectation. The QOJ 390.20 realization of the IOI case study provides an independent check of the absolute claim.
Pith's reading of the paper (a machine's summary, not the authors')
This paper tries to establish that an LLM can be trained to act as a judge for competitive-programming solutions: given only the problem statement and a candidate program, the model must predict whether the program survives the hidden official tests, say why it fails when it does, and return a concrete failing input that anyone can run and check. That matters because official judges spend a finite submission budget and return only a verdict, while test-time scaling—generating many candidates or revising a faulty attempt—needs cheap feedback that does not consume that budget. The authors train ARBITER, a 35B mixture-of-experts verifier, with reinforcement learning in an execution sandbox, rewarding it mainly for counterexamples that pass an input validator and actually break the candidate. Training raises counterexample success on incorrect programs from 40.61% to 62.67% and mean binary accept/reject accuracy on their new USACO-JUDGE benchmark from 71.41% to 87.41%; in an IOI 2026 case study, ARBITER-guided refinement lifts an open 120B solver to an expected 385.46 points, above the 361.12 gold threshold, under the official 50-submission cap. If the claims hold, a comparatively small verifier can supply the evidence needed for both choosing among solutions and repairing them, without spending scarce official submissions.
What carries the argument. The load-bearing object is the episode interface and the evidence gate built around it. The verifier's final call is \texttt{submit}($v, d, g$): a verdict among Accepted, Wrong Answer, Time Limit Exceeded, Runtime Error, Memory Limit Exceeded, and Compile Error; a free-text diagnosis; and, for any non-Accepted verdict other than Compile Error, a deterministic Python 3 generator whose stdout is exactly one input. The judge withholds the hidden tests, reference solution, validator, checker, and even the result of checking the verifier's own counterexample, so the model must build its evidence inside the sandbox, by compiling, running, differential-testing, and stress-testing the candidate. The reward design carries the argument: agreeing with an unaccepted reference verdict earns only 0.05 of the 1.00 available, running the generator adds 0.05, a legal input 0.10, a validated input that breaks the candidate 0.75, and matching the failure type the final 0.05, with an extra 0.10 for overturning an officially accepted verdict—a deliberately asymmetric gate, because a counterexample can prove failure while the absence of one cannot prove acceptance. The paper reports that this proxy is meaningful: counterexamples succeed for 71.6% of human-rated correct diagnoses and only 10.3% of incorrect or vacuous ones, and an alternating adaptive length penalty keeps the trained policy from drifting into unboundedly long rollouts.
Core claim. The central claim is that verification with a replayable counterexample is a trainable task, and that a model trained to reward only executable evidence becomes a useful substitute for a large share of official judge feedback. ARBITER takes a problem statement $P$ and a candidate program $C$, runs Python and shell commands in a sandbox to test $C$ on inputs it constructs itself, then ends its episode with a triple $(v, d, g)$: a verdict, a natural-language diagnosis, and a deterministic Python generator for one input. An external judge then runs $C$ on the official tests for the reference verdict $v^\star$; if $g$ is supplied, it validates the generated input against the problem constraints, runs a reference solution, and applies the output checker, and only a legal input that breaks $C$ counts as a verified failure. Reinforcing this with reinforcement learning—a small reward (0.05 of 1.00) for agreeing on an unaccepted verdict but the bulk of the credit (0.75) reserved for a breaking counterexample—raises counterexample success from 40.61% to 62.67% overall and mean binary verdict accuracy on USACO-JUDGE from 71.41% to 87.41%, and on the hard splits brings the 35B model past non-agentic verifiers up to 1.6T parameters. The same verifier outperforms published baselines in selecting among regenerated candidate pools and, in the IOI 2026 case study, guides an open 120B solver to an expected 385.46 points versus 334.6 with online-judge feedback; the appendix discloses that the two comparison arms differ in the number of internal refinement trajectories and that ARBITER's reviews use no submission slots, an asymmetry the abstract's headline does not state.
The premise it rests on. The entire argument assumes the authors' re-hosted online judge faithfully reproduces official contest judging: a validator-approved input that breaks a candidate under their hand-written validators and custom checkers is taken to certify the candidate's failure, and that same oracle computes both every training reward and the headline metrics, so a single buggy validator or checker would inflate all reported numbers at once.
If the paper is right (Pith's inferences, not the authors')
A trained verifier of this kind can supply the feedback that test-time scaling needs—picking among generated candidates and revising failed attempts—without consuming any official submission budget, since ARBITER's reviews use no judge slots.
The evidence gate converts diagnosis quality into a mechanically checkable training signal, so a verifier can be improved at explaining bugs without large-scale human labels.
In code selection, one ARBITER call per candidate beats the strongest published selection baseline in all eight reported settings (for example 82.6 versus 73.7 pass@1 for a 30B solver on LiveCodeBench with pools of 64), and eight calls move closer to the perfect-selection upper bound.
Officially accepted programs are not beyond suspicion: ARBITER broke USACO submissions that scored 100 on official tests, one through a quadratic cleanup loop that times out on a legal 200,000-element input and one through a sentinel collision that yields a wrong answer.
In refinement, the verdict-plus-diagnosis condition beats verdict-plus-counterexample in ten of twelve settings, indicating that the natural-language diagnosis, not the counterexample source, carries most of the repair signal.
Editorial inference: if the oracle-fidelity assumption holds, the same evidence-gated recipe should transfer to any setting where a candidate artifact can be executed against a checkable oracle and authoritative evaluations are rationed—unit-test generation, proof scripts checked by a trusted kernel, or data-pipeline validation—because the mechanism is not specific to competitive programming.
Editorial inference: the reward asymmetry the authors build in (a counterexample proves failure, its absence proves nothing) suggests learned verifiers will always be better at pruning bad candidates than at certifying good ones, so downstream systems should treat an Accepted verdict as a ranking signal rather than a guarantee; the paper's fallback to uniform selection when nothing is Accepted is
Editorial inference: a testable extension is to measure whether counterexample-grounded reinforcement learning continues to scale with backbone size and sampling budget, since the paper's curves show one ARBITER sample beating eight samples of the untrained backbone on every split, leaving open the question of where the remaining gap to frontier verifiers goes.
Editorial inference: because ARBITER found bugs inside officially accepted submissions, running it over the accepted history of a fixed contest would quantify how often official acceptance is fragile, a cheap audit the paper does not perform.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Test-time scaling for large language models (LLMs) can involve generating multiple candidate solutions for selection or revising an earlier attempt. To guide selection and revision, we need a verifier that can estimate candidate correctness and provide evidence of failure. In competitive programming, however, official judging systems provide authoritative verdicts under finite submission budgets, without revealing failure explanations or failing inputs. To provide informative feedback without consuming official submissions, we formulate a task in which a model judges and diagnoses a candidate program using only the problem statement and source code, with generated counterexamples serving as checkable evidence. To evaluate this task, we build USACO-JUDGE from the USACO 2026 contests, using accepted and unaccepted programs written by human experts and multiple LLMs. We then train ARBITER, a 35B mixture-of-experts (MoE) agentic verifier, with reinforcement learning in a generic execution sandbox. Training raises counterexample success on incorrect programs from 40.61% to 62.67% and mean binary verdict accuracy on USACO-JUDGE from 71.41% to 87.41%, approaching frontier verification performance. Beyond verification, ARBITER outperforms baselines in both code selection and refinement. In an IOI 2026 case study, ARBITER-guided refinement raises GPT-OSS-120B's expected score above the gold-medal threshold under the official submission limit.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-14
The authors' main claim, as the classifier picked it out 0.77
“Beyond verification, ARBITER outperforms baselines in both code selection and refinement.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.22 · human participants 0.14 · controlled comparison 0.91 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.95 · mentions code or data availability 0.08
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Test-time scaling for large language models (LLMs) can involve generating multiple candidate solutions for selection or revising an earlier attempt. | background 1.00 | 0.05 | 0.02 | hedged possibility 0.63 |
| To guide selection and revision, we need a verifier that can estimate candidate correctness and provide evidence of failure. | background 0.82 | 0.07 | 0.03 | asserted 1.00 |
| In competitive programming, however, official judging systems provide authoritative verdicts under finite submission budgets, without revealing failure explanations or failing inputs. | gap 0.51 | 0.06 | 0.02 | asserted 1.00 |
| To provide informative feedback without consuming official submissions, we formulate a task in which a model judges and diagnoses a candidate program using only the problem statement and source code, with generated counterexamples serving as checkable evidence. | method 0.55 | 0.70 | 0.07 | asserted 0.97 |
| To evaluate this task, we build USACO-JUDGE from the USACO 2026 contests, using accepted and unaccepted programs written by human experts and multiple LLMs. | method 0.90 | 0.96 | 0.03 | asserted 1.00 |
| We then train ARBITER, a 35B mixture-of-experts (MoE) agentic verifier, with reinforcement learning in a generic execution sandbox. | method 0.97 | 0.95 | 0.05 | asserted 0.99 |
| Training raises counterexample success on incorrect programs from 40.61% to 62.67% and mean binary verdict accuracy on USACO-JUDGE from 71.41% to 87.41%, approaching frontier verification performance. | result 1.00 | 0.96 | 0.27 | asserted 1.00 |
| Beyond verification, ARBITER outperforms baselines in both code selection and refinement. | result 1.00 | 0.92 | 0.32 | asserted 1.00 |
| In an IOI 2026 case study, ARBITER-guided refinement raises GPT-OSS-120B's expected score above the gold-medal threshold under the official submission limit. | result 1.00 | 0.92 | 0.31 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.96
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-14: 1 request, 7,038 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (9)
Figures are the authors', reproduced from the paper.
References (69)
- [1] CodeForces , author=. Hugging Face repository , howpublished=. 2025 , publisher= on Pith
- [5] Jingwei Shi and Xinxiang Yin and Jing Huang and Shengyu Tao and Jinman Zhao , editor =. CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions , booktitle =. 2026 , url =. doi:10.18653/V1/2026.ACL-LONG.108 , timestamp = on Pithdoi:10.18653/v1/2026.acl-long.108
- [11] Vatsal Venkatkrishna and Indraneil Paul and Iryna Gurevych , title =. Trans. Mach. Learn. Res. , volume =. 2026 , url = on Pith
- [12] Lunjun Zhang and Arian Hosseini and Hritik Bansal and Mehran Kazemi and Aviral Kumar and Rishabh Agarwal , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url = on Pith
- [13] Ansong Ni and Srini Iyer and Dragomir Radev and Veselin Stoyanov and Wen. International Conference on Machine Learning,. 2023 , url = on Pith
- [14] Jeevana Priya Inala and Chenglong Wang and Mei Yang and Andr. Fault-Aware Neural Code Rankers , booktitle =. 2022 , url = on Pith
- [16] Nat McAleese and Rai Michael Pokorny and Juan Felipe Ceron Uribe and Evgenia Nitishinskaya and Maja Trebacz and Jan Leike , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2407.00215 , eprinttype =. 2407.00215 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2407.00215
- [18] Bei Chen and Fengji Zhang and Anh Nguyen and Daoguang Zan and Zeqi Lin and Jian. CodeT: Code Generation with Generated Tests , booktitle =. 2023 , url = on Pith
- [23] Wenhao Zeng and Yuling Shi and Xiaodong Gu and Chao Hu and Chaofan Wang and Yuhao Cui and Hongting Zhou and Mengnan Qi and Jianqiao Wangni and Zhaojian Yu and Shuzheng Gao and Kai Cai and Shilin He , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2606.28436 , eprin... on PithPith's reviewarXivdoi:10.48550/arxiv.2606.28436
- [25] Mehrzad Samadi and Aleksander Ficek and Sean Narenthiran and Siddhartha Jain and Wasi Uddin Ahmad and Somshubra Majumdar and Vahid Noroozi and Boris Ginsburg , editor =. Scaling Test-Time Compute to Achieve. Proceedings of the 64th Annual Meeting of the Association for Computa... on Pithdoi:10.18653/v1/2026.acl-long.1532
- [26] Martin Mare. Security of Grading Systems , journal =. 2021 , doi = on Pith
- [29] Joongwon Kim and Wannan Yang and Kelvin Niu and Hongming Zhang and Yun Zhu and Eryk Helenowski and Ruan Silva and Zhengxing Chen and Srinivasan Iyer and Manzil Zaheer and Daniel Fried and Hannaneh Hajishirzi and Sanjeev Arora and Gabriel Synnaeve and Ruslan Salakhutdinov and A... on PithPith's reviewarXivdoi:10.48550/arxiv.2604.16529
- [30] Yufan Zhuang and Chandan Singh and Liyuan Liu and Yelong Shen and Dinghuai Zhang and Jingbo Shang and Jianfeng Gao and Weizhu Chen , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2602.03094 , eprinttype =. 2602.03094 , timestamp = on Pithdoi:10.48550/arxiv.2602.03094
- [31] Mark Chen and Jerry Tworek and Heewoo Jun and Qiming Yuan and Henrique Pond. Evaluating Large Language Models Trained on Code , journal =. 2021 , url =. 2107.03374 , timestamp = on PithPith's reviewarXiv
- [32] Jacob Austin and Augustus Odena and Maxwell I. Nye and Maarten Bosma and Henryk Michalewski and David Dohan and Ellen Jiang and Carrie J. Cai and Michael Terry and Quoc V. Le and Charles Sutton , title =. CoRR , volume =. 2021 , url =. 2108.07732 , timestamp = on PithPith's reviewarXiv
- [33] Naman Jain and King Han and Alex Gu and Wen. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , booktitle =. 2025 , url = on Pith
- [34] Stefano Maggiolo and Giovanni Mascellani , title =. Olympiads in Informatics , volume =. 2012 , url = on Pith
- [36] Zilin Zhu and Chengxing Xie and Xin Lv and slime Contributors , title =. 2025 , howpublished = on Pith
- [39] Introducing. 2026 , howpublished = on Pith
- [41] Zheye Deng and Weixiang Yan and Changlong Yu and Jiashu Wang , editor =. AlphaQuanter: An End-to-End Tool-Augmented Agentic Reinforcement Learning Framework for Stock Trading , booktitle =. 2026 , url =. doi:10.18653/V1/2026.FINDINGS-ACL.456 , timestamp = on Pithdoi:10.18653/v1/2026.findings-acl.456
show 49 more
- [43] Hunter Lightman and Vineet Kosaraju and Yuri Burda and Harrison Edwards and Bowen Baker and Teddy Lee and Jan Leike and John Schulman and Ilya Sutskever and Karl Cobbe , title =. The Twelfth International Conference on Learning Representations,. 2024 , url = on Pith
- [44] Charlie Victor Snell and Jaehoon Lee and Kelvin Xu and Aviral Kumar , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url = on Pith
- [45] Noah Shinn and Federico Cassano and Ashwin Gopinath and Karthik Narasimhan and Shunyu Yao , editor =. Reflexion: language agents with verbal reinforcement learning , booktitle =. 2023 , url = on Pith
- [46] Jiazheng Zhang and Ziche Fu and Zhiheng Xi and Wenqing Jing and Mingxu Chai and Wei He and Guoqiang Zhang and Chenghao Fan and Chenxin An and Wenxiang Chen and Zhicheng Liu and Haojie Pan and Dingwei Zhu and Tao Gui and Qi Zhang and Xuanjing Huang , editor =. AgentV-RL: Scalin... on Pithdoi:10.18653/v1/2026.findings-acl.1156
- [47] Agentic Reward Modeling: Verifying GUI Agent via Progressive Trajectory-Grounded Interaction , author=. 2026 , eprint= on Pith
- [48] Yuxuan Wan and Tianqing Fang and Zaitang LI and Yintong Huo and Wenxuan Wang and Haitao Mi and Dong Yu and Michael R. Lyu , editor =. Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification , booktitle =. 2026 , url =... on Pithdoi:10.18653/v1/2026.findings-acl.1243
- [49] Yuwei Zhao and Ziyang Luo and Yuchen Tian and Hongzhan Lin and Weixiang Yan and Annan Li and Jing Ma , booktitle =. 2025 , pages = on Pith
- [51] Anthropic . Introducing Claude Opus 4.8 . Anthropic, 2026. URL https://www.anthropic.com/news/claude-opus-4-8 on Pith
- [52] Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian - Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. UR... on Pith
- [53] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URL https://... on PithPith's reviewarXiv
- [54] Chaoqun Cui, Jing Huang, Shijing Wang, Liming Zheng, Qingchao Kong, and Zhixiong Zeng. Agentic reward modeling: Verifying gui agent via progressive trajectory-grounded interaction, 2026. URL https://arxiv.org/abs/2602.00575 on PithPith's reviewarXiv
- [55] DeepSeek - AI. Deepseek-v4: Towards highly efficient million-token context intelligence. CoRR, abs/2606.19348, 2026. doi:10.48550/ARXIV.2606.19348. URL https://doi.org/10.48550/arXiv.2606.19348 on Pithdoi:10.48550/arxiv.2606.19348
- [56] Ahmed El - Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera Gilaberte, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 contri... on PithPith's reviewarXivdoi:10.48550/arxiv.2502.06807
- [57] Shaopeng Fu, Xingxing Zhang, Li Dong, Di Wang, and Furu Wei. Refinerl: Advancing competitive programming with self-refinement reinforcement learning. CoRR, abs/2604.00790, 2026. doi:10.48550/ARXIV.2604.00790. URL https://doi.org/10.48550/arXiv.2604.00790 on PithPith's reviewarXivdoi:10.48550/arxiv.2604.00790
- [58] Naman Jain, King Han, Alex Gu, Wen - Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar - Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on L... on Pith
- [59] Hongchao Jiang, Yiming Chen, Yushi Cao, Hung yi Lee, and Robby T. Tan. C ode J udge B ench: Benchmarking LLM -as-a-judge for coding tasks. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 19416--19448. Ass... on Pithdoi:10.18653/v1/2026.acl-long.888
- [60] Kimi Team . Kimi K2.5: visual agentic intelligence. CoRR, abs/2602.02276, 2026. doi:10.48550/ARXIV.2602.02276. URL https://doi.org/10.48550/arXiv.2602.02276 on PithPith's reviewarXivdoi:10.48550/arxiv.2602.02276
- [61] Qingyao Li, Xinyi Dai, Weiwen Liu, Xiangyang Li, Yasheng Wang, Ruiming Tang, Yong Yu, and Weinan Zhang. Atgen: Adversarial reinforcement learning for test case generation. CoRR, abs/2510.14635, 2025. doi:10.48550/ARXIV.2510.14635. URL https://doi.org/10.48550/arXiv.2510.14635 on Pithdoi:10.48550/arxiv.2510.14635
- [62] Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \' e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po - Sen Huang, Johannes Welbl, Sven ... on PithPith's reviewarXivdoi:10.48550/arxiv.2203.07814
- [63] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-1... on Pith
- [64] Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu, and Yahui Zhou. Skywork-reward-v2: Scaling preference data curation via human-ai synergy. CoRR, abs/2507.01352, 2025. doi:10.48550/ARXIV.2507.... on PithPith's reviewarXivdoi:10.48550/arxiv.2507.01352
- [65] Zeyao Ma, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, and Jie Tang. Dynamic scaling of unit tests for code reward modeling. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for... on Pithdoi:10.18653/v1/2025.acl-long.343
- [66] Zeyao Ma, Jing Zhang, Xiaokang Zhang, Jiaxi Yang, Zongmeng Zhang, Jiajun Zhang, Yuheng Jing, Lei Zhang, Hao Zheng, Wenting Zhao, Junyang Lin, and Binyuan Hui. Scaling agentic verifier for competitive coding. CoRR, abs/2602.04254, 2026. doi:10.48550/ARXIV.2602.04254. URL https:... on Pithdoi:10.48550/arxiv.2602.04254
- [67] Stefano Maggiolo and Giovanni Mascellani. Introducing CMS : A contest management system. Olympiads in Informatics, 6: 0 86--99, 2012. URL https://ioi.te.lv/oi/shtm/INFOL107.shtml on Pith
- [68] Martin Mare s . Security of grading systems. Olympiads in Informatics, 15: 0 37--52, 2021. doi:10.15388/ioi.2021.04 on Pithdoi:10.15388/ioi.2021.04
- [69] Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen - Tau Yih, Sida I. Wang, and Xi Victoria Lin. LEVER: learning to verify language-to-code generation with execution. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Sca... on Pith
- [70] OpenAI. gpt-oss-120b & gpt-oss-20b model card. CoRR, abs/2508.10925, 2025. doi:10.48550/ARXIV.2508.10925. URL https://doi.org/10.48550/arXiv.2508.10925 on PithPith's reviewarXivdoi:10.48550/arxiv.2508.10925
- [71] OpenAI . Introducing GPT-5.5 . OpenAI, 2026. URL https://openai.com/index/introducing-gpt-5-5/ on Pith
- [72] Archiki Prasad, Elias Stengel - Eskin, Justin Chih - Yao Chen, Zaid Khan, and Mohit Bansal. Learning to generate unit tests for automated debugging. CoRR, abs/2502.01619, 2025. doi:10.48550/ARXIV.2502.01619. URL https://doi.org/10.48550/arXiv.2502.01619 on PithPith's reviewarXivdoi:10.48550/arxiv.2502.01619
- [73] Qwen Team . Qwen3 technical report. CoRR, abs/2505.09388, 2025. doi:10.48550/ARXIV.2505.09388. URL https://doi.org/10.48550/arXiv.2505.09388 on PithPith's reviewarXivdoi:10.48550/arxiv.2505.09388
- [74] Qwen Team . Qwen3.6-35B-A3B : Agentic coding power, now open to all, April 2026. URL https://qwen.ai/blog?id=qwen3.6-35b-a3b on Pith
- [75] Mehrzad Samadi, Aleksander Ficek, Sean Narenthiran, Siddhartha Jain, Wasi Uddin Ahmad, Somshubra Majumdar, Vahid Noroozi, and Boris Ginsburg. Scaling test-time compute to achieve IOI gold medal with open-weight models. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and Da... on Pith
- [76] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024. doi:10.48550/ARXIV.2402.03300. URL https://doi.org/10.4... on PithPith's reviewarXivdoi:10.48550/arxiv.2402.03300
- [77] Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. Natural language to code translation with execution. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Proces... on Pithdoi:10.18653/v1/2022.emnlp-main.231
- [78] Jingwei Shi, Xinxiang Yin, Jing Huang, Shengyu Tao, and Jinman Zhao. Codehacker: Automated test case generation for detecting vulnerabilities in competitive programming solutions. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (eds.), Proceedings of the ... on Pith
- [79] Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can language models solve olympiad programming? CoRR, abs/2404.10952, 2024. doi:10.48550/ARXIV.2404.10952. URL https://doi.org/10.48550/arXiv.2404.10952 on PithPith's reviewarXivdoi:10.48550/arxiv.2404.10952
- [80] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information P... on Pith
- [81] Shiven Sinha, Shashwat Goel, Ponnurangam Kumaraguru, Jonas Geiping, Matthias Bethge, and Ameya Prabhu. Can language models falsify? evaluating algorithmic reasoning with counterexample creation. CoRR, abs/2502.19414, 2025. doi:10.48550/ARXIV.2502.19414. URL https://doi.org/10.... on PithPith's reviewarXivdoi:10.48550/arxiv.2502.19414
- [82] Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 ... on Pith
- [83] Lingxiao Tang, Muyang Ye, Zhaoyang Chu, Xiaoxue Ren, Zhongxin Liu, Lingfeng Bao, and He Ye. An iterative test-and-repair framework for competitive code generation. CoRR, abs/2604.05560, 2026. doi:10.48550/ARXIV.2604.05560. URL https://doi.org/10.48550/arXiv.2604.05560 on PithPith's reviewarXivdoi:10.48550/arxiv.2604.05560
- [84] Weixi Tong and Tianyi Zhang. Codejudge: Evaluating code generation with large language models. In Yaser Al - Onaizan, Mohit Bansal, and Yun - Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, N... on Pithdoi:10.18653/v1/2024.emnlp-main.1118
- [85] Siddarth Venkatraman, Vineet Jain, Sarthak Mittal, Vedant Shah, Johan S. Obando - Ceron, Yoshua Bengio, Brian R. Bartoldson, Bhavya Kailkhura, Guillaume Lajoie, Glen Berseth, Nikolay Malkin, and Moksh Jain. Recursive self-aggregation unlocks deep thinking in large language mod... on Pithdoi:10.48550/arxiv.2509.26626
- [86] Yuxuan Wan, Tianqing Fang, Zaitang LI, Yintong Huo, Wenxuan Wang, Haitao Mi, Dong Yu, and Michael R. Lyu. Inference-time scaling of verification: Self-evolving deep research agents via test-time rubric-guided verification. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, an... on Pith
- [87] Zhexu Wang, Yiping Liu, Yejie Wang, Wenyang He, Bofei Gao, Muxi Diao, Yanxu Chen, Kelin Fu, Flood Sung, Zhilin Yang, Tianyu Liu, and Weiran Xu. Ojbench: A competition level code benchmark for large language models. CoRR, abs/2506.16395, 2025. doi:10.48550/ARXIV.2506.16395. URL... on Pithdoi:10.48550/arxiv.2506.16395
- [88] Tingqiang Xu, Hangrui Zhou, Tianle Cai, Alex Gu, and Kaifeng Lyu. Beyond problem solving: Uoj-bench for evaluating code generation, hacking, and repair in competitive programming. CoRR, abs/2606.12864, 2026. doi:10.48550/ARXIV.2606.12864. URL https://doi.org/10.48550/arXiv.2606.12864 on PithPith's reviewarXivdoi:10.48550/arxiv.2606.12864
- [89] Jiazheng Zhang, Ziche Fu, Zhiheng Xi, Wenqing Jing, Mingxu Chai, Wei He, Guoqiang Zhang, Chenghao Fan, Chenxin An, Wenxiang Chen, Zhicheng Liu, Haojie Pan, Dingwei Zhu, Tao Gui, Qi Zhang, and Xuanjing Huang. Agentv-rl: Scaling reward modeling with agentic verifier. In Maria Li... on Pith
- [90] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . Open... on Pith
- [91] Yuwei Zhao, Ziyang Luo, Yuchen Tian, Hongzhan Lin, Weixiang Yan, Annan Li, and Jing Ma. C ode J udge-eval: Can large language models be good judges in code understanding? In Proceedings of the 31st International Conference on Computational Linguistics, pp.\ 73--95. Association... on Pith
- [92] Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors. slime: An llm post-training framework for rl scaling. https://github.com/THUDM/slime, 2025. GitHub repository. Corresponding author: Xin Lv on Pith
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment