Pith. sign in

REVIEW 1 major objections 4 minor 32 references

Given only a problem statement and one reference solution, a coding agent can audit an official test suite — finding 589 accepted-but-buggy AtCoder submissions its judge had passed.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A target-blind agent-generated test suite with a consensus-and-validator certification chain found 906 verified accepted-but-buggy submissions on AtCoder and led all baselines on post-cutoff Codeforces problems.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A careful, genuinely useful audit of online-judge suites: the 589/906 verified accepted-but-buggy ledgers are real empirical artifacts, but 'verified' is conditional on the consensus oracle, and the CF-fresh pool is partly self-referential. the 1 major comments →

arxiv 2608.01715 v1 pith:DQCYZ473 submitted 2026-08-03 cs.SE cs.AI

Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch

Shuyang Xie , Shuxiao Xie , Feng Zhu , Yanli Ji , Wangmeng Zuo This is my paper
classification cs.SE cs.AI
keywords test-suite auditingcoding agentscompetitive programmingadversarial test generationonline judge verdictsinput legality validationconsensus oracleaccepted-but-buggy submissions
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to turn test-suite adequacy from an assumption into an audited property. Its claim: an off-the-shelf coding agent, shown only a problem statement and one human reference solution, can construct adversarial test suites that expose submissions the official judge accepted even though they are buggy — 589 verified cases in a sample of 20,375 accepted AtCoder submissions, with a five-engine union floor of 906. Every finding is certified by a chain that does not rely on the official judge: independent accepted solutions must agree on the expected output, brute-force solvers settle disputes, and a per-problem validator certifies each killing input as legal. The same test-building method works where no official suite exists, out-covering five reproduced baselines on 41 post-cutoff Codeforces problems at every tested input budget. A sympathetic reader would care because datasets, benchmarks, and training signals treat 'accepted' as ground truth; this work shows that verdict can be checked and supplies the checking machinery.

Core claim

The paper claims that a target-blind coding agent — shown only a problem statement and one accepted human reference solution, never the official tests, verdicts, or submissions under audit — can audit a judge's hidden suite. The agent builds adversarial inputs that expose what the official suite misses, and a three-link certification chain makes each exposure reliable: expected output is consensus among multiple independently written accepted solutions, brute-force solvers settle disagreements, and a statement-derived validator certifies every killing input legal. Concretely: one agent found 589 verified accepted-but-buggy submissions among 20,375 audited AtCoder accepted submissions, with a

What carries the argument

The load-bearing mechanism is the certification chain, which turns an output disagreement into a certified bug without trusting the official judge. It has three links: a consensus oracle that defines the expected output as agreement among independently written accepted human solutions (eight per AtCoder problem, at least two successful runs per Codeforces problem); brute-force adjudication of disputes; and a strict per-problem input validator, written from the statement alone while blind to the killing inputs, that certifies each failing input as legal before it can score. The second mechanism is the target-blind auditor: an off-the-shelf coding agent receives only the statement and one refe

Load-bearing premise

The chain assumes that agreement among several accepted human reference solutions defines true expected output; if those references share a systematic misinterpretation of a problem, the oracle can certify a wrong verdict — the paper's own retraction of 123 flags on one problem shows this failure mode is real, not hypothetical.

What would settle it

Re-run the certification chain on a problem whose true output is independently computable by brute force, and compare every oracle 'expected output' against brute-force ground truth. A single certified killing input whose oracle output disagrees with brute force while the killed submission agrees with it would falsify the claim that the ledger contains verified bugs. A cheaper check: scan the released 589-entry ledger for any entry whose consensus can be overturned by a larger reference pool or an independent solver, the way 123 flags on abc164_e were retracted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • 'Accepted' stops meaning 'correct': online-judge verdicts, and the datasets and benchmarks built on them, become objects that can be audited rather than authorities to be trusted.
  • The certification chain — consensus oracle, brute-force adjudication, legality validator — is a reusable pattern for verifying findings from any test generator without access to the judge's hidden tests.
  • Where no official suite exists, agent-built suites can supply the evaluation, beating mutation-based and LLM-generated baselines on fresh problems.
  • The released ledger, validators, and judging harness let anyone machine-recheck every certified count, so the paper's numbers are executable rather than asserted.
  • Official suites remain the stronger single judge; agent suites complement them by closing the blind spot rather than replacing the human baseline.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same audit pattern could transfer beyond competitive programming — to specification-driven code generation, unit-test generation, or dataset cleaning — wherever a consensus of independent reference implementations can seed an expected-output oracle; the main obstacle is that such references rarely exist outside that setting.
  • Because the 1,094 dropped illegal-input candidates were not regenerated, the 906 floor is almost certainly an undercount of the accepted-but-buggy submissions even in this 8.9% sample; a validator-aware generator could raise the floor.
  • The paper's own abc164_e retraction — 123 flags overturned because the 8-solution oracle was the outlier — suggests the consensus oracle should be stress-tested with larger or algorithmically more diverse reference pools before a finding is trusted in future audits.
  • The 1.87% Wilson upper bound on the coverage-layer false-discovery rate implies downstream users should treat the ledger as a strong prior rather than an absolute census, and report the unconfirmed risk-layer rows explicitly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 4 minor

Summary. The paper proposes an audit framework in which an off-the-shelf coding agent, given only a problem statement and one accepted human solution, constructs adversarial test suites. A certification chain—a consensus oracle over multiple accepted reference solutions, brute-force adjudication of disputes, and a per-problem input validator—is used to verify that flagged submissions are genuinely buggy without relying on the official judge. On 106 AtCoder problems, one agent arm produces 589 verified accepted-but-buggy submissions, with a five-arm legality- and tolerance-clean union floor of 906; on the known-bug REJ panel all five engines stay within 1.7pp of the re-judged official logic-bug coverage. On 41 post-cutoff Codeforces problems with no official suites, the agent arm leads five reproduced baselines at every tested input budget (cov@50 = 0.952 vs 0.809 for the strongest baseline). The same certification pattern is applied in both settings, and the paper releases machine-checkable ledgers, validators, and judging harnesses.

Significance. If the central claims hold, the paper makes a substantial contribution: it turns suite adequacy from an assumed property into an audited one, demonstrates that target-blind coding agents can expose bugs official suites miss, and shows the same method can supply tests when no official suite exists. The paper is unusually careful: the AtCoder funnel includes reproducibility filtering, consensus-oracle corrections (including the abc164_e retraction), legality gates, tolerance filtering, a signed human-adjudicated sample with a Wilson 95% upper FDR bound of 1.87%, and explicit exclusion of TLE from all bug claims. The artifacts and the single Linux judging harness are strengths, as is the honest treatment of noninferiority (only two pre-designated engines clear the formal test). The central caveat is that every 'verified' finding is relative to the consensus-oracle assumption—accepted human reference solutions are semantically correct on the tested inputs—and the CF-fresh buggy pool is partly self-referential. These caveats are acknowledged, but they affect the strength of the headline claims.

major comments (1)
  1. [Section 3 and Section 6 (consensus oracle)] The CF-fresh buggy pool is defined as the union of the arms' own certified findings. An arm's cov@k is then measured against a pool that its own test-generation outputs helped to construct. The paper reports that removing the agent arm leaves the pool unchanged, but it does not report leave-one-out checks for the other five arms. If, say, CodeContests+ uniquely discovers a bug, that bug enters the pool and only CodeContests+ receives credit for it, inflating its measured coverage relative to a pool built from independent ground truth. This affects the central claim that the agent 'leads all five reproduced baselines at every tested input budget.' Please report cov@k on pools constructed by excluding each arm in turn, or construct an independent bug pool (e.g., brute-force-verified small cases or an external known-bug set) and re-run the comparison.
minor comments (4)
  1. [Section 6 (TCE sweep)] The text 'normalized cov@10.65versus the agent at0.32' appears to be a typo for 'cov@1 0.65 versus the agent at 0.32.' Please fix the spacing and numeric formatting.
  2. [Section 4 and Appendix A (re-adjudication protocol)] The description says that for wrong-answer rows the reviewing model saw neither expected/got values nor buggy source; please clarify how the output comparison was performed without these values and how 'disagreed with 0 oracle outputs' was recorded.
  3. [Appendix D.1 (agent prompt)] AGENTS.md tells the agent that 'an input is legal if docs/ref.cpp runs OK on it,' while Section 3 requires a statement-derived per-problem validator. Please clarify the relationship between construction-time legality checking via the reference and the later certification-chain validator, and why the former does not weaken the latter.
  4. [Figure 3] The inset label 'same gap, k ≤ 5' is ambiguous. State explicitly that the inset magnifies the gap between the agent and the best baseline for k ≤ 5.

Circularity Check

1 steps flagged

CF-fresh coverage is measured on a buggy pool defined by the arms themselves; the AtCoder core claims remain externally grounded.

specific steps
  1. self definitional [Section 5, 'Buggy-pool membership and comparison arms'; Section 7, 'Sample scope'; Figure 3 caption]
    "A sample-passing solution enters the pool when some arm’s legal input exposes it and the three-reference consensus oracle of §3 certifies the failure; the pool is the union of the arms’ certified findings. ... On the union pool, cov@k gives a relative ordering of the compared arms, not an absolute rate. ... Compared arms’ certified-finding union bounds the CF-fresh buggy pool; arm-independent pool construction remains open."

    cov@k is the fraction of the buggy pool killed by an arm's inputs, and the buggy pool is defined as the union of the arms' own certified findings. Every arm's certified findings therefore belong to the denominator and are killed by that arm, so each arm's coverage contains its own contribution by construction. Removing the agent's findings does not remove the selection: the pool still consists only of bugs that at least one compared arm exposed, a biased sample for ranking the arms. The claimed all-budget lead is thus partly an artifact of the pool's definition rather than an externally fixed ground truth; the paper concedes arm-independent pool construction remains open.

full rationale

The AtCoder audit is substantially self-contained: the REJ known-bug panel is an externally defined pool of official rejections, the certified ledger is filtered through reproducibility, consensus correction, legality, and tolerance gates, and post-hoc independent solvers and brute-force checks provide external adjudication. The expected-output oracle is an assumption about human reference correctness, but the paper audits it; an assumption is not a circular reduction. No load-bearing self-citation or imported uniqueness theorem appears. The one genuine self-referential step is the CF-fresh benchmark. The buggy pool is defined as the union of the arms' certified findings, and cov@k is the fraction of that pool each arm kills; therefore each arm's own certified findings enter the denominator and numerator by construction. The paper's leave-the-agent-out check removes only the agent's unique contribution; it does not remove the pool's dependence on the arm composition, so the all-budget lead is partly an artifact of the pool definition. The paper acknowledges this ('arm-independent pool construction remains open'), but the relative ordering claim still reduces in part to the pool's self-definition. Hence score 4, not higher: the AtCoder ledger and oracle audits carry independent empirical content, and the CF-fresh result is explicitly framed as a relative ordering.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new physical or formal entities are postulated. The CF-fresh benchmark and the certified ledger are constructed artifacts with released data, not ad hoc theoretical constructs.

free parameters (4)
  • agent input budget = 50
    Design choice set before evaluation; coverage and cost both depend on it. Not fitted to data.
  • AtCoder per-problem sample caps = 200 accepted / 300 rejected
    Convenience sample rule; limits the audit to 8.9% of the accepted pool and defines the in-sample floor.
  • noninferiority margin = 0.02
    Pre-specified statistical margin, not estimated from data.
  • CF-fresh date window = Mar 1 to Jun 9, 2026
    Harvest window chosen to postdate the declared cutoff; affects pool freshness.
axioms (4)
  • domain assumption Consensus oracle correctness: majority of independently accepted human solutions gives the true expected output.
    Used in Section 3 to define killed and all verified buggy labels; audited in Section 6 but not proven.
  • domain assumption Statement-derived validators correctly encode input legality; false accepts are the only direction that could create false kills.
    Sections 3 and 7: legality means validator-passing, not semantic ground truth; sanity checks are structural, not exhaustive.
  • domain assumption Post-cutoff premise: the model training-data cutoff (Jan 2026) guarantees no memorization of CF-fresh problems.
    Section 5: only one vendor declares a cutoff; others publish none, so freshness is not guaranteed per engine.
  • domain assumption Local judging environment reproduces official judge semantics, including stack-limit removal and exact-output judging.
    Appendix A: Linux-only harness; TLE is excluded to avoid hardware dependence.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch." pith.science (2026). https://pith.science/paper/DQCYZ473

@misc{pith2026260801715,
  author       = {Pith},
  title        = {Pith review of: Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DQCYZ473}},
  note         = {Machine review of arXiv:2608.01715}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Online-judge verdicts and the datasets and benchmarks built on them are treated as ground truth for evaluating and training large language models for code. Yet prior audits have sounded a warning: official suites accept buggy submissions. These audits, however, stop at the warning and offer no practical remedy. Our remedy has two parts: an off-the-shelf coding agent, serving as a test-suite auditor, both builds adversarial test suites to expose what official suites miss and supplies these suites where no official suite exists; a certification chain determines whether each agent-flagged submission is genuinely buggy without relying on the official judge: multiple independently written accepted solutions agree on the expected output for every test, brute-force solutions settle disagreements, and a per-problem validator certifies each failing input legal. One such agent identifies 589 verified accepted-but-buggy submissions among AtCoder's 20,375 audited accepted submissions; extending the same certification to all five agents yields a union floor of 906 such submissions. Five agents, scored separately, each stay within 1.7pp of official-suite coverage on logic bugs those suites catch. On post-cutoff Codeforces problems with no available official suites, the same test-building method leads all five reproduced baselines at every tested input budget. Where an official suite exists, the agent audits suite adequacy instead of assuming it; where none exists, agent suites catch the most buggy submissions among methods we reproduced and tested.

Figures

Figures reproduced from arXiv: 2608.01715 by Feng Zhu, Shuxiao Xie, Shuyang Xie, Wangmeng Zuo, Yanli Ji.

Figure 1
Figure 1. Figure 1: The auditor framework and its certification chain (§ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Micro-pooled logic-bug coverage (26,682 REJ submissions; 106 AtCoder problems; TLE-only excluded; [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: CF-fresh subset coverage cov@𝑘 (order-free) for all six arms. Coverage averages over 63 adjudicated buggy solutions. The official arm comprises the sample tests, so its zero coverage is by construction: all pool members pass them. The inset magnifies the gap for 𝑘 ≤ 5; the dotted line marks the agent’s 50-input design budget, beyond which its full set is used. sample arm has zero coverage by construction b… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 23 canonical work pages · 3 internal anchors

  1. [1]

    ClaudeCode:AnAgenticCodingToolfortheTerminal

    Anthropic. ClaudeCode:AnAgenticCodingToolfortheTerminal. https://github.com/anthropics/ claude-code, 2025

    work page 2025

  2. [2]

    Claude Opus 4.8.https://www.anthropic.com/news/claude-opus-4-8, 2026

    Anthropic. Claude Opus 4.8.https://www.anthropic.com/news/claude-opus-4-8, 2026

    work page 2026

  3. [3]

    Codecontests-o: Powering llms via feedback-driven iterative test case generation

    Jianfeng Cai, Jinhua Zhu, Ruopei Sun, Kangwen Zhao, Dongyun Xue, Mingxiao Feng, Wengang Zhou, and Houqiang Li. Codecontests-o: Powering llms via feedback-driven iterative test case generation. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors,Findings of the Association for Computational Linguistics (ACL 2026), pages 1054–1072...

    work page 2026

  4. [4]

    Can llms generate reliable test case generators? a study on competition-level programming problems, 2025

    Yuhan Cao, Zian Chen, Kun Quan, Ziliang Zhang, Yu Wang, Xiaoning Dong, Yeqi Feng, Guanzhong He, Jingcheng Huang, Jianhao Li, Yixuan Tan, Jiafu Tang, Yilin Tang, Junlei Wu, Qianyu Xiao, Can Zheng, Shouchen Zhou, Yuxiang Zhu, Yiming Huang, and Tianxing He. Can llms generate reliable test case generators? a study on competition-level programming problems, 20...

    arXiv 2025

  5. [5]

    Codet: Code generation with generated tests

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. InThe Eleventh International Conference on Learning Representations (ICLR 2023). OpenReview.net, 2023. URLhttps://openreview.net/forum?id=ktrw68Cmu9c

    work page 2023

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, HarriEdwards,YuriBurda,NicholasJoseph,GregBrockman,AlexRay,RaulPuri,GretchenKrueger,Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Wint...

    work page 2021

  7. [7]

    Deepseek-v4: Towards highly efficient million-token context intelligence, 2026

    DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026. URL https://arxiv.org/abs/2606.19348

    arXiv 2026

  8. [8]

    Klear-codetest: Scalable test case generation for code reinforcement learning, 2025

    Jia Fu, Xinyu Yang, Hongzhi Zhang, Yahui Liu, Jingyuan Zhang, Qi Wang, Fuzheng Zhang, and Guorui Zhou. Klear-codetest: Scalable test case generation for code reinforcement learning, 2025. URLhttps: //arxiv.org/abs/2508.05710

    Pith/arXiv arXiv 2025

  9. [9]

    Google Antigravity: An Agentic Development Platform

    Google. Google Antigravity: An Agentic Development Platform. https://antigravity.google/, 2025

    work page 2025

  10. [10]

    Gemini3.1ProModelCard

    GoogleDeepMind. Gemini3.1ProModelCard. https://deepmind.google/models/model-cards/ gemini-3-1-pro/, 2026

    work page 2026

  11. [11]

    Hardtests: Synthesizing high-quality test cases for llm coding, 2025

    Zhongmou He, Yee Man Choi, Kexun Zhang, Jiabao Ji, Junting Zhou, Dejia Xu, Ivan Bercovich, Aidan Zhang, and Lei Li. Hardtests: Synthesizing high-quality test cases for llm coding, 2025. URLhttps: //arxiv.org/abs/2505.24098

    Pith/arXiv arXiv 2025

  12. [12]

    Solvita: Enhancing Large Language Models for Competitive Programming via Agentic Evolution

    Han Li, Jinyu Tian, Rili Feng, Yuqiao Du, Chong Zheng, Chenyu Wang, Chenchen Liu, Shihao Li, Xinping Lei, Yifan Yao, Weihao Xie, Letian Zhu, and Jiaheng Liu. Solvita: Enhancing large language models for competitive programming via agentic evolution, 2026. URLhttps://arxiv.org/abs/2605.15301

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  13. [13]

    Taco: Topics in algorithmic code generation dataset, 2023

    Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset, 2023. URLhttps://arxiv.org/abs/2312.14852

    Pith/arXiv arXiv 2023

  14. [14]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Fre- itas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Ec- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Mas- son d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, P...

    2022

  15. [15]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Alice Oh, Tristan Naumann, 10 Coding Agents as Test-Suite AuditorsPreprint Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors,Advances in Neural Informatio...

    work page 2023

  16. [16]

    Zhang, Zhenpeng Chen, Federica Sarro, Mark Harman, Gang Huang, and Yun Ma

    Kaibo Liu, Yudong Han, Jie M. Zhang, Zhenpeng Chen, Federica Sarro, Mark Harman, Gang Huang, and Yun Ma. Who judges the judge: An empirical study on online judge tests.ISSTA 2023 (32nd ACM SIGSOFT International Symposium on Software Testing and Analysis), 2023. doi:10.1145/3597926.3598060

    arXiv 2023

  17. [17]

    Zhang, Federica Sarro, Gang Huang, and Yun Ma

    Kaibo Liu, Yudong Han, Yiyang Liu, Zhenpeng Chen, Jie M. Zhang, Federica Sarro, Gang Huang, and Yun Ma. Trickybugs: A dataset of corner-case bugs in plausible programs. InProceedings of the 21st International Conference on Mining Software Repositories, MSR ’24, page 113–117. ACM, April 2024. doi:10.1145/3643991.3644870. URLhttp://dx.doi.org/10.1145/364399...

    work page arXiv 2024

  18. [18]

    Zhang, Mark Harman, Yudong Han, Yun Ma, Yihong Dong, Ge Li, and Gang Huang

    Kaibo Liu, Zhenpeng Chen, Yiyang Liu, Jie M. Zhang, Mark Harman, Yudong Han, Yun Ma, Yihong Dong, Ge Li, and Gang Huang. Llm-powered test case generation for detecting bugs in plausible programs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63rd Annual Meeting of the Association for Computation...

    work page 2025

  19. [19]

    Codex CLI: A Lightweight Coding Agent for the Terminal.https://github.com/openai/ codex, 2025

    OpenAI. Codex CLI: A Lightweight Coding Agent for the Terminal.https://github.com/openai/ codex, 2025

    work page 2025

  20. [20]

    GPT-5.4 Thinking System Card

    OpenAI. GPT-5.4 Thinking System Card. https://deploymentsafety.openai.com/gpt-5-4- thinking/gpt-5-4-thinking.pdf, 2026

    work page 2026

  21. [21]

    OpenCode: An Open-Source AI Coding Agent for the Terminal.https:// opencode.ai/, 2025

    OpenCode Contributors. OpenCode: An Open-Source AI Coding Agent for the Terminal.https:// opencode.ai/, 2025

    work page 2025

  22. [22]

    Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir R

    Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir R. Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. In Joaquin...

    work page 2021

  23. [23]

    Codehacker: Automated test case generation for detecting vulnerabilities in competitive programming solutions

    Jingwei Shi, Xinxiang Yin, Jing Huang, Jinman Zhao, and Shengyu Tao. Codehacker: Automated test case generation for detecting vulnerabilities in competitive programming solutions. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors,Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1:...

    work page 2026

  24. [24]

    Logicase: Effective test case generation from logicaldescriptionincompetitiveprogramming

    Sicheol Sung, Aditi, Dogyu Kim, Yo-Sub Han, and Sang-Ki Ko. Logicase: Effective test case generation from logicaldescriptionincompetitiveprogramming. InProceedingsoftheThirty-FourthInternationalJointConfer- ence on Artificial Intelligence (IJCAI 2025), pages 7742–7750. ijcai.org, 2025. doi:10.24963/ijcai.2025/861. URLhttps://doi.org/10.24963/ijcai.2025/861

    work page doi:10.24963/ijcai.2025/861 2025

  25. [25]

    Co-evolving LLM coder and unit tester via reinforcement learning

    Yinjie Wang, Ling Yang, Ye Tian, Ke Shen, and Mengdi Wang. Co-evolving LLM coder and unit tester via reinforcement learning. In Danielle Belgrave, Cheng Zhang, Laura N. Montoya, Hsuan-Tien Lin, Razvan Pascanu, Piotr Koniusz, Marzyeh Ghassemi, Nancy Chen, Iván Vladimir Meza Ruíz, and Arturo Loaiza-Bonilla, editors,Advances in Neural Information Processing ...

    work page 2025

  26. [26]

    Codecontests+: High-quality test case generation for competitive programming.Conference on Empirical Methods in Natural Language Processing,

    Zihan Wang, Siyao Liu, Yang Sun, Hongyan Li, and Kai Shen. Codecontests+: High-quality test case generation for competitive programming.Conference on Empirical Methods in Natural Language Processing,

    work page

  27. [27]

    Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming

    Tingqiang Xu, Hangrui Zhou, Tianle Cai, Alex Gu, and Kaifeng Lyu. Beyond problem solving: Uoj-bench for evaluating code generation, hacking, and repair in competitive programming, 2026. URLhttps: //arxiv.org/abs/2606.12864

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  28. [28]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. InAdvances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. URL http://papers.nips. 11 Coding Agents as Test-Suite AuditorsPreprint cc/paper_files/paper/...

    work page 2024

  29. [29]

    Zheyuan Yang, Zexi Kuang, Xue Xia, and Yilun Zhao. Can llms generate high-quality test cases for algorithm problems? testcase-eval: A systematic evaluation of fault coverage and exposure.Annual Meeting of the Association for Computational Linguistics, 2025. doi:10.48550/arXiv.2506.12278

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2506.12278 2025

  30. [30]

    Themis: Automated constraint- aware test synthesis framework for code reinforcement learning

    Shengyu Ye, Qi Liu, Hao Jiang, Zheng Zhang, Heng Yu, and Zhenya Huang. Themis: Automated constraint- aware test synthesis framework for code reinforcement learning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 34432–34440, 2026

    work page 2026

  31. [31]

    Autocode: Llms as problem setters for competitive programming, 2025

    Shang Zhou, Zihan Zheng, Kaiyuan Liu, Zeyu Shen, Zerui Cheng, Zexing Chen, Hansen He, Jianzhu Yao, Huanzhi Mao, Qiuyang Mang, Tianfu Fu, Beichen Li, Dongruixuan Li, Wenhao Chai, Zhuang Liu, Aleksandra Korolova, Peter Henderson, Natasha Jaques, Pramod Viswanath, Saining Xie, and Jingbo Shang. Autocode: Llms as problem setters for competitive programming, 2...

    arXiv 2025

  32. [2025]

    doi:10.48550/arXiv.2506.05817

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.05817

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.