Pith. sign in

REVIEW 3 major objections 6 minor 61 references

ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ExeCRE claims execution-consistency patterns can stand in for ground-truth tests when estimating the reliability of LLM-generated reference code, and that gating self-correction on that estimate improves final correctness.

desk verdict Solid, honest paper: ExeCRE's Dawid–Skene on binary execution consistency is a genuine new technique for gating self-correction, and the empirical reduction in misleading feedback is large; the main caveat—majority consensus can masquerade as reliability—is acknowledged but not solved. read the letter →

arxiv 2608.04439 v1 pith:QDK2P6P5 submitted 2026-08-05 cs.SE cs.AI

Yiru Dong , Richong Zhang , Fanshuang Kong , Si Chen This is my paper · ORCID
classification cs.SEcs.AI
keywords largelanguagemodelscodegenerationself-correctionreliabilityestimationexecutionconsistencyDawid-Skenereferenceprogram-of-thoughtreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the reliability of LLM-generated reference code can be estimated without hidden tests or LLM judgment, by executing several candidate codes on many generated inputs and analyzing who agrees with whom. Concretely, ExeCRE samples ten brute-force reference codes, runs them on roughly three hundred schema-constrained inputs per problem, and turns each output into a binary signal: does this code match the plurality output on this input? A Dawid–Skene model then treats inputs as items and codes as noisy annotators, inferring a latent 'reliable behavior' state per input and a per-code error rate. Only a code whose inferred reliability clears a threshold is allowed to generate verification tests; otherwise the pipeline falls back to public tests. On 182 LiveCodeBench problems across four models, this raises Pass@1 (the fraction of problems solved on the first attempt) relative to the TextGrad-style baseline it is built on and cuts the number of misleading feedback cases on already-correct code roughly eightfold on GPT-5.2. The practical upshot is that generated code can police itself through execution agreement before it is trusted as an oracle.

What carries the argument

The load-bearing object is a binary execution-consistency matrix $U$, where $U_{ij}=1$ if candidate code $C_j$'s output on input $I_i$ equals the plurality output across all candidates on that input, and $U_{ij}=0$ otherwise; failed executions are NaN and excluded. On this matrix the paper runs the Dawid–Skene EM algorithm, originally an annotator-agreement model: inputs are items, codes are annotators, and each input has a latent binary state indicating which side of the partition is the reliable behavior. The inferred parameters $\hat{p}_q$ (class priors) and $\hat{\pi}^{(j)}_{qa}$ (per-code error rates) combine into a reliability score $\alpha_j=\hat{p}_1\hat{\pi}^{(j)}_{11}+\hat{p}_0\hat{\pi}^{(j)}_{00}$, the expected probability that code $C_j$ matches the latent consistency state. The machinery does the work of distinguishing a code that is correct but sometimes disagrees with a buggy majority from a code that is systematically on the wrong side, which plain voting cannot do.

What would settle it

Take a problem where seven of ten sampled reference codes implement the same wrong threshold (say, 'subtotal > 25' instead of 'subtotal >= 25'), generate inputs that rarely hit the boundary value, and run ExeCRE; if the selected code clears the 0.95 threshold yet fails the hidden tests that do hit the boundary, the reliability estimate has adopted an incorrect reference, contradicting the paper's robustness claim.

Watch

Extended reading notes

Core claim

The central discovery is that the latent variable separating reliable from unreliable behavior on each input can be recovered from a binary projection of execution outputs, and that the resulting per-code error rates track semantic correctness better than simple majority voting, LLM self-judgment, or fine-tuned judge models. In the paper's own comparison on DeepSeek-V3.2, ExeCRE identifies correct reference code with an F1 of 76.62, ahead of ExeCRE-Voting (70.83), the best judge-model baseline (72.64), and LLM-judgment baselines (below 63). The same filtering reduces misleading feedback on already-correct code from 113.2 to 14.0 problems on GPT-5.2 and from 30.0 to 1.2 on LLaMA-3.1-8B-Instruct over the 182-problem set. The paper also reports that replacing the schema-driven input generator with fuzzing or LLM-generated inputs keeps reliability identification F1 in the same band, and that applying the same reliability gate to code-based math reasoning on GSM8K lifts accuracy on Qwen2.5-Coder-32B from 90.28% to 92.96% and on LLaMA-3.1-8B from 75.16% to 78.85%.

Load-bearing premise

The estimate only works if the plurality of sampled candidate codes on a given input is usually the semantically correct behavior; if most candidates share the same bug that the generated inputs do not expose, the latent state can be defined by the wrong side and an incorrect code can receive a high reliability score.

Editorial extensions

If this is right

  • Reference codes can be screened before they generate tests, so self-correction iterates only on failures that are likely real; the misleading-feedback counts drop by roughly an order of magnitude across all four evaluated models.
  • The reliability gate transfers beyond LeetCode-style problems: on GSM8K program-of-thought reasoning, the same estimator decides when generated code should be trusted versus when the model should answer directly.
  • The estimator is not tied to a particular input generator; replacing schema-driven inputs with fuzzing or LLM-generated tests keeps reliability identification F1 between 0.79 and 0.81 on function-call problems.
  • The 0.95 adoption threshold is not a delicate knob: precision and recall trade off smoothly while F1 stays stable over a broad range, which makes the method practical without per-benchmark tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because the projection measures agreement with the panel's plurality rather than with ground truth, a single correct code surrounded by many wrong codes that share one bug will be scored low; weighting codes by an external prior (e.g., model confidence or prior pass rates) could make the estimator robust to that asymmetry.
  • Editorial extension: the same consistency matrix could be reused to estimate the reliability of generated test inputs themselves, not just reference codes, turning the method into a generic quality gate inside longer agentic coding loops.
  • Editorial extension: the paper's binary projection discards the identity of the outputs, so two codes can appear consistent yet differ on edge cases; coupling ExeCRE with an oracle-free disagreement measure that bounds pointwise error could give a certificate-like lower bound on correctness.
  • Editorial extension: since schema extraction fails on 3–19% of problems depending on the model, a perturbation-based or learned input constructor might extend the reliability gate to repository-level tasks where schemas are harder to extract.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ExeCRE, a framework that estimates the reliability of LLM-generated reference code without ground-truth tests. ExeCRE constructs many inputs from an LLM-extracted schema, executes up to 10 sampled candidate codes on these inputs, projects the raw outputs to binary majority-consistency labels, and fits a Dawid–Skene model to infer latent consistency states and per-code error rates. The resulting reliability score alpha_j is used to either select a trusted reference code for building oracle tests in a self-correction pipeline or to fall back to public tests when no candidate exceeds a threshold. The authors evaluate on 182 LiveCodeBench problems with GPT-5.2, DeepSeek-V3.2, Qwen2.5-Coder-32B-Instruct, and LLaMA-3.1-8B-Instruct, plus a smaller GSM8K program-of-thought study. They report improved Pass@1 relative to several baselines, substantially reduced misleading feedback, and competitive F1 for semantic reliability identification.

Significance. If the empirical findings hold, the paper makes a useful contribution: it provides a practical, execution-only method for estimating code reliability in self-correction pipelines and demonstrates, with hidden-test grounding, that consistency-based aggregation can reduce harmful feedback. The strongest evidence is the large reduction in misleading feedback on already-correct code (Table 6) and the RQ2 hidden-test evaluation showing that the reliability score has meaningful semantic discrimination. The paper is also transparent about its main limitation: a majority of wrong candidates that share a bug can define the 'reliable' side. Because the central mechanism is empirical rather than guaranteed, the value of the paper depends on how well the evaluation isolates the majority-wrong regime. The manuscript includes reproducible artifacts, a threshold-sensitivity analysis, an input-generator ablation, and careful contamination caveats, all of which strengthen the presentation.

major comments (3)
  1. [Sec. 3.2.2, Sec. 3.2.4, Sec. 7, Table 8] The binary projection U_ij = 1 iff O_ij equals the modal output of the sampled codes for input i, and the reliability score alpha_j = p1*pi^(j)_{1,1} + p0*pi^(j)_{0,0} rewards agreement with the inferred latent consistency state. This latent state is derived from the majority side of each input, so a buggy consensus can receive high reliability. Section 7 concedes this failure mode. Table 8 does not resolve it: it buckets instances by the number of wrong output groups, but it does not report whether the wrong candidates form the majority, nor does it report incorrect-adoption rates conditional on the modal output being semantically wrong. The one-group row (10.2% incorrect adoption) is precisely the regime where the majority is likely to be wrong, yet the table does not separate that case. Please add an analysis that isolates inputs or problems where the modal output is incorrect and report incorrect-adoption and selected-code accuracy in those cases. If the error is high in that regime, the claim that alpha tracks semantic reliability should be revised to an explicit diversity assumption; if the error is low, the analysis would substantially strengthen the paper.
  2. [Sec. 5.1, Table 2] The abstract and RQ1 claim that ExeCRE 'consistently improves' effectiveness, but the Qwen2.5-Coder-32B result does not clearly support this: ExeCRE is 32.5±1.4 versus TextGrad's 31.5±1.0, a difference within one standard deviation, and no significance tests or confidence intervals are reported anywhere in Table 2. The claim should be supported with paired significance tests across the five runs or softened to say that ExeCRE improves on the models where the difference is significant. Without this, 'consistently' is not established for the smallest two models in the main comparison.
  3. [Sec. 3.2.2, Sec. 5.2] The paragraph after the projection states that 'reliable behavior may correspond to the minority side when several codes share the same systematic mistake.' This is not operationalized in the DS fit: because U_ij labels the modal output as 1 for every input, a code that is always in the majority will have a high expected agreement with the latent state even if that majority is systematically wrong. The RQ2 evaluation provides empirical grounding, but it does not directly measure how often the inferred latent state is the incorrect side. Please add a diagnostic that reports, over the evaluation problems, the fraction of inputs where the modal output is incorrect and how often the selected code is correct on those inputs.
minor comments (6)
  1. [Table 2] In the manuscript text, the entries in Table 2 are concatenated without clear column separation (e.g., '62.7±1.899.6±1.874.2±3.534.9±3.1'); the final PDF must render this as a proper table with separate All/Easy/Medium/Hard columns.
  2. [Sec. 5.2, Tables 4 and 5] The RQ2 reliability-identification results are reported without standard deviations or confidence intervals; since the paper elsewhere emphasizes mean±std over five runs, the same reporting should be applied to the F1/Precision/Recall tables.
  3. [Sec. 3.2.2] The tie-breaking rule for the majority projection is described as 'arbitrarily selected'; this can affect U and therefore alpha, and the tie-breaking rule should be specified and preferably evaluated for sensitivity.
  4. [Sec. 5.2.1] The illustrative example reports scores 13/21 and 11/21 for voting, but does not show how these numbers are derived from the three inputs and seven codes; please include the computation.
  5. [Sec. 5.4.2] The input-generator ablation is limited to 65 function-call problems; please state whether this subset is representative of the full 182-problem set and whether the conclusion transfers to stdin-style problems.
  6. [Sec. 7] The statement that invalid schemas 'usually lead to low reliability scores rather than confident adoption' is not quantified; reporting the distribution of alpha for schema-failure cases would make the robustness argument more concrete.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ExeCRE's reliability score is an explicit operationalization of execution consistency, and the central claims are anchored by external hidden-test validation and Pass@1 comparisons.

full rationale

Walking the derivation chain: execution outputs O are projected to binary consistency labels U (Section 3.2.2), Dawid–Skene estimates latent consistency states and per-code error rates from U (Section 3.2.3), and the reliability score alpha_j = p_1 * pi^(j)_{1,1} + p_0 * pi^(j)_{0,0} is explicitly defined as the expected probability that C_j agrees with the inferred latent consistency states (Section 3.2.4). This makes alpha a function of the candidate codes themselves, but the paper does not claim alpha is semantic correctness by construction. Section 2 states 'Our goal is not to formally verify full semantic correctness, but to estimate whether a candidate code is reliable enough for subsequent use,' and Section 3.2.2 explicitly says the projection 'does not assume that the majority output is correct.' The claim that this consistency-based score tracks semantic correctness is an empirical hypothesis, tested in RQ2 against hidden-test ground truth (Tables 4 and 5) and through downstream Pass@1 comparisons (Table 2). Section 7 candidly acknowledges the failure mode where 'ExeCRE can adopt an incorrect reference code when many wrong candidates share a bug that generated inputs do not expose'; this is a validity limitation, not a circular derivation. There are no load-bearing self-citations (the only self-citation is the Zenodo artifact), no imported uniqueness theorem, no fitted parameter renamed as a prediction, and no known result merely renamed. The derivation is self-contained: the reliability estimate is computed from execution outputs and then independently evaluated against external correctness labels.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central claim rests on hand-set experimental budgets, a statistical identifiability assumption for the consistency projection, a domain assumption that majority agreement tracks correctness, and external hidden-test evaluation that supplies the only ground truth.

free parameters (4)
  • Reliability adoption threshold tau = 0.95 (chosen, not fitted)
    Used in Section 3.2.4 and Section 4.5 to decide whether a reference code is adopted; sensitivity analysis in Figure 4 shows stability, but the value is hand-set.
  • Candidate and input budgets = 10 codes, 5 schemas, 300 inputs per code
    Implementation choices in Section 4.5 that set the execution matrix size to approximately 10 by 300; these budgets affect how well consistency patterns are estimated.
  • Collection size cap = 20
    In Section 3.1, List, Group, and Matrix sizes are bounded at 20 to limit execution overhead, which may under-expose bugs that only appear on large inputs.
  • Schema discard failure threshold = 10 percent execution failures
    In Section 3.1, schema candidates are discarded if more than 10 percent of input-code executions fail, which changes which inputs enter the reliability matrix.
assumptions (5)
  • domain assumption Candidate codes are conditionally independent given the latent consistency state.
    Required for the Dawid-Skene likelihood in Section 3.2.3; violated when same-LLM candidates share bugs, as acknowledged in Section 7.
  • ad hoc to paper The binary majority-consistency projection preserves enough signal for the latent class to track semantic correctness.
    Section 3.2.2 defines U_ij from the majority output; if the majority is buggy, the latent state can flip to the buggy side. The paper provides only empirical robustness evidence, not a structural justification.
  • domain assumption LLM-generated brute-force reference codes are more likely correct than optimized solutions and remain executable on small inputs.
    Motivates the reference-code approach in Sections 2 and 3.3, building on ALGO; it is not proven for every problem or model.
  • domain assumption Schema extraction and generated execution inputs are valid and diverse enough to expose behavioral differences.
    Input construction in Section 3.1 reports schema success rates between 81.3 and 96.7 percent depending on the model; failed schemas weaken reliability estimation.
  • standard math Dawid-Skene EM converges to a meaningful maximum for binary labels with 10 codes and 300 inputs.
    The paper uses the standard EM formulation and Crowd-Kit, but provides no identifiability or convergence analysis for this particular binary projection.
invented entities (1)
  • Latent consistency state T_iq
    purpose: Hidden class per input in the Dawid-Skene EM; it defines which side of the binary output partition is treated as reliable behavior.
    The latent state is inferred only from the same execution outputs it is used to judge, so it has no falsifiable handle outside the consistency matrix.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation." pith.science (2026). https://pith.science/paper/QDK2P6P5

@misc{pith2026260804439,
  author       = {Pith},
  title        = {Pith review of: ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QDK2P6P5}},
  note         = {Machine review of arXiv:2608.04439}
}
read the original abstract

Large language models (LLMs) have made notable progress in code generation, but they still struggle on challenging tasks that require sophisticated algorithms or complex implementations. Recent methods increasingly use code execution as feedback, especially in self-correction pipelines that construct verification signals from generated code. However, these pipelines often depend on supervision signals whose reliability is unknown, which can introduce misleading feedback, unnecessary revisions, and incorrect final answers. To address this issue, we propose ExeCRE, an Execution-Consistency guided code Reliability Estimation framework. Instead of judging candidate code by tests or LLM feedback, ExeCRE estimates code reliability by statistically analyzing consistency patterns in execution outputs over a large number of randomly generated inputs. It collects execution outputs over generated inputs, projects them into consistency signals, and applies the Dawid-Skene model to infer latent code reliability. We integrate ExeCRE into self-correction for code generation. Experiments show that ExeCRE consistently improves both effectiveness and stability, while substantially reducing misleading correction signals. Under GPT-5.2 on LiveCodeBench, the average number of misleading feedback cases on already correct code drops from 113.2 with a representative self-correction baseline to 14.0 with ExeCRE. As an additional study, we apply the same reliability estimation strategy to code-based mathematical reasoning and observe similar benefits. These results suggest that ExeCRE enables more reliable use of generated code in execution-based pipelines.

Figures

Figures reproduced from arXiv: 2608.04439 by the authors.

Figure 1
Figure 1. Self-correction pipelines with and without ExeCRE. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of ExeCRE. Given a problem, ExeCRE constructs execution inputs and collects execution outputs from [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Pass@1 over self-correction iterations. 5.3.2 Iterative Self-Correction Stability [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Comparison between the original PoT pipeline and [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 11 canonical work pages

  1. [1]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al . 2024. A survey on evaluation of large language models.ACM transactions on intelligent systems and technology15, 3 (2024), 1–45. doi:10.1145/3641289

    doi:10.1145/3641289 2024

  2. [2]

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. Codet: Code generation with generated tests.arXiv preprint arXiv:2207.10397(2022). doi:10.48550/arXiv.2207.10397

    arXiv doi:10.48550/arxiv.2207.10397 2022

  3. [3]

    Mouxiang Chen, Zhongxin Liu, He Tao, Yusu Hong, David Lo, Xin Xia, and Jianling Sun. 2024. B4: Towards optimal assessment of plausible code solutions with plausible tests. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1693–1705. doi:10.1145/3691620.3695536

    arXiv 2024

  4. [4]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.arXiv preprint arXiv:2211.12588(2022). doi:10.48550/arXiv.2211. 12588

    arXiv doi:10.48550/arxiv.2211.12588 2022

  5. [5]

    Xiancai Chen, Zhengwei Tao, Kechi Zhang, Changzhi Zhou, Xinyu Zhang, Wanli Gu, Yuanpeng He, Mengdi Zhang, Xunliang Cai, Haiyan Zhao, et al. 2025. Revisit self-debugging with self-generated tests for code generation. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 18003–18023. doi:10.18653/...

    doi:10.18653/v1/2025.acl-long.881 2025

  6. [6]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168(2021). doi:10.48550/arXiv.2110.14168

    arXiv doi:10.48550/arxiv.2110.14168 2021

  7. [7]

    Giuseppe Crupi, Rosalia Tufano, and Gabriele Bavota. 2026. Improving Code Generation via Small Language Model-as-a-judge.arXiv preprint arXiv:2602.11911 (2026). doi:10.48550/arXiv.2602.11911

    work page doi:10.48550/arxiv.2602.11911 2026

  8. [8]

    Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estima- tion of observer error-rates using the EM algorithm.Journal of the Royal Statistical Society: Series C (Applied Statistics)28, 1 (1979), 20–28. doi:10.2307/2346806

    doi:10.2307/2346806 1979

Show all 61 references
  1. [9]

    Yangruibo Ding, Marcus J Min, Gail Kaiser, and Baishakhi Ray. 2024. CYCLE: Learning to Self-Refine the Code Generation.Proceedings of the ACM on Pro- gramming Languages8, OOPSLA1 (2024). doi:10.1145/3649825

    2024 doi

  2. [10]

    Jinhao Dong, Jun Sun, Wenjie Zhang, Jin Song Dong, and Dan Hao. 2025. Con- tested: Consistency-aided tested code generation with llm.Proceedings of the ACM on Software Engineering2, ISSTA, 596–617. doi:10.1145/3728902

    2025 doi

  3. [11]

    Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2024. Self-collaboration code genera- tion via chatgpt.ACM Transactions on Software Engineering and Methodology33, 7 (2024), 1–38. doi:10.1145/3672459

    2024 doi

  4. [12]

    Yiru Dong, Richong Zhang, Fanshuang Kong, and Si Chen. 2026. ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation. Zenodo. doi:10.5281/zenodo.21737670

    2026 doi

  5. [13]

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch

  6. [14]

    Andrew Estornell and Yang Liu. 2024. Multi-llm debate: Framework, principals, and interventions.Advances in Neural Information Processing Systems37 (2024), 28938–28964. doi:10.52202/079017-0911

    2024 doi

  7. [15]

    Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shu- vendu K Lahiri. 2024. Llm-based test-driven interactive code generation: User study and empirical evaluation.IEEE Transactions on Software Engineering50, 9 (2024), 2254–2268. doi:10.1109/TSE.2024.3428972

    2024

  8. [16]

    Zhiyu Fan, Haifeng Ruan, Sergey Mechtaev, and Abhik Roychoudhury. 2024. Oracle-guided program selection from large language models. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 628–640. doi:10.1145/3650212.3680308

    2024

  9. [17]

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In International conference on machine learning. PMLR, 10764–10799

    2023

  10. [18]

    Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve. 2024. Rlef: Grounding code llms in execution feedback with reinforcement learning.arXiv preprint arXiv:2410.02089(2024). doi:10.48550/arXiv.2410.02089

    2024 doi arXiv

  11. [19]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Tora: A tool-integrated reasoning agent for mathematical problem solving.arXiv preprint arXiv:2309.17452(2023). doi:10. 48550/arXiv.2309.17452

    2023 doi arXiv

  12. [20]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783 (2024). doi:10.48550/arXiv.2407.21783

    2024 doi arXiv

  13. [21]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yifan Wu, YK Li, et al. 2024. DeepSeek-Coder: when the large language model meets programming–the rise of code intelligence.arXiv preprint arXiv:2401.14196(2024). doi:10.48550/arXiv.2...

    2024 doi arXiv

  14. [23]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Mea- suring coding challenge competence with apps.arXiv preprint arXiv:2105.09938 (2021). doi:10.48550/arXiv.2105.09938

    2021 doi arXiv

  15. [24]

    Baizhou Huang, Shuai Lu, Xiaojun Wan, and Nan Duan. 2024. Enhancing large language models in coding through multi-perspective self-consistency. InProceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1429–1450. doi:1...

    2024 doi

  16. [25]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2.5-coder technical report.arXiv preprint arXiv:2409.12186(2024). doi:10.48550/arXiv.2409.12186

    2024 doi arXiv

  17. [26]

    Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024. Map- coder: Multi-agent code generation for competitive problem solving. InProceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 4912–4944. doi:10.18...

    2024 doi

  18. [27]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974(2024). doi:10...

    2024 doi arXiv

  19. [28]

    Xue Jiang, Yihong Dong, Yongding Tao, Huanyu Liu, Zhi Jin, and Ge Li. 2025. ROCODE: Integrating Backtracking Mechanism and Program Analysis in Large Language Models for Code Generation. InProceedings of the IEEE/ACM 47th International Conference on Software Engineering. 334–34...

    2025

  20. [29]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770(2023). doi:10.48550/ arXiv.2310.06770

    2023 doi arXiv

  21. [30]

    Dongwon Jung, Wenxuan Zhou, and Muhao Chen. 2025. Code Execution as Grounded Supervision for LLM Reasoning. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 24822–24833. doi:10.18653/ v1/2025.emnlp-main.1260

    2025

  22. [31]

    Hung Le, Hailin Chen, Amrita Saha, Akash Gokul, Doyen Sahoo, and Shafiq Joty. 2023. Codechain: Towards modular code generation through chain of self- revisions with representative sub-modules.arXiv preprint arXiv:2310.08992(2023). doi:10.48550/arXiv.2310.08992

    2023 doi arXiv

  23. [32]

    Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter. 2023. Chain of code: Reasoning with a language model-augmented code emulator.arXiv preprint arXiv:2312.04474(2023). doi:10.48550/arXiv.2312.04474

    2023 doi arXiv

  24. [33]

    Jierui Li, Hung Le, Yingbo Zhou, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. 2025. Codetree: Agent-guided tree search for code generation with large language models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computa...

    2025 doi

  25. [34]

    Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. 2024. More agents is all you need.arXiv preprint arXiv:2402.05120(2024). doi:10.48550/arXiv.2402.05120

    2024 doi arXiv

  26. [35]

    Qingyao Li, Xinyi Dai, Xiangyang Li, Weinan Zhang, Yasheng Wang, Ruiming Tang, and Yong Yu. 2025. Codeprm: Execution feedback-enhanced process reward model for code generation. InFindings of the Association for Computational Linguistics: ACL 2025. 8169–8182. doi:10.18653/v1/20...

    2025 doi

  27. [36]

    Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al . 2025. Deepseek- v3. 2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556(2025). doi:10.48550/arXiv.2512.02556

    2025 doi arXiv

  28. [37]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in neural information processing systems 36 (2023), 21558–21572. doi:10.52202/075280-0943

    2023 doi

  29. [38]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation.arXiv preprint arXiv:2402.19173(2024). doi:10.48550/arXiv.2402.19173

    2024 doi arXiv

  30. [39]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al

  31. [40]

    Xinji Mai, Haotian Xu, Weinong Wang, Yingying Zhang, Wenqiang Zhang, et al

  32. [41]

    Davide Molinelli, Alberto Martin-Lopez, Elliott Zackrone, Beyza Eken, Michael D Ernst, and Mauro Pezzè. 2025. Tratto: A neuro-symbolic approach to deriving axiomatic test oracles.Proceedings of the ACM on Software Engineering2, ISSTA (2025), 1887–1909. doi:10.1145/3728960

    2025 doi

  33. [42]

    Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. Clarifygpt: A framework for enhancing llm-based code generation via requirements clarification.Proceedings of the ACM on Software Engineering1, FSE (2024), 2332–2354. doi...

    2024 doi

  34. [43]

    OpenAI. 2025. Introducing GPT-5.2. https://openai.com/index/introducing-gpt- 5-2/

    2025

  35. [44]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems36 (2023), 53728–53741. doi:10.52202/075280-2338

    2023 doi

  36. [45]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems36 (2023), 8634–8652. doi:10. 52202/075280-0377

    2023

  37. [46]

    Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2022. Recitation-augmented language models.arXiv preprint arXiv:2210.01296(2022). doi:10.48550/arXiv.2210.01296

    2022 doi arXiv

  38. [47]

    Weixi Tong and Tianyi Zhang. 2024. Codejudge: Evaluating code generation with large language models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 20032–20051. doi:10.18653/v1/2024.emnlp-main. 1118

    2024 doi

  39. [48]

    Dmitry Ustalov, Nikita Pavlichenko, and Boris Tseitlin. 2024. Learning from Crowds with Crowd-Kit.Journal of Open Source Software9, 96 (2024), 6227. doi:10.21105/joss.06227

    2024 doi

  40. [49]

    Thomas Jean-Michel Valentin, Ardi Madadi, Gaetano Sapia, and Marcel Böhme

  41. [50]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171(2022). doi:10.48550/arXiv.2203.11171

    2022 doi arXiv

  42. [51]

    Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. 2023. Examining inter-consistency of large language models collaboration: An in-depth analysis via debate. InFindings of the Association for Computational Linguistics: EMNLP

    2023

  43. [52]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems36 (2023), 11809–11822. doi:10.52202/075280-0517

    2023 doi

  44. [53]

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. 2025. Optimizing generative ai by backpropagating language model feedback.Nature639, 8055 (2025), 609–616. doi:10.1038/s41586-025-08661-4

    2025 doi

  45. [54]

    Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li

  46. [55]

    Tianyi Zhang, Tao Yu, Tatsunori Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida Wang. 2023. Coder reviewer reranking for code generation. In International Conference on Machine Learning. PMLR, 41832–41846

    2023

  47. [56]

    doi:10.18653/v1/2023.findings-emnlp.508

    7572–7590. doi:10.18653/v1/2023.findings-emnlp.508

    2023 doi

  48. [60]

    Advances in Neural Information Processing Systems36 (2023), 54769–54784

    Algo: Synthesizing algorithmic programs with generated oracle verifiers. Advances in Neural Information Processing Systems36 (2023), 54769–54784. doi:10. 52202/075280-2389

    2023

  49. [62]

    Li Zhong, Zilong Wang, and Jingbo Shang. 2024. Debug like a human: A large language model debugger via verifying runtime execution step by step. InFindings of the Association for Computational Linguistics: ACL 2024. 851–870. doi:10.18653/ v1/2024.findings-acl.49 Received 2026-...

    2024

  50. [2023]

    doi:10.52202/075280-2019 ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation ASE ’26, October 12–16, 2026, Munich, Germany

    Self-refine: Iterative refinement with self-feedback.Advances in neural information processing systems36 (2023), 46534–46594. doi:10.52202/075280-2019 ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation ASE ’26, October 12–16, 2026, ...

    2023 doi

  51. [2024]

    InForty-first international conference on machine learning

    Improving factuality and reasoning in language models through multiagent debate. InForty-first international conference on machine learning

  52. [2025]

    InThe Thirty-ninth Annual Conference on Neural Information Processing Systems

    Agentic RL Scaling Law: Spontaneous Code Execution for Mathematical Problem Solving. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems

  53. [2026]

    InProceedings of the AAAI Conference on Artificial Intelligence, Vol

    Incoherence as oracle-less measure of error in LLM-based code generation. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 33305– 33313. doi:10.1609/aaai.v40i39.40616

    doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.