Pith. sign in

REVIEW 5 major objections 5 minor 42 references

ConAIR:Consistency-Augmented Iterative Interaction Framework to Enhance the Reliability of Code Generation

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Consistency voting over LLM-generated tests is unreliable, so ConAIR has users correct the most suspicious test and iteratively co-evolves tests and code, lifting GPT-3.5's code-generation accuracy by 32.9% in about four interaction rounds.

desk verdict A genuinely new human-in-the-loop code-repair loop, but the load-bearing test-ranking heuristic goes unablated and the pseudocode has an argmax/argmin bug. read the letter →

arxiv 2411.15587 v1 pith:YXE44K2Y submitted 2024-11-23 cs.SE

Jinhao Dong , Jun Sun , Wenjie Zhang , Jin Song Dong , Dan Hao This is my paper · ORCID
classification cs.SE
keywords codegenerationlargelanguagemodelsconsistencyvotingLLM-generatedtestshuman-in-the-loopiterativerefinementco-evolutionofand
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Code-generation models are usually judged by consistency: generate many candidate programs and many tests, then pick the program that passes the most tests and shares behavior with the most peers. The paper's central observation is that the tests are produced by the same unreliable model, so this vote is built on a flawed indicator; the authors measure an average 37.7% error rate in LLM-generated tests on HumanEval, HumanEval+, and MBPP. ConAIR therefore inserts a cheap human step into the voting loop: in each round it uses code-to-test consistency voting to pick the test that the fewest candidates pass, asks the user to correct that test, repairs the candidates that fail it, and repeats. With an average of about four correction rounds, the framework lifts GPT-3.5's Pass@1 from 68.38% to 93.71% on HumanEval, outperforming both the previous consistency-based post-processor MPSC and the stronger general model GPT-4o. The claim is that consistency is valuable only when the indicator is trustworthy, and that a small amount of targeted user feedback can make it trustworthy enough to drive large reliability gains.

What carries the argument

The load-bearing object is a pair of opposite consistency votes. $Con_{c\to t}(t, C) = \sum_{c} Con(c,t)$ counts how many candidate programs satisfy a test, and the test with the lowest count is treated as the most likely wrong test and sent to the user for correction; $Con_{t\to c}$ checks which programs satisfy all corrected tests and selects the final output. The iterative rank-correct-fix algorithm (Algorithm 1) alternates these votes: correct one test per round, discard or repair programs that fail it, re-run the surviving programs against the remaining unknown tests, and stop when a program passes every test. The mechanism works by making both indicators more reliable over time, since better tests make code-to-test voting trustworthy and better programs make test-to-code voting trustworthy.

What would settle it

Re-run ConAIR on HumanEval with the ranking rule reversed — have the user correct the test that the most programs pass first instead of the least-passed one. If Pass@1 does not drop, the least-passed-test heuristic is not what drives the gain; similarly, if measuring the error rate of the tests selected by $Con_{c\to t}$ shows they are no more likely to be wrong than the average generated test, the ranking mechanism is not doing the claimed localization.

Watch

Extended reading notes

Core claim

ConAIR establishes that the failure mode of consistency-based code generation is not the voting mechanism but the consistency indicator. Because tests generated by the same LLM carry a high error rate (37.7% on average across HumanEval, HumanEval+, and MBPP), a buggy program can pass more generated tests than a correct one and therefore win the majority vote. The proposed solution is a rank-correct-fix co-evolution loop: rank tests by how many candidate programs pass them (code-to-test consistency, $Con_{c\to t}$), select the least-passed test as the most likely erroneous, have a user correct that test's expected output, then use the same LLM to repair every candidate that fails the corrected test. The loop terminates when a program passes all corrected tests or all tests have been corrected. The paper reports that this converges in an average of 4.53 interaction rounds and, built on GPT-3.5, improves Pass@1 by an average of 32.9% over the base model, 11.1% over MPSC, and 12.32% over GPT-4o, with further gains when the base model is GPT-4o or o1.

Load-bearing premise

The loop depends on the heuristic that the test passed by the fewest candidate programs is the one most likely to be erroneous, and on the assumption that a human (or surrogate) can correct that test accurately and cheaply; if the suspicious test is actually correct or the correction is wrong, the loop can discard good programs or propagate bad outputs.

Editorial extensions

If this is right

  • A weak base generator (GPT-3.5) plus a few corrected tests can beat a much stronger generator (GPT-4o) on HumanEval, HumanEval+, and MBPP, so test-quality correction can substitute for raw model scale.
  • The same loop generalizes to stronger base models: building ConAIR on GPT-4o lifts HumanEval Pass@1 from 84.67 to 97.59 with ground-truth-simulated user feedback, and building it on o1 reaches 100% Pass@1 on HumanEval.
  • User effort stays small in practice: an average of 4.53 interaction rounds across the three benchmarks, and a user study in which participants spent less time and rated the test-fixing setting easier than writing or fixing code directly.
  • The framework is cheap enough to be practical: ConAIR-o1 on HumanEval costs about $0.56 and 2.26 minutes per problem, roughly one-tenth the time and cost of using o1 alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The rank-correct-fix loop is a generic recipe for any domain where an LLM generates both candidate answers and its own verifiers (assertions, unit tests, formal specifications), not just code; the same least-trusted-verifier heuristic could select what a human should check.
  • Because the paper attributes the gain to correcting tests rather than to the initial code candidates, a testable corollary is that improving the test-generation step alone, without the code-repair step, would capture a large share of the improvement; the authors do not run this ablation.
  • The 37.7% test error rate suggests that pre-filtering generated tests by cross-model agreement before any human round could reduce the number of interaction rounds below four, which the paper does not explore.
  • ConAIR-o1's near-parity with ConAIR-GT on HumanEval indicates that a strong reasoning model can stand in for the human on well-specified problems, but its 48.3% error rate on MBPP warns that on under-specified problems the surrogate oracle itself needs the same kind of scrutiny.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes ConAIR, an interactive code-generation framework that iteratively ranks LLM-generated tests by how many generated code candidates pass them, asks the user (or a simulator) to correct the most inconsistent test, and then uses the corrected test to repair code candidates. It reports experiments on HumanEval, HumanEval+, and MBPP with two simulated feedback variants (ground-truth-based and OpenAI o1-based), showing large Pass@1 improvements over GPT-3.5, GPT-4o, and prior consistency post-processors such as MPSC, with an average of about 4.5 interaction rounds. A small user study and a time/cost analysis are also included.

Significance. The central idea—improving consistency-based code generation by having the user validate only the most suspicious generated tests—is practical and potentially valuable. The method has no fitted parameters, the evaluation uses external ground truth, and the reported gains over GPT-3.5 and MPSC are large. If the results are reproducible, ConAIR would be a useful lightweight post-processing technique. However, the paper's main efficiency claim depends on an unvalidated test-selection heuristic, and the manuscript contains several internal inconsistencies in the algorithm description and in the reported numbers. These issues currently prevent the results from being fully trusted.

major comments (5)
  1. [Section 3.2, Eq. (2); Section 3.3, Algorithm 1 line 3] The ranking rule is specified inconsistently. Eq. (2) defines Con_c->t as the count of codes that pass a test, and Section 3.2 states that the lower this value, the more likely the test is incorrect and therefore the test should be selected for correction. Algorithm 1 line 3, however, sets t_w <- argmax_{t in T_unk} Con_c->t(t, C), which selects the test passed by the most codes—the opposite rule. Since this ranking is the mechanism claimed to keep user effort to about four rounds, the paper must state which rule was actually implemented and correct the algorithm/text mismatch.
  2. [Section 3.2; Section 5.1.2, Table 1] The central assumption that the test passed by the fewest codes is the most likely to be erroneous—and hence the best target for user correction—is never validated. No ablation compares the proposed selection with random test selection, with the most-passed test, or with asking the user to correct all tests. Without such a comparison, the reported four-round efficiency cannot be attributed to consistency-based ranking; random selection might achieve similar results. This ablation is needed to support the paper's key practical claim.
  3. [Section 5.1.2, Table 1; Abstract; Introduction] The headline '32.9% improvement' is a relative improvement over the GPT-3.5 baseline, not an absolute gain. The absolute Pass@1 gains in Table 1 are +25.33, +21.25, and +17.10 on HumanEval, HumanEval+, and MBPP, with a mean of 21.2 percentage points; 32.9% only holds as a relative percentage. The abstract and introduction state '33% improvement' without this qualification. The paper should report both absolute and relative improvements to avoid ambiguity.
  4. [Section 5.3, Table 5; Section 5.1.2, Table 2] Some reported numbers are internally inconsistent. Table 5 lists ConAIR_o1 on MBPP with Pass@1 = 73.68, while Table 1 reports 74.30 for the same configuration. In addition, Section 5.1.2 credits ConAIR_o1 (GPT-4o-based) with raising HumanEval from 84.67 to 97.59, but Table 2 shows 97.59 is the ConAIR_GT result, while ConAIR_o1 reaches 94.96. These discrepancies must be reconciled, and the paper should clarify which subset or run each table reports.
  5. [Section 5.1; Tables 1 and 2] No variance, error bars, or multiple independent runs are reported for any experiment. Because all numbers come from stochastic LLM sampling and iterative fixing, the differences of a few percentage points between ConAIR and baselines may fall within run-to-run noise. The authors should report repeated-run statistics or explicitly justify and clearly state the single-run nature of the evaluation.
minor comments (5)
  1. [Section 3.3, Algorithm 1 header] The algorithm header swaps the variable names: it says 'Input: test case set C; code set T', while the text and the rest of the algorithm use T for tests and C for codes.
  2. [Section 5.1.2, Table 2] The MBPP column header reads 'Pass@1 Pass@1 Pass@5'; it should presumably be 'Pass@1 Pass@2 Pass@5'.
  3. [Section 5.2.1 and 5.2.2] The user study uses only 6 participants and 20 problems, and the reported Pass@1 for ConAIR and Writing Code are identical (90 vs 90). The Wilcoxon tests are applied only to time, not to correctness; the conclusion that ConAIR is more efficient than writing code should be framed accordingly.
  4. [Section 5.1.2] The text says that 'the results of ConAIR_o1 on HumanEval and HumanEval+ are similar,' but Table 1 shows Pass@1 values of 92.45 and 78.03 for these two datasets, which are not similar; this appears to be a misstatement.
  5. [Abstract; Section 5.2.2, Table 3] The abstract and introduction say the average number of interaction rounds is 4, while Table 3 reports an average of 4.53 across the three benchmarks; please reconcile the wording.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ConAIR's reported gains are empirical measurements against external ground-truth benchmarks, with no fitted parameter renamed as a prediction and no load-bearing self-citation.

full rationale

The paper's central claim is that ConAIR, a four-round interactive test-correction loop, improves code-generation accuracy over GPT-3.5, MPSC, and GPT-4o. The derivation chain is not circular: the method defines consistency voting via Eq. (1) and Eq. (2), selects a test for user correction, applies the correction, fixes code, and iterates. No parameter is fitted to the benchmark data and then reported as a prediction. The evaluation uses external ground-truth tests (HumanEval, HumanEval+, MBPP) and standard Pass@k metrics, so the measured 32.9% average improvement is an empirical outcome rather than an artifact of the method's own definitions. The fewest-passing-codes heuristic in Section 3.2 is an unvalidated design assumption, and the paper's own Limitation section acknowledges that performance depends on corrected-test quality; however, an unvalidated heuristic is a correctness/evaluation concern, not circularity. The noted mismatch that Algorithm 1 line 3 uses argmax over Con_c->t while the text says to select the most inconsistent test is an internal inconsistency or implementation bug, not a circular reduction. No self-citations are load-bearing, and no uniqueness theorem or imported ansatz is used to force the result. Therefore the paper is self-contained against external benchmarks and receives a circularity score of 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The framework does not introduce free parameters fitted to data or invented entities. It relies on four domain assumptions about the reliability of consistency voting, LLM repair ability, user correctness, and test sufficiency.

assumptions (4)
  • domain assumption The test passed by the fewest code candidates is the most likely to be erroneous (Con_{c->t} ranking).
    Used to select the test for user correction (Section 3.2, Eq. (2), Algorithm 1 line 3). No proof or stress test shows this holds when most generated codes are wrong.
  • domain assumption The base LLM can repair a code when given a corrected failing test.
    The fixing stage (Algorithm 1 lines 11-13) assumes that providing the corrected test as feedback is sufficient for the LLM to produce a passing fix. Empirical success is reported, but this is a load-bearing premise.
  • domain assumption User feedback on the selected test is reliable.
    The framework relies on the user correcting the selected test correctly. The paper's own Limitations section states that incorrectly corrected tests mislead the model.
  • domain assumption A code that passes all corrected tests is correct.
    The termination condition (Algorithm 1 lines 21-23) treats passing all corrected tests as evidence of correctness. With few LLM-generated tests, this is a weak certificate, as shown by the lower scores on HumanEval+.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ConAIR:Consistency-Augmented Iterative Interaction Framework to Enhance the Reliability of Code Generation." pith.science (2026). https://pith.science/paper/YXE44K2Y

@misc{pith2026241115587,
  author       = {Pith},
  title        = {Pith review of: ConAIR:Consistency-Augmented Iterative Interaction Framework to Enhance the Reliability of Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YXE44K2Y}},
  note         = {Machine review of arXiv:2411.15587}
}
read the original abstract

Code generation techniques generate code snippets automatically based on the problem requirements in natural language. Recently, large language models (LLMs) achieve the SOTA performance on code generation. However, LLMs still struggle at times to generate accurate code, which diminishes their promised efficiency as developers must spend significant effort evaluating and debugging the generated code. To improve the reliability and quality of the generated codes, researchers propose to leverage Consistency to obtain a better code based on generating and ranking multiple candidates. The existing approach is problematic as Consistency thinks a code is better when (1) the code pass more tests (inter-consistency) (2) more codes share the same behavior (intra-consistency). However, because the tests are also generated by LLMs, they could be wrong as well. As a result, majority voting based on testing results is unreliable. Relying solely on consistency is insufficient to address this issue; integrating user feedback is essential for effectively guiding consistency. We show that with minimal human effort, performance can be significantly enhanced. We propose Consistency-Augmented Iterative Interaction Framework to Enhance the Reliability of Code Generation, ConAIR, which is an approach that aims to improve the performance of a code generator through two distinctive ingredients, i.e., (1) lightweight user effort for validating the correctness of selected tests; and (2) a dynamic strategy for ranking, localizing and correcting multiple tests and codes. Overall, we propose a lightweight interaction framework that incorporates user feedback to correct identified tests and guide the iterative process. The iteration rounds are only 4 in average with the help of consistency. With only lightweight human efforts, we can achieve an improvement of 33% towards the base model.

Figures

Figures reproduced from arXiv: 2411.15587 by the authors.

Figure 1
Figure 1. ConAIR built on GPT-3.5 surpasses the SOTA general LLM GPT-4o and the SOTA post-processing technique MPSC on all datasets. Secondly, we propose a dynamic strategy to fix and maintain a set of consistent tests and code candidates. Specifically, ConAIR operates iteratively, using a rank-correct-fix co-evolution process in each iteration to gradually improve the quality of both code and tests. During this co-evolution … view at source ↗
Figure 2
Figure 2. The motivating example for limitations of relying solely on consistency (HumanEval/135) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The motivating example illustrating the consequences of ignoring the prerequisite required for [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The overview of ConAIR the correct code “n ** 2” only passes one test. Therefore, when test quality is low, majority voting based on testing results is unreliable, a limitation overlooked by current methods. Existing techniques[4, 12] that rely solely on consistency ar…
Figure 5
Figure 5. Figure 5: The co-evolution process of ConAIR reliability from the perspective of the codes and is denoted as Con𝑐→𝑡 (𝑡, C) = ∑︁ 𝑐 Con(𝑡, 𝑐) (2) The lower the consistency voting (Con𝑐→𝑡 ), the more likely it is that the test is incorrect. We rank the tests based on Con𝑐→𝑡 and sel…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 15 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Meet Claude

    anthropic 2024. Meet Claude. anthropic. https://www.anthropic.com/claude

    work page 2024

  3. [3]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  4. [4]

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023. CodeT: Code Generation with Generated Tests. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net. https://openreview.net/forum?id=ktrw68Cmu9c

    work page 2023

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

  6. [6]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017)

    2017

  7. [7]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021)

    arXiv 2021

  8. [8]

    Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2024. Self-collaboration code generation via chatgpt. ACM Transactions on Software Engineering and Methodology 33, 7 (2024), 1–38

    2024

Show all 42 references
  1. [9]

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2021. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arXiv:2103.10360 (2021)

    2021 arXiv

  2. [10]

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999 (2022)

    2022 arXiv

  3. [11]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024)

    2024 arXiv

  4. [12]

    Baizhou Huang, Shuai Lu, Xiaojun Wan, and Nan Duan. 2024. Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok...

    2024 doi

  5. [13]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 (2024)

    2024 arXiv

  6. [14]

    Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2312–2323

    2023

  7. [15]

    Nate Kushman and Regina Barzilay. 2013. Using semantic unification to generate regular expressions from natural language. North American Chapter of the Association for Computational Linguistics (NAACL)

    2013

  8. [16]

    Jia Li, Ge Li, Zhuo Li, Zhi Jin, Xing Hu, Kechi Zhang, and Zhiyi Fu. 2023. Codeeditor: Learning to edit source code with pre-trained models. ACM Transactions on Software Engineering and Methodology 32, 6 (2023), 1–22

    2023

  9. [17]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161 (2023)

    2023 arXiv

  10. [18]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2024). , Vol. 1, No. 1, Article . Pub...

    2024

  11. [19]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568 (2023)

    2023 arXiv

  12. [20]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems 36 (2024)

    2024

  13. [21]

    Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning . PMLR, 26106–26128

    2023

  14. [22]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 (2022)

    2022 arXiv

  15. [23]

    Don Norman. 2013. The design of everyday things: Revised and expanded edition . Basic books

    2013

  16. [24]

    Introducing GPT-4o and more tools to ChatGPT free users

    OpenAI 2023. Introducing GPT-4o and more tools to ChatGPT free users . OpenAI. https://openai.com/index/gpt-4o- and-more-tools-to-chatgpt-free/

    2023

  17. [25]

    GIntroducing OpenAI o1-preview

    OpenAI 2024. GIntroducing OpenAI o1-preview. OpenAI. https://openai.com/index/introducing-openai-o1-preview/

    2024

  18. [26]

    Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax networks for code generation and semantic parsing. arXiv preprint arXiv:1704.07535 (2017)

    2017 arXiv

  19. [27]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  20. [28]

    Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I Wang. 2022. Natural language to code translation with execution. arXiv preprint arXiv:2204.11454 (2022)

    2022 arXiv

  21. [29]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems. 8634–8652

    2023

  22. [30]

    Keith E Stanovich and Richard F West. 2000. Advancing the rationality debate. Behavioral and brain sciences 23, 5 (2000), 701–717

    2000

  23. [31]

    Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2022. Recitation-augmented language models. arXiv preprint arXiv:2210.01296 (2022)

    2022 arXiv

  24. [32]

    Zeyu Sun, Qihao Zhu, Yingfei Xiong, Yican Sun, Lili Mou, and Lu Zhang. 2020. Treegen: A tree-based transformer architecture for code generation. In Proceedings of the AAAI conference on artificial intelligence , Vol. 34. 8984–8991

    2020

  25. [33]

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al . 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 (2022)

    2022 arXiv

  26. [34]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  27. [35]

    Han Wang, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. 2024. Soft Self-Consistency Improves Language Model Agents. arXiv preprint arXiv:2402.13212 (2024)

    2024 arXiv

  28. [36]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, K...

    2023

  29. [37]

    Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. 2023. Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate. In Findings of the Association for Computational Linguistics: EMNLP 2023. 7572–7590

    2023

  30. [38]

    Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. arXiv preprint arXiv:1704.01696 (2017)

    2017 arXiv

  31. [39]

    Luke Zettlemoyer and Michael Collins. 2007. Online learning of relaxed CCG grammars for parsing to logical form. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) . 678–687

    2007

  32. [40]

    Luke S Zettlemoyer and Michael Collins. 2012. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. arXiv preprint arXiv:1207.1420 (2012)

    2012 arXiv

  33. [41]

    Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. 2023. Algo: Synthesizing algorithmic programs with generated oracle verifiers. Advances in Neural Information Processing Systems 36 (2023), 54769–54784

    2023

  34. [42]

    Tianyi Zhang, Tao Yu, Tatsunori Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida Wang. 2023. Coder reviewer reranking for code generation. In International Conference on Machine Learning . PMLR, 41832–41846. , Vol. 1, No. 1, Article . Publication date: November 2018

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.