Pith. sign in

REVIEW 4 major objections 5 minor 30 references

MineValiCoder claims that a closed loop of test-case mining and mutual code-test validation lets LLM-generated tests replace human-written tests in test-driven development, reaching 96.34% Pass@1 on HumanEval.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 04:39 UTC pith:R3SGWCMP

load-bearing objection A coherent, well-ablated automated TDD pipeline with strong reported gains, held back by single-run numbers, underspecified hyperparameters, and a self-validation step that is weaker than claimed. the 4 major comments →

arxiv 2607.22471 v1 pith:R3SGWCMP submitted 2026-07-24 cs.SE cs.AI

MineValiCoder: Reliable Code Generation with Test Case Quality Mining and Bipartite Graph-Based Mutual Validation

Zhen Zhao , Qihang Yang , Feifei Dai , Xiangfang Li , Bo Li This is my paper
classification cs.SE cs.AI
keywords Large language modelsCode generationTest-driven developmentSelf-consistencyBipartite graphMutual validationPass@1Automated testing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that fully automated, human-free test-driven development is attainable: a language model can write its own test cases, filter them for quality, use them to refine candidate code, and then select the best candidate by validating code and tests against each other. If this is right, the dependence on human-crafted test cases—the main obstacle to deploying TDD when only natural-language requirements exist—disappears. The authors support the claim with a three-module framework, MineValiCoder, and report Pass@1 scores of 96.34% on HumanEval, 87.40% on MBPP, 64.00% on APPS, and 51.33% on LiveCodeBench across four LLMs. On HumanEval and MBPP those scores come within roughly a percentage point of a ground-truth-test TDD baseline, suggesting self-generated tests can substitute for human-written ones.

Core claim

The central discovery is that code quality and test-case quality can be improved in a mutually reinforcing loop rather than treating tests as fixed inputs. TCQM samples diverse raw test suites from the problem statement and keeps only the test cases whose expected outputs are re-predicted consistently by the same model. Parallel TDD refinement then runs independent repair pipelines, accepting a repair only if it fixes the target failure without breaking any previously passing test. BiCoTeV builds a bipartite graph of code candidates and surviving tests, propagates reliability scores between the two node sets, and filters out candidates that pass low-credibility tests while failing high-credi

What carries the argument

The load-bearing mechanism is the three-module pipeline: (1) TCQM, which generates diverse test suites and filters them by self-consistency—retaining a test (input, expected output) only if the model re-predicts the same output; (2) Parallel TDD Refinement, which runs N independent repair branches with a monotonicity constraint (each new code must fix the target failed test and keep all previously passed tests passing); and (3) BiCoTeV, which builds a bipartite graph with code candidates on one side and test cases on the other, propagates scores bidirectionally (Equations 3–4), and applies a consistency-based filter (Equation 5) to remove overfitted candidates before selecting the highest-sc

Load-bearing premise

The load-bearing premise is that an LLM's consistency—re-predicting the same expected output for a generated input—is a reliable proxy for test correctness; if the model repeats the same wrong answer across its sampled suites, faulty tests survive and can mislead both refinement and final selection. The paper's own measurements show residual invalid-test rates of 6.7% to 15.9% across benchmarks, so the premise is approximate rather than exact.

What would settle it

On a fresh benchmark with known-correct reference implementations, run the TCQM filter alone and count how many retained tests are invalid; then group them by problem. If the problems with surviving invalid tests coincide with the problems where MineValiCoder selects the wrong final code, the mutual-validation stage is not compensating for filter error and the reliability claim fails. A second check: replace the self-validation prompt's validator with a different model and measure whether the retained-test validity rate changes; a large drop would show the filter depends on the generator's own

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Human-crafted tests become optional: the gap to a ground-truth-test TDD baseline shrinks to 0.16 points on HumanEval (96.34 vs. 96.50) and 0.80 points on MBPP (87.40 vs. 88.20).
  • The gains hold on LiveCodeBench, a benchmark designed to resist data contamination, which the authors read as evidence the improvement comes from the mechanism, not from memorization.
  • The framework lifts weaker open-source models substantially on HumanEval (e.g., a 4B-parameter backbone jumps from 76.22 direct prompting to 92.07 with MineValiCoder), suggesting the pipeline can amplify smaller models.
  • The monotonicity constraint and the overfitting filter each add measurable Pass@1 (robust filtering +4.24 on HumanEval, +4.67 on APPS), indicating stability mechanisms matter as much as test quality.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The self-consistency filter assumes independent errors: when the test generator is also the validator, systematic misunderstandings can be re-predicted consistently and survive. A natural extension is to validate with a different model or with an independent execution-based checker; the paper leaves this untested.
  • Because BiCoTeV scores tests by how many high-scoring codes pass them, the ranking encodes a majority opinion of the candidate pool. A test that is correct but catches a rare bug could be down-weighted if only one candidate passes it; the authors' overfitting rule partially addresses this but does not eliminate the majority bias.
  • The pipeline is not specific to Python or to function-level synthesis; the same mining-and-mutual-validation loop could be applied to other languages, to repository-scale tasks, or to specification-level validation, which the authors do not explore.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MineValiCoder, a fully automated test-driven code generation framework that replaces human-written tests with LLM-generated ones. The pipeline has three modules: TCQM generates N diverse test suites and filters each test case using a one-shot re-prediction check (Eq. 2); Parallel TDD Refinement runs N independent, monotonicity-constrained repair loops to produce code candidates (Algorithm 1); BiCoTeV builds a bipartite graph of code-test pass/fail relations and iteratively propagates reliability scores (Eqs. 3–4), then applies a consistency-based overfitting filter (Eq. 5). Experiments on HumanEval, MBPP, ET variants, APPS, and LiveCodeBench across GPT-4, Qwen2.5-Coder-7B, Qwen3-4B, and Llama3.1-8B report large Pass@1 gains, e.g., GPT-4 scores 96.34% on HumanEval and 64.00% on the APPS subset, within 0.2–0.8% of the PyCapsule ground-truth-test upper bound.

Significance. If the results hold, the framework would be a useful contribution: it provides a coherent closed-loop design, ablates each module, uses a weak model (Qwen3-4B) as the test generator to demonstrate that high-quality tests can come from a lightweight backbone, releases code, and compares explicitly against a ground-truth-test upper bound. However, the central premise — that the self-validation filter produces trustworthy test cases — is weakened by the paper’s own measurements in Table V, and the headline margins are not supported by repeated runs or a clearly defined, benchmark-comparable evaluation. The idea is promising, but the empirical evidence as presented does not yet establish the claimed reliability gains.

major comments (4)
  1. [Section III.C, Eq. (2) and Table V] Eq. (2) filters a generated test (I,O) by re-prompting the same LLM once with (Q,I) and keeping the test only if the re-predicted output equals O. This is not self-consistency as defined in Wang et al. [22], which requires agreement across multiple independent samples; one re-prediction can simply reproduce a correlated bias. The paper’s own Table V shows that after self-validation, 93.34% of HumanEval, 87.26% of MBPP, and 84.11% of APPS tests agree with ground-truth code, meaning 6.7%, 12.7%, and 15.9% of retained tests are invalid. With an average of 4–5 tests per suite, a large fraction of suites contain at least one invalid test. Algorithm 1 line 10 enforces monotonicity (P ⊆ P_new), so if the initial code happens to pass an invalid test, that wrong behavior is locked in. BiCoTeV then propagates scores from such tests through Eqs. (3)–(4), and Eq. (5) cannot flag overfitting when the
  2. [Section IV.C, Table IV] The ablation study is underspecified. Table IV reports the effect of enabling TCQM, Parallel TDD, and BiCoTeV independently, but when TCQM is disabled (rows with '–' under TCQM), the source of test cases is not stated. For example, the row with only BiCoTeV (89.63%) requires both code candidates and test cases; without TCQM, where do the raw tests come from, and without Parallel TDD, how are the code candidates produced? If raw LLM-generated tests are used, the BiCoTeV-only result is not a clean test of BiCoTeV; if ground-truth tests are substituted, the comparison is unfair. Without specifying the test and candidate sources in each ablation cell, the claimed marginal contributions of the modules are not interpretable.
  3. [Section IV.A and Table II] All results are reported as single numbers with no variance estimates. The paper’s motivation is that LLM outputs are stochastic, and the method itself relies on N=8 independent samples and varied temperatures, so single runs are not sufficient to support the claim of 'significantly outperforms' baselines. The authors should provide at least 3–5 repeated runs (or a bootstrap over problems) with mean ± std or confidence intervals, and ideally a significance test for the headline comparisons on HumanEval, MBPP, and APPS.
  4. [Section IV.A, APPS and LiveCodeBench subsets] The APPS evaluation uses a nonstandard balanced 150-problem subset (50 per difficulty level), and LiveCodeBench uses 150 randomly sampled problems. The paper does not state whether the baseline results (Direct, CoT, Self-Planning, Reflexion, CodeT, MapCoder) were re-run on these exact subsets or are taken from prior publications. If they are taken from prior work that used the full APPS benchmark or a different LiveCodeBench sample, the comparisons are invalid because the difficulty mix changes. Please clarify the provenance of each baseline number, and if necessary re-run baselines on the same evaluation protocol.
minor comments (5)
  1. [Abstract / Introduction] The phrase 'significantly outperforms' is used in the abstract and contributions, but no statistical significance is established; see the major comment on repeated runs. Please soften this claim or provide significance evidence.
  2. [Section II.B / References] PyCapsule is cited as [6] (Adnan et al.), but the reference list shows an arXiv self-debugging paper; it would be helpful to explicitly name PyCapsule in the reference entry and in the comparison in Table III, so the reader can verify the upper-bound numbers.
  3. [Section IV.C] Typo: 'competetion' should be 'competition' in the ablation motivation paragraph. Also, the sentence 'The result indicate that...' should be 'The results indicate...'.
  4. [Figure 3 / Section IV.D] Figure 3 shows that Llama3.1-8B peaks below N=8 and declines at higher N, but the text says 'increasing N consistently improves accuracy' for 'most models'. The non-monotonicity for Llama is acknowledged, but the figure would benefit from error bars or a clearer statement about per-model optimal N.
  5. [Section III.E, Eq. (5)] The consistency-based overfitting criterion is described as 'reveals that the code performs well on less reliable test cases but fails on highly credible ones', but no formal justification is given. A brief explanation or a small illustrative example would improve readability and clarify why the max/min comparison is chosen over alternative order statistics.

Circularity Check

0 steps flagged

No significant circularity: core pipeline outputs are benchmarked against ground-truth tests; the self-validation weakness is a robustness concern, not a definitional or fitted-input circularity.

full rationale

Walking the paper's derivation chain: TCQM filters raw tests with Eq. 2, Parallel TDD uses the retained tests as feedback, BiCoTeV ranks candidates by a bipartite mutual-scoring recursion, and final Pass@1 is computed against ground-truth benchmark tests. No predicted quantity is defined in terms of the empirical target, and no fitted value is renamed as a prediction. Eq. 2 is a heuristic validity filter, not a derivation: its output quality is audited externally in Table V against ground-truth code, so residual test errors are measured rather than assumed away. Eqs. 3-4 define a synchronous fixed-point scoring algorithm; iterating to a fixed point is not circular inference, and the final selected code is evaluated independently on held-out ground-truth tests. The paper contains no load-bearing self-citation: references such as [22] and [26] are independent prior work, and no uniqueness theorem is imported. The main substantive weakness—using the same LLM to generate and validate tests—is a correlated-error robustness concern that the paper itself partially acknowledges (Section III.E admits residual stochasticity; Table V reports 6.66%–15.89% invalid retained tests). That weakens the strength of the 'reliable tests' framing but does not make the framework's results equivalent to its inputs by construction. N=8 is a hyperparameter selected for cost-accuracy tradeoff and is not a predicted result. Overall, no significant circularity is present.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The framework's load-bearing tuning knobs are N=8 (explicitly fitted on HumanEval in Fig. 3 and then used for all headline results), the unreported damping factor d in Eqs. 3-4, the unreported R_max in Algorithm 1, and hand-chosen decoding temperatures. The central claim rests on four domain assumptions about LLM behavior (self-consistency implies correctness; consensus implies reliability; monotonicity on retained tests is safe; the Eq. 5 score-ordering identifies overfitting) plus an unanalyzed convergence premise for the scoring iteration. No new physical entities are introduced; the bipartite graph is a modeling construct.

free parameters (4)
  • Sampling size N = 8
    Number of test-suite generation rounds, parallel TDD pipelines, and code candidates. Determined by systematic experiments on HumanEval as an accuracy-cost tradeoff (Fig. 3, Section IV.D), then used for all headline results.
  • Damping factor d (BiCoTeV) = not reported
    Controls score retention in Eqs. 3-4; its value never appears in the paper.
  • Max retries R_max (Algorithm 1) = not reported
    Per-failed-test repair bound in Algorithm 1, line 6; numeric value not specified.
  • Decoding temperatures = code: [0.8, 1.2, 1.5]; optimization: 0.6; top_p: 0.9
    Hand-chosen to balance exploration and stability; how the three code-generation temperatures map onto the N=8 branches is not specified (Section IV.A).
axioms (5)
  • domain assumption Self-consistency implies correctness: a test case (I, O) is valid if the same LLM re-predicts O from I (Eq. 2)
    Load-bearing premise of TCQM. Table V shows 6.7-15.9% of retained tests still fail ground-truth code, so the premise holds only approximately; correlated LLM errors would defeat it. The paper acknowledges residual errors in Section III.E.
  • domain assumption Consensus tracks truth: in BiCoTeV, tests passed by high-scoring codes and codes passing high-scoring tests are the reliable ones (Eqs. 3-4)
    Requires the candidate pool to contain enough correct code that majority behavior is correct. If all N=8 candidates share a systematic test-generation bias, the mutual reinforcement fixed point can certify a shared mistake.
  • domain assumption Monotonicity is safe: accepting a repair only if it fixes the target test without regressing previously passed tests (Algorithm 1, line 10) preserves correctness
    Only safe if the retained tests are correct; an erroneous retained test that code happens to pass becomes a locked constraint that biases all later repairs.
  • ad hoc to paper The Eq. 5 criterion identifies overfitting: max score over failed tests > min score over passed tests implies the candidate is overfit
    A heuristic introduced for this framework without derivation. In principle it can discard a correct candidate that fails one high-score erroneous test while passing moderate-score correct tests. Empirically the ablation (Table VI) shows a net gain.
  • standard math The synchronous mutual-reinforcement iteration with global normalization converges to a meaningful ranking (Eqs. 3-4)
    The paper asserts 'iterates until convergence' without analyzing convergence for this exact formulation (synchronous updates, joint normalization over both node sets, damping), and the damping value d is never reported.

pith-pipeline@v1.3.0-alltime-deepseek · 15667 in / 30965 out tokens · 303135 ms · 2026-08-01T04:39:34.493870+00:00 · methodology

0 comments
read the original abstract

Large Language Model (LLM)-based Test-Driven Development (TDD) has advanced automated code generation. However, existing approaches depend heavily on human-crafted test cases and cannot operate effectively when only natural-language requirements are available. Although recent work enables automatic test generation, it often overlooks the inherent stochasticity of LLMs, leading to two key defects: faulty tests generate misleading feedback that distorts code optimization, while mixed-quality test cases produce conflicting evaluation signals that hinder reliable code selection. To address these challenges, we propose MineValiCoder, a collaborative closed-loop TDD framework based on the mutual reinforcement of test-case quality and code quality. MineValiCoder comprises three modules. The Test Case Quality Mining (TCQM) module filters faulty test cases through self-validation, providing reliable optimization supervision. The Parallel TDD Refinement module iteratively optimizes code and generates diverse high-quality code candidates using validated test-case feedback. The Bipartite Graph-Based Code-Test Mutual Validation (BiCoTeV) module dynamically models code-test interactions and performs mutual validation scoring for stable and reliable optimal-code selection. Extensive evaluations across four LLMs and mainstream benchmarks show that MineValiCoder significantly outperforms state-of-the-art methods. Specifically, it achieves Pass@1 scores of 96.34% on HumanEval, 87.40% on MBPP, 64.00% on APPS, and 51.33% on LiveCodeBench. These results demonstrate the effectiveness of MineValiCoder in mitigating LLM stochasticity and improving the reliability of automated code generation.

Figures

Figures reproduced from arXiv: 2607.22471 by Bo Li, Feifei Dai, Qihang Yang, Xiangfang Li, Zhen Zhao.

Figure 1
Figure 1. Figure 1: Key challenges in automated TDD with LLM-generated [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of MineValiCoder. The framework first employs TCQM to generate validated test cases, followed by Parallel [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance sensitivity to the sampling size [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 14 linked inside Pith

  1. [1]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remezet al., “Code llama: Open foundation models for code,”arXiv preprint arXiv:2308.12950, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Codet5+: Open code large language models for code un- derstanding and generation,

    Y . Wang, H. Le, A. Gotmare, N. Bui, J. Li, and S. Hoi, “Codet5+: Open code large language models for code un- derstanding and generation,” inProceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp. 1069–1088

    2023

  3. [3]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    Llama: Open and efficient founda- tion language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient founda- tion language models,”arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Test-driven develop- ment and llm-based code generation,

    N. S. Mathews and M. Nagappan, “Test-driven develop- ment and llm-based code generation,” inProceedings of the 39th IEEE/ACM International Conference on Auto- mated Software Engineering, 2024, pp. 1583–1594

    2024

  6. [6]

    Large language model guided self-debugging code generation,

    M. Adnan, Z. Xu, and C. C. Kuhn, “Large language model guided self-debugging code generation,”arXiv preprint arXiv:2502.02928, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Large language models as test case generators: Performance evaluation and enhancement,

    K. Li and Y . Yuan, “Large language models as test case generators: Performance evaluation and enhancement,” arXiv preprint arXiv:2404.13340, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Test wars: A comparative study of sbst, symbolic execution, and llm-based approaches to unit test generation,

    A. Abdullin, P. Derakhshanfar, and A. Panichella, “Test wars: A comparative study of sbst, symbolic execution, and llm-based approaches to unit test generation,” in 2025 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 2025, pp. 221–232

    2025

  9. [9]

    Valtest: Automated validation of language model generated test cases,

    H. Taherkhani and H. Hemmati, “Valtest: Automated validation of language model generated test cases,”arXiv preprint arXiv:2411.08254, 2024

    arXiv 2024

  10. [10]

    Codecon- tests+: High-quality test case generation for competitive programming,

    Z. Wang, S. Liu, Y . Sun, H. Li, and K. Shen, “Codecon- tests+: High-quality test case generation for competitive programming,”arXiv preprint arXiv:2506.05817, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Evaluating large language models trained on code,

    M. Chen, “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  12. [12]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Leet al., “Program synthesis with large language models,”arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  13. [13]

    Measuring coding challenge competence with apps,

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song et al., “Measuring coding challenge competence with apps,”arXiv preprint arXiv:2105.09938, 2021

    Pith/arXiv arXiv 2021

  14. [14]

    Livecodebench: Holistic and contamination free evaluation of large lan- guage models for code,

    N. Jain, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large lan- guage models for code,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 58 791– 58 831

    2025

  15. [15]

    Starcoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chimet al., “Starcoder: may the source be with you!”arXiv preprint arXiv:2305.06161, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Self-refine: Iterative refinement with self- feedback,

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y . Yanget al., “Self-refine: Iterative refinement with self- feedback,”Advances in Neural Information Processing Systems, vol. 36, pp. 46 534–46 594, 2023

    2023

  17. [17]

    Blueprint2code: a multi-agent pipeline for reliable code generation via blueprint planning and repair,

    K. Mao, B. Hu, R. Lin, Z. Li, G. Lu, and Z. Zhang, “Blueprint2code: a multi-agent pipeline for reliable code generation via blueprint planning and repair,”Frontiers in Artificial Intelligence, vol. 8, p. 1660912, 2025

    2025

  18. [18]

    Codetree: Agent-guided tree search for code generation with large language models,

    J. Li, H. Le, Y . Zhou, C. Xiong, S. Savarese, and D. Sahoo, “Codetree: Agent-guided tree search for code generation with large language models,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguis- tics: Human Language Technologies (Volume 1: Long Papers), 2025, pp. 3711–3726

    2025

  19. [19]

    Debatecoder: Towards collective intelligence of llms via test case driven llm debate for code generation,

    J. Chen, K. Du, X. Dai, W. Zhang, X. Wang, Y . Wang, R. Tang, W. Zhang, and Y . Yu, “Debatecoder: Towards collective intelligence of llms via test case driven llm debate for code generation,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 12 055– 12 065

    2025

  20. [20]

    Mapcoder: Multi-agent code generation for competitive problem solving,

    M. A. Islam, M. E. Ali, and M. R. Parvez, “Mapcoder: Multi-agent code generation for competitive problem solving,”arXiv preprint arXiv:2405.11403, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Divide-and-conquer meets consensus: Unleash- ing the power of functions in code generation,

    J. Chen, H. Tang, Z. Chu, Q. Chen, Z. Wang, M. Liu, and B. Qin, “Divide-and-conquer meets consensus: Unleash- ing the power of functions in code generation,”Advances in Neural Information Processing Systems, vol. 37, pp. 67 061–67 105, 2024

    2024

  22. [22]

    Self- consistency improves chain of thought reasoning in lan- guage models,

    X. Wang, J. Wei, D. Schuurmans, Q. V . Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self- consistency improves chain of thought reasoning in lan- guage models,” inThe Eleventh International Conference on Learning Representations, 2023

    2023

  23. [23]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  24. [24]

    Self-planning code generation with large language models,

    X. Jiang, Y . Dong, L. Wang, Z. Fang, Q. Shang, G. Li, Z. Jin, and W. Jiao, “Self-planning code generation with large language models,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 7, pp. 1–30, 2024

    2024

  25. [25]

    Reflexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,”Advances in Neural Information Processing Systems, vol. 36, pp. 8634–8652, 2023

    2023

  26. [26]

    Codet: Code generation with gener- ated tests,

    B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “Codet: Code generation with gener- ated tests,” inThe Eleventh International Conference on Learning Representations, 2023

    2023

  27. [27]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,”Advances in Neural Information Processing Systems, vol. 36, pp. 21 558–21 572, 2023

    2023

  28. [28]

    Qwen2. 5-coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Luet al., “Qwen2. 5-coder technical report,”arXiv preprint arXiv:2409.12186, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Ka- dian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025