Pith. sign in

REVIEW 3 major objections 6 minor 5 cited by

Misread specifications, not weak code, are the fixable bottleneck in LLM code generation—rewriting the requirement lifts Pass@1 by 29.6%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

An LLM-based specification-alignment method that lifts the model's perceived requirements from generated code and applies ten alignment rules improves Pass@1 by about 30% over existing baselines.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The Pass@1 gains are likely real, but the paper hasn't established that specification alignment—rather than a weak filter on the same LLM's tests—is doing the work. the 3 major comments →

arxiv 2509.01313 v2 pith:E26EBO3L submitted 2025-09-01 cs.SE

Aligning Requirement for Large Language Model's Code Generation

Zhao Tian , Junjie Chen This is my paper
classification cs.SE
keywords specification alignmentcode generationlarge language modelsrequirements engineeringmulti-agent systemsspecification liftingPass@1competitive programming benchmarks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models often fail at programming tasks not because they cannot code but because they perceive the written specification differently from what was intended. The authors propose treating that specification misalignment as the root cause and fixing it before code is planned, implemented, or repaired. Their system, Specine, first checks whether initial code is correct; if not, it lifts the code into a structured requirement description that exposes what the model actually understood, then applies ten alignment rules borrowed from requirements engineering to rewrite the input specification. Across four LLMs and five competition benchmarks, the paper reports that this consistently beats ten baselines, with an average 29.60% relative improvement in Pass@1 over the strongest baseline. The practical claim is that correcting the model's perception of the problem is a more effective lever than debugging its output.

Core claim

The paper's central claim is that where LLM code generation goes wrong, the dominant error is usually upstream: the model forms an implicit, and partial, interpretation of the input specification, and every later stage—planning, implementation, repair—compounds that misreading. Specine makes the hidden interpretation explicit by lifting the initial code into a requirement DSL, a structured template listing background, functional requirements, inputs, outputs, examples, external APIs, and extra explanation. The lifted specification is compared with the original input, and ten pre-defined rules generate an aligned specification that the LLM can implement correctly. The authors report that this

What carries the argument

The load-bearing mechanism is the lifted specification: a DSL-structured description extracted from the generated code that shows what the LLM actually understood, including its functional reading of queries, input/output assumptions, and any omitted constraints. Against this, the aligner applies ten named alignment rules—Specification Background, Specification Purpose, Key Concepts, Input Requirements, Output Requirements, Examples with Explanations, Edge/Corner Cases, APIs, Error Handling Requirements, and Hints or Tips—each drawn from software requirements engineering. Selection among candidate alignments is driven by a hierarchical pass-ratio criterion: public tests first, LLM-generated

Load-bearing premise

The selection loop trusts the LLM-generated test set to tell which aligned specification is better, and the paper reports those tests are correct only 60.17% to 72.92% of the time—so a tester that is wrong in a systematic way could steer the whole alignment process toward code that passes its own flawed tests.

What would settle it

Compare Specine's private Pass@1 when the tester agent's generated tests are replaced one-for-one with canonical dataset tests while everything else stays fixed. If the gains largely survive, then test selection is not the active ingredient; if they shrink sharply, the headline 29.60% improvement depends on generated tests being right, exactly the weakest link Section 6.3 identifies.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Specification alignment is a pre-processing step, so it can sit in front of existing prompt-based and agent-based generators rather than competing with them.
  • The lifted specification gives developers a readable account of what the model misunderstood, which could also be used to improve the original problem statement for human readers.
  • The ten rules are not equally powerful: Examples with Explanations, Specification Purpose, and Output Requirements fix the most problems, giving specification authors a concrete priority list.
  • Gains continue to accumulate up to ten iterations, unlike baselines that plateau, implying alignment is a scalable search direction rather than a one-shot fix.
  • Even without public test cases, Specine still beats all ten baselines, suggesting the approach works in settings where no executable examples are provided.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same lift–compare–align loop should transfer to non-competitive code generation (web apps, data scripts) by swapping the DSL ingredients, but that transfer is untested.
  • The reported gains are bounded by the accuracy of the model-generated tests (60.17–72.92% correct), so a reasonable testable extension is to measure Specine's private Pass@1 as a function of generated-test accuracy and look for a cliff where the selection signal becomes noise.
  • If generated tests are the selection oracle, improving tester-agent quality—for example with type-checking or coverage guidance—should yield further gains, a direction the authors mention but do not implement.
  • The failure analysis attributing 23 of 60 failures to tester tests and 25 to misapplied rules suggests the largest remaining headroom is in rule application and test quality, not in the lifting step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Specine, an agent-based specification-alignment technique for LLM code generation. Specine first identifies misaligned input specifications using a coder/tester dual-agent check, then lifts the LLM-perceived specification from the generated code into a seven-attribute requirement DSL, and finally aligns the lifted specification with the original input specification using ten predefined rules in an iterative greedy search. The evaluation compares Specine with ten baselines on four LLMs and five competition-level benchmarks; the headline result is an average 29.60% improvement over the best baseline in Pass@1, with significance tests, ablations, and hyper-parameter sensitivity analyses. The paper also releases code and data.

Significance. If the reported gains are reproducible, the paper makes a useful contribution by shifting part of the code-generation pipeline from code repair to upstream requirement alignment. The evaluation is broad (20 LLM x benchmark subjects, 10 baselines, private held-out tests, repeated runs for Specine, ablations), and the idea of making the LLM's perceived specification explicit through a DSL is novel and potentially transferable. However, the causal claim that the improvements arise from specification alignment depends on the greedy search's selection signal (LLM-generated tests) being a valid proxy for correctness; the paper's own measurements show that signal is correct only 60-72% of the time. The final verdict therefore turns on whether the authors can validate that proxy and correct the statistical reporting.

major comments (3)
  1. [§3.3, §6.3, Table 2] The central mechanism depends on the greedy search selecting aligned specifications by the hierarchical pass ratio on public + LLM-generated tests. Section 6.3 reports generated-test accuracy of 60.17%-72.92% and defends the signal by a 'voting principle.' This defense assumes errors are random noise. Since the tester agent and the coder agent are the same base LLM and both process the same input specification, systematic misperception of the specification (the very situation Specine targets) can be shared: the LLM-generated tests will encode the same wrong interpretation, incorrect code will pass them, and the observed pass-ratio improvement is self-confirming rather than evidence of alignment. The private held-out tests evaluate only the final chosen code, not the selection process; the Specine_woPTC rows in Table 2 show the method operates without public tests. I ask for concrete vali
  2. [§5.1.2, §5.3.2] The reported Wilcoxon signed-rank p-values (all < 2.40×10^-7 and < 1.77×10^-7) are inconsistent with the stated comparison over 20 subjects. For an exact paired signed-rank test with n=20, the minimum possible one-sided p-value is 1/2^20 ≈ 9.5×10^-7 (two-sided ≈ 1.9×10^-6). The reported values are below these bounds. Please state the unit of analysis, the sample size, and whether the test is one- or two-sided. If the test is over individual problems (e.g., n=300 or 165), say so and justify the pairing; if it is over the 20 aggregate subjects, recompute the p-values. This is load-bearing for the 'statistically significant superiority' claims.
  3. [§6.2, §6.3] Figure 6 defines an alignment rule as 'effective' when it improves the test pass ratio of generated code. This criterion is identical to the selection signal whose correctness is questioned above. Given that Section 6.3 reports only 60.17%-72.92% accuracy of LLM-generated test cases and Section 6.1 attributes 23 of 60 failures to tester-generated tests, the ranking of rule contributions (e.g., Examples with Explanations being most effective) may be an artifact of the flawed test set. Please provide a sensitivity analysis: recompute rule effectiveness using only public tests, or using only LLM-generated test cases that are verified correct against canonical solutions. This would also strengthen the practical recommendation that developers should emphasize specific specification ingredients.
minor comments (6)
  1. [§4.1] The random sampling of 300 APPS and xCodeEval problems should report the random seed and confirm that the same sampled problems are used across all techniques and LLMs.
  2. [§3.1] The tester-agent prompt asks for 'a representative set' of test cases but the number is not fixed. Please report the distribution of the number of generated test cases per problem; this affects the pass-ratio resolution and the greedy search.
  3. [§1, §5.1.2] The '29.60% improvement' and ranges like '29.60%~93.55%' should be explicitly defined as relative improvement over the best baseline per subject, averaged across subjects, rather than absolute percentage-point differences. As written the reader cannot reconstruct this from Table 1.
  4. [References] Reference [11] lists 'John Doe' as the author; this placeholder should be replaced with the actual IEEE standard author/editor information.
  5. [§7] 'the standard derivations are only 0.008' should read 'standard deviations'; also 'demonstrats' in §5.3.2 is a typo.
  6. [Figure 2 and §3.3] The figure labels 'Public Pass Ratio' and 'Generated Pass Ratio' should be explicitly mapped to the 'primary pass ratio' and 'secondary pass ratio' terminology of Section 3.3.

Circularity Check

0 steps flagged

No significant circularity: headline Pass@1 is measured on held-out private tests, independent of the generated-test selection signal.

full rationale

The central effectiveness claim is evaluated on held-out private test cases: the paper samples three private test cases to serve as executable public tests and explicitly removes them from private evaluation ("we remove the three cases from private test cases for performance evaluation"). The greedy search in Section 3.3 selects aligned specifications using a hierarchical pass ratio over public and LLM-generated tests, but the headline Pass@1 and AvgPassRatio are computed on separate private tests, so the reported improvement is not defined by the selection signal. The ten alignment rules are pre-defined from requirements-engineering literature and are not learned from the benchmarks; no parameter is fitted to the target metric. The lifted specification is derived from generated code, but it serves only as an intermediate representation to prompt the aligner, not as the evaluation target. Self-citations (e.g., [51] for metrics and setup, and several earlier papers in related work) are methodological or contextual and do not carry the load-bearing premise that specification alignment improves code generation. The acknowledged 60.17%–72.92% accuracy of LLM-generated tests raises an internal-validity concern that the selection signal may be noisy or biased, but this is a possible threat to the causal mechanism rather than a circularity: the private-test results are independent of the generated tests, and the paper does not define success in terms of generated-test pass ratio. Therefore no circular step can be exhibited by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The paper's central mechanism rests on assumptions about what a failed test indicates, the fidelity of code-to-DSL lifting, and the reliability of LLM-generated tests as a selection signal. These are domain assumptions, not standard math.

free parameters (3)
  • Maximum alignment iterations N = 10
    Chosen by hand; Section 5.2 shows Specine's Pass@1 keeps rising as N grows, so the reported gains depend on allowing 10 iterations.
  • Sampled public test cases per APPS problem = 3
    Section 4.1: three private test cases are reassigned as public for APPS; this choice affects misalignment identification and the hierarchical selection signal.
  • LLM generation temperature = 0.8
    Section 4.4 sets temperature to 0.8 for all models; standard for sampling but a chosen constant that influences code diversity and repair.
axioms (4)
  • domain assumption Failure of generated code on public or LLM-generated tests indicates specification misalignment.
    Section 3.1 uses test failure as the trigger for alignment; incorrect code can also arise from planning, implementation, or test errors.
  • domain assumption The lifted DSL specification extracted from erroneous code faithfully represents the LLM's perceived specification.
    Section 3.2 assumes the lifter can reconstruct intent from low-level code; Section 6.1 reports 12 of 60 failure cases due to inaccurate lifted specs.
  • domain assumption LLM-generated test cases are correct often enough (above 50%) to serve as a selection signal.
    Section 6.3 reports 60.17%-72.92% accuracy and argues this still helps by voting; this is an empirical claim, not a guarantee.
  • domain assumption Ten alignment rules derived from software requirements engineering cover the relevant misalignment ingredients.
    Section 3.3 assumes the rule set is complete enough; the Specine_woAR variant shows large degradation, supporting relevance, but completeness is not proven.
invented entities (1)
  • Requirement specification DSL (seven attributes) no independent evidence
    purpose: Standardized representation of LLM-perceived specification lifted from code
    Introduced in Section 3.2; it is a designed schema, validated only inside this paper's pipeline, not against an external benchmark.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Aligning Requirement for Large Language Model's Code Generation." pith.science (2026). https://pith.science/paper/E26EBO3L

@misc{pith2026250901313,
  author       = {Pith},
  title        = {Pith review of: Aligning Requirement for Large Language Model's Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E26EBO3L}},
  note         = {Machine review of arXiv:2509.01313}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Code generation refers to the automatic generation of source code based on a given programming specification, which has garnered significant attention particularly with the advancement of large language models (LLMs). However, due to the inherent complexity of real-world problems, the LLM-generated code often fails to fully align with the provided specification. While state-of-the-art agent-based techniques have been proposed to enhance LLM code generation, they overlook the critical issue of specification perception, resulting in persistent misalignment issues. Given that accurate perception of programming specifications serves as the foundation of the LLM-based code generation paradigm, ensuring specification alignment is particularly crucial. In this work, we draw on software requirements engineering to propose Specine, a novel specification alignment technique for LLM code generation. Its key idea is to identify misaligned input specifications, lift LLM-perceived specifications, and align them to enhance the code generation performance of LLMs. Our comprehensive experiments on four state-of-the-art LLMs across five challenging competitive benchmarks by comparing with ten state-of-the-art baselines, demonstrate the effectiveness of Specine. For example, Specine outperforms the most effective baseline, achieving an average improvement of 29.60% across all subjects in terms of Pass@1.

Figures

Figures reproduced from arXiv: 2509.01313 by Junjie Chen, Zhao Tian.

Figure 1
Figure 1. Figure 1: An example from Codeforces with GPT-4o-mini [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of Specine [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Correspondence between alignment rules and key [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Influence of the number of iterations (𝑁) in terms of Pass@1 (↑) In addition, the improvement achieved by increasing the number of iterations in Specine surpasses that of all baselines. Specifically, as 𝑁 increases from 1 to 10, Specine achieves an average improve￾ment of 47.97% in terms of Pass@1 across all subjects, whereas other baselines show an average improvement of 13.27%∼45.40%. Similarly, for the … view at source ↗
Figure 6
Figure 6. Figure 6: Effectiveness of each alignment rule in Specine solution. Specifically, the accuracy of LLM-generated test cases is 60.17%∼72.92% on average across different LLMs and benchmarks, indicating that a non-negligible proportion of LLM-generated test cases contain errors. We acknowledge that the presence of incor￾rectly generated test cases is a general challenge for all LLM-based approaches, primarily due to in… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. An Iterative Test-and-Repair Framework for Competitive Code Generation

    cs.SE 2026-04 unverdicted novelty 7.0

    FixAudit improves LLM code generation on competitive programming benchmarks by training a shared model for iterative code-aware test generation and repair, achieving 35%+ gains in Pass@1 over baselines on the same 7B model.

  2. Programmers Are Poor and Overconfident Judges of LLM-Generated Assertions

    cs.SE 2026-07 accept novelty 6.5

    Developers are nearly three times more likely to correctly accept correct LLM assertions than to reject incorrect ones, and accompanying natural-language comments provide no net benefit and can increase overconfidence.

  3. REAgent: Requirement-Driven LLM Agents for Software Issue Resolution

    cs.SE 2026-04 unverdicted novelty 6.0

    REAgent improves LLM patch generation for software issues by 17.4% on average through automated construction, quality checking, and iterative refinement of structured issue-oriented requirements.

  4. An Iterative Test-and-Repair Framework for Competitive Code Generation

    cs.SE 2026-04 accept novelty 6.0

    A four-stage RL framework that iteratively repairs one code candidate via a code-reading Auditor and a Fixer outperforms larger models and CURE/Specine on competitive programming Pass@1.

  5. Bridging the Gap between User Intent and LLM: A Requirement Alignment Approach for Code Generation

    cs.SE 2026-04 unverdicted novelty 5.0

    REA-Coder improves LLM code generation by iteratively aligning requirements with model understanding and verifying outputs against the aligned spec.

Reference graph

Works this paper leans on

63 extracted references · 27 canonical work pages · cited by 4 Pith papers

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    Google AI. 2025. Gemini Developer API. https://ai.google.dev/gemini-api/docs/

    work page 2025

  3. [3]

    Anil Altinay, Joseph Nash, Taddeus Kroes, Prabhu Rajasekaran, Dixin Zhou, Adrian Dabrowski, David Gens, Yeoul Na, Stijn Volckaert, Cristiano Giuffrida, et al. 2020. BinRec: dynamic binary lifting and recompilation. InProceedings of the Fifteenth European Conference on Computer Systems. 1–16

    work page 2020

  4. [4]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732(2021)

    Pith/arXiv arXiv 2021

  5. [5]

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. CodeT: Code Generation with Generated Tests. InThe Eleventh International Conference on Learning Representations

    work page 2022

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  7. [7]

    Xiangping Chen, Xing Hu, Yuan Huang, He Jiang, Weixing Ji, Yanjie Jiang, Yanyan Jiang, Bo Liu, Hui Liu, Xiaochen Li, et al. 2025. Deep learning-based software engineering: progress, challenges, and opportunities.Science China Information Sciences68, 1 (2025), 111102

    work page 2025

  8. [8]

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2024. Teaching Large Language Models to Self-Debug. InThe Twelfth International Conference on Learning Representations

    work page 2024

  9. [9]

    Codeforces. 2025. https://codeforces.com/

    work page 2025

  10. [10]

    Google DeepMind. 2025. https://deepmind.google/

    work page 2025

  11. [11]

    John Doe. 2011. Recommended practice for software requirements specifications (ieee).IEEE, New York(2011)

    2011

  12. [12]

    Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, and Zhi Jin. 2024. Code- Score: Evaluating Code Generation by Learning Code Execution.ACM Trans. Softw. Eng. Methodol.(2024)

    work page 2024

  13. [13]

    Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2024. Self-collaboration code genera- tion via chatgpt.ACM Transactions on Software Engineering and Methodology33, 7 (2024), 1–38

    2024

  14. [14]

    EvalPlus. 2025. Code Sanitizer. https://github.com/evalplus/evalplus/blob/master/ evalplus/sanitize.py

    work page 2025

  15. [15]

    EvalPlus. 2025. EvalPlus Leaderboard. https://evalplus.github.io/leaderboard. html

    work page 2025

  16. [16]

    Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, et al. 2025. Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling.arXiv preprint arXiv:2507.23370(2025). ICSE ’26, April 12–18, 2026, Rio de Janeiro, Brazil Zhao Tian and Junjie Chen

    Pith/arXiv arXiv 2025

  17. [17]

    Shuzheng Gao, Cuiyun Gao, Wenchao Gu, and Michael Lyu. 2025. Search-based llms for code optimization. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 254–266

    work page 2025

  18. [18]

    Martin Glinz. 2000. Problems and deficiencies of UML as a requirements specifi- cation language. InTenth International Workshop on Software Specification and Design. IWSSD-10 2000. IEEE, 11–22

    work page 2000

  19. [19]

    Sol Greenspan, John Mylopoulos, and Alex Borgida. 1994. On formal requirements modeling languages: RML revisited. InProceedings of 16th International Conference on Software Engineering. IEEE, 135–147

    1994

  20. [20]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

    Pith/arXiv arXiv 2025

  21. [21]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence.arXiv preprint arXiv:2401.14196(2024)

    Pith/arXiv arXiv 2024

  22. [22]

    Qi Guo, Xiaofei Xie, Shangqing Liu, Ming Hu, Xiaohong Li, and Lei Bu. 2025. Intention is All You Need: Refining Your Code from Your Intention. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society

    2025

  23. [23]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). https://openreview.net/forum?id=sD...

    2021

  24. [24]

    Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al . 2023. Metagpt: Meta programming for multi-agent collaborative framework.arXiv preprint arXiv:2308.00352(2023)

    Pith/arXiv arXiv 2023

  25. [25]

    Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui. 2023. Agentcoder: Multi-agent-based code generation with iterative testing and opti- misation.arXiv preprint arXiv:2312.13010(2023)

    Pith/arXiv arXiv 2023

  26. [26]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186(2024)

    Pith/arXiv arXiv 2024

  27. [27]

    Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez

    Md. Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024. Map- Coder: Multi-Agent Code Generation for Competitive Problem Solving. InPro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers). 4912–4944

    2024

  28. [28]

    Naman Jain, Tianjun Zhang, Wei-Lin Chiang, Joseph E Gonzalez, Koushik Sen, and Ion Stoica. 2023. Improving code style for accurate code generation. In NeurIPS 2023 Workshop on Synthetic Data Generation with Generative AI

    work page 2023

  29. [29]

    Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. 2024. Self-planning code generation with large language models.ACM Transactions on Software Engineering and Methodology33, 7 (2024), 1–30

    2024

  30. [30]

    Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the effect of AI code generators on supporting novice learners in introductory programming. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–23

    work page 2023

  31. [31]

    Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. 2024. XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval. InProceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Lon...

    work page 2024

  32. [32]

    Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2025. Structured chain-of-thought prompt- ing for code generation.ACM Transactions on Software Engineering and Method- ology34, 2 (2025), 1–23

    2025

  33. [33]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode.Science378, 6624 (2022), 1092–1097

    2022

  34. [34]

    Feng Lin, Dong Jae Kim, and Tse-Hsun (Peter) Chen. 2025. SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents. In2025 IEEE/ACM 46th International Conference on Software Engineering (ICSE)

    2025

  35. [35]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems 36 (2023), 21558–21572

    2023

  36. [36]

    2012.Requirements engineering

    Linda A Macaulay. 2012.Requirements engineering. Springer Science & Business Media

    work page 2012

  37. [37]

    Noble Saji Mathews and Meiyappan Nagappan. 2024. Test-Driven Develop- ment and LLM-based Code Generation. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1583–1594

    work page 2024

  38. [38]

    Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. Clarifygpt: A framework for enhancing llm-based code generation via requirements clarification.Proceedings of the ACM on Software Engineering1, FSE (2024), 2332–2354

    2024

  39. [39]

    Nico Naus, Freek Verbeek, Sagar Atla, and Binoy Ravindran. 2024. Poster: For- mally Verified Binary Lifting to P-Code. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. 4973–4975

    work page 2024

  40. [40]

    Homepage of Specnie. 2025. https://github.com/tianzhaotju/Specine

    work page 2025

  41. [41]

    Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama

    Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2024. Is Self-Repair a Silver Bullet for Code Generation?. InThe Twelfth International Conference on Learning Representations

    2024

  42. [42]

    OpenAI. 2024. Introducing GPT-4o and more tools to ChatGPT free users. (2024)

    work page 2024

  43. [43]

    OpenAI. 2025. The most powerful platform for building AI products. https: //openai.com/api/

    work page 2025

  44. [44]

    Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The impact of ai on developer productivity: Evidence from github copilot.arXiv preprint arXiv:2302.06590(2023)

    Pith/arXiv arXiv 2023

  45. [45]

    LeetCode The World’s Leading Online Programming Learning Platform. 2025. https://leetcode.com/

    work page 2025

  46. [46]

    1996.Requirements engineering: An overview

    Klaus Pohl. 1996.Requirements engineering: An overview. Citeseer

    work page 1996

  47. [47]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530(2024)

    Pith/arXiv arXiv 2024

  48. [48]

    Zhao Tian, Junjie Chen, and Zhi Jin. 2023. Code difference guided adversarial example generation for deep code models. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 850–862

    2023

  49. [49]

    Zhao Tian, Junjie Chen, Dong Wang, Qihao Zhu, Xingyu Fan, and Lingming Zhang. [n. d.]. LEAM++: Learning for Selective Mutation Fault Construction. ACM Transactions on Software Engineering and Methodology([n. d.])

    work page

  50. [50]

    Zhao Tian, Junjie Chen, and Xiangyu Zhang. 2023. On-the-fly improving perfor- mance of deep code models via input denoising. In2023 38th IEEE/ACM Interna- tional Conference on Automated Software Engineering (ASE). IEEE, 560–572

    work page 2023

  51. [51]

    Zhao Tian, Junjie Chen, and Xiangyu Zhang. 2025. Fixing Large Language Models’ Specification Misunderstanding for Better Code Generation. In2025 IEEE/ACM 46th International Conference on Software Engineering (ICSE)

    2025

  52. [52]

    Zhao Tian, Junjie Chen, Qihao Zhu, Junjie Yang, and Lingming Zhang. 2022. Learning to construct better mutation faults. InProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–13

    2022

  53. [53]

    Zhao Tian, Honglin Shu, Dong Wang, Xuejie Cao, Yasutaka Kamei, and Junjie Chen. 2024. Large language models for equivalent mutant detection: How far are we?. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1733–1745

    2024

  54. [54]

    Bingyang Wei. 2024. Requirements are All You Need: From Requirements to Code with LLMs.arXiv preprint arXiv:2406.10101(2024)

    Pith/arXiv arXiv 2024

  55. [55]

    Frank Wilcoxon, S Katti, Roberta A Wilcox, et al . 1970. Critical values and probability levels for the Wilcoxon rank sum test and the Wilcoxon signed rank test.Selected tables in mathematical statistics1 (1970), 171–259

    work page 1970

  56. [56]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al

  57. [57]

    Zhen Yang, Fang Liu, Zhongxing Yu, Jacky Wai Keung, Jia Li, Shuo Liu, Yifan Hong, Xiaoxue Ma, Zhi Jin, and Ge Li. 2024. Exploring and unleashing the power of large language models in automated code translation.Proceedings of the ACM on Software Engineering1, FSE (2024), 1585–1608

    2024

  58. [58]

    Pamela Zave and Michael Jackson. 1997. Four dark corners of requirements engineering.ACM transactions on Software Engineering and Methodology (TOSEM) 6, 1 (1997), 1–30

    1997

  59. [59]

    Huan Zhang, Wei Cheng, Yuhan Wu, and Wei Hu. 2024. A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinement. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1319–1331

    2024

  60. [60]

    Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023. Self-Edit: Fault-Aware Code Editor for Code Generation.arXiv preprint arXiv:2305.04087(2023)

    Pith/arXiv arXiv 2023

  61. [61]

    Naifeng Zhang, Sanil Rao, Mike Franusich, and Franz Franchetti. 2025. Towards Semantics Lifting for Scientific Computing: A Case Study on FFT.arXiv preprint arXiv:2501.09201(2025)

    Pith/arXiv arXiv 2025

  62. [62]

    Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B Tenenbaum, and Chuang Gan. 2023. Planning with Large Language Models for Code Genera- tion. InThe Eleventh International Conference on Learning Representations

    work page 2023

  63. [2019]

    arXiv preprint arXiv:1910.03771(2019)

    Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771(2019)

    Pith/arXiv arXiv 1910

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.