Pith. sign in

REVIEW 4 major objections 6 minor 76 references

Code models know a repair instruction is wrong and still follow it, permanently corrupting the code.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 17:44 UTC pith:RT7OHABV

load-bearing objection Solid empirical chain showing know-vs-follow asymmetry under wrong repair instructions; the irrecoverability claim is real but partly confounded by the self-repair ceiling already measured in RQ2. the 4 major comments →

arxiv 2607.04537 v1 pith:RT7OHABV submitted 2026-07-05 cs.SE cs.AI

Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse

Raj Jaiswal , Anany Singh Divy , Savar Bhasin , Adi Bajpai , Tanuja Ganu , Rajiv Ratn Shah This is my paper
classification cs.SE cs.AI
keywords code language modelsblind obedienceinstruction followingautomated program repairghost errorsiterative repairirrecoverable damageRunBugRun
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Benchmarks for code language models assume the instructions they receive are correct. This paper studies the opposite case: what happens when a model is given a plausible but wrong diagnosis of a bug and objective test failures contradict that diagnosis. Across five models and a progressive chain of four experiments on algorithmic Python repair problems, the authors find a consistent pattern they call Blind Obedience. Models correctly label the instruction as incorrect when asked only to evaluate it, yet in the generation setting they follow it anyway, patching the wrong location and introducing additional structural faults the authors call Ghost Errors. Those faults compound under iterative incorrect guidance. When the same models are later given correct self-guided repair from the corrupted state, most problems never recover past the original buggy baseline; elevated reasoning budgets do not reverse the damage and often cause models to spend their token budget thinking without emitting code. The work argues that pass rates alone cannot see this semantic drift, and that production coding agents inherit a behavioral liability whenever human or agent instructions are wrong.

Core claim

Code language models exhibit Blind Obedience: they correctly identify an incorrect repair instruction as wrong in an evaluator role, yet systematically follow that same instruction when generating a patch, introducing Ghost Errors beyond the original bug. The resulting semantic corruption is frequently irrecoverable under subsequent correct self-guided iterative repair, and extended reasoning does not reverse it.

What carries the argument

A progressive four-experiment chain (RQ1–RQ4) on the RunBugRun Python subset that contrasts correct instructions, deliberately misdirected incorrect instructions, and self-thinking, then measures Ghost Error accumulation under iterative wrong guidance and the Irrecoverable Damage Rate under later correct self-repair, with McNemar tests confirming the evaluator–generator asymmetry.

Load-bearing premise

The claim rests on the idea that single-function algorithmic Python problems, deterministic unit tests, and one-paragraph location-misdirected wrong instructions stand in well enough for the messy misdiagnoses of real multi-file production repair.

What would settle it

Re-run the same RQ1–RQ4 protocol on multi-file production repair tasks with realistic human misdiagnoses: if models then resist wrong instructions once tests fail, or if correct self-repair routinely restores the original buggy baseline after Ghost Errors, Blind Obedience and irrecoverable collapse do not transfer.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper argues that code LLMs exhibit Blind Obedience: they correctly label incorrect repair instructions as wrong in an evaluator role, yet systematically follow those same instructions when generating patches. Using RunBugRun’s 538 deterministic Python problems and five models, four progressive experiments (RQ1–RQ4) show that incorrect instructions degrade pass rates more than self-thinking alone (RQ1; McNemar T1 vs T2, all p<0.001), that self-guided iterative repair has a low recovery ceiling (RQ2), that iterative wrong instructions compound Ghost Errors with flat escape rates (RQ3), and that subsequent correct self-repair often fails to restore the corrupted state (RQ4). Extended reasoning does not reverse the pattern. The authors frame this as a behavioral property invisible to standard pass-rate benchmarks and consequential for production coding agents.

Significance. If the causal story holds, the work identifies a deployment-relevant failure mode that existing instruction-following benchmarks structurally cannot see, because they assume correct instructions. Strengths include a progressive, falsifiable design; objective execution oracles; an independent awareness probe decoupled from generation; paired McNemar tests; public prompts/code; and explicit recovery ceilings under matched self-repair protocols. The evaluator–generator asymmetry is a crisp empirical contribution even if external validity to multi-file agents remains open. The main scientific value is behavioral measurement of instruction–evidence conflict in code repair, not a new model architecture.

major comments (4)
  1. [§3.4, §5.4, Figs. 7–9] §3.4 / §5.4 / Figs. 8–9: The central claim that Blind Obedience drives irrecoverable semantic collapse rests on RQ4’s lower recovery relative to RQ2. RQ2 and RQ4 use the same self-repair protocol but different problem populations (T3 single-pass failures vs. RQ3 damage-filtered obedient cases; Fig. 1) and different start states. Without a matched control that runs the same five self-guided passes on the RQ3-eligible problems from the original buggy patch (or from a non-adversarial edit of comparable magnitude), the Irrecoverable Damage Rate conflates Ghost-Error corruption with problems already near the model’s self-repair ceiling (large unresolved bars in Fig. 7). The plateauing curves in Figs. 8–9 look structurally like RQ2, so causal attribution to Blind Obedience needs this control or an explicit within-problem comparison of recovery probability from clean vs. ghost-corrupted starts.
  2. [§3.4 and Appendix RQ4] §3.4 vs Appendix RQ4 skeleton: Recovery/irrecoverability is defined inconsistently. The main text measures recovery against the buggy-patch baseline (tests passed by the original dataset patch) and calls failure to cross that baseline irrecoverable; the appendix operationalizes escape as failed==0 and errored==0 (full pass) and success=true as irrecoverable. These are not equivalent. Please state a single primary criterion, report both full-pass recovery and baseline-crossing rates, and ensure figures/tables use the stated metric. This is load-bearing for the Irrecoverable Damage Rate quantity.
  3. [§3.3, §5.3, §6.3] §3.3 / §5.3: Ghost Errors are central to the narrative but under-specified as a measured quantity. Damage is filtered by increased failed+errored counts relative to the buggy baseline, and qualitative examples show wrong-location edits, yet the paper does not systematically distinguish (i) residual original bugs, (ii) new syntactic/runtime faults, and (iii) semantic drift that preserves some tests while breaking others. A compact taxonomy or automated diff/AST/test-delta characterization over RQ3 passes would make “compounding Ghost Errors” and “semantic collapse” operational rather than largely interpretive, and would strengthen the link from obedience to structural corruption.
  4. [Abstract, §1, §4.1, §8] §4.1 / §8 and production claims in Abstract/§1/§7: Incorrect instructions are single-paragraph, human- or proxy-generated location misdirections on single-function algorithmic problems; RQ3’s proxy (GPT-5.1 Codex) sees code only and never tests. The Limitations section correctly notes this is a hard setting for obedience to persist, but the manuscript repeatedly draws direct consequences for multi-file agents (Copilot, Cursor, Devin). Either temper production claims to “lower-bound behavioral risk under clean oracles,” or add at least one multi-file / noisier-feedback stress check (even small-n) so transfer is not solely an untested axiom. As written, the external-validity gap is larger than the abstract’s deployment language suggests.
minor comments (6)
  1. [Figure 1] Figure 1’s progressive narrowing is useful but hard to read without a table of exact eligibility counts and filters per model/RQ; add a small table mirroring the figure numbers.
  2. [§5.1, Appendix McNemar] §5.1 cites “Table 9” for McNemar results; in the manuscript these appear in the Appendix as a McNemar table—align numbering and cross-references.
  3. [Abstract, §1, §3.3] Terminology “Ghost (Unknown) Errors” is used interchangeably with “Ghost Errors”; pick one term and define it once early (preferably with the operational filter used in RQ3).
  4. [Figs. 6–10] Several figure captions assert conclusions (“never escape,” “cannot reverse”) that belong in Results text; keep captions descriptive and move interpretive claims to §5–§6.
  5. [§2] Related work is instruction-following heavy; a short comparison to prior adversarial/incorrect-prompt robustness and multi-turn repair (beyond the cited works) would help position novelty more sharply.
  6. [§1, Fig. 1, References] Minor prose issues: duplicated citation clusters in §1; “self repairment” in Fig. 1 caption; arXiv-style future dates on some model cards—normalize bibliography formatting.

Circularity Check

0 steps flagged

Empirical behavioral measurement with independent test-oracle outcomes; no derivation reduces to its inputs by construction.

full rationale

This paper is not a first-principles derivation that claims to predict a quantity from a fitted or self-defined premise. Its load-bearing results are pass/fail counts obtained by executing model-generated Python against held-out deterministic RunBugRun tests under controlled instruction conditions (T1/T2/T3; RQ1–RQ4). Blind Obedience is operationalized as the measured asymmetry between (i) independent CORRECT/INCORRECT classification of T2 instructions and (ii) generation under those same instructions, confirmed by McNemar on paired T1 vs T2 outcomes—not by defining the target as the fit. Ghost Errors and Irrecoverable Damage Rate are likewise post-hoc labels for observed test degradation and failure to re-cross the original buggy-patch baseline after five self-guided passes; they do not force those outcomes by construction. Progressive filtering (T3 failures → T2-with-damage → RQ3-obedient) is experimental subsetting, not circular prediction. There is no fitted parameter renamed as a forecast, no uniqueness theorem imported from overlapping authors, no ansatz smuggled via self-citation, and no renaming of a known closed-form result. Causal confounds (e.g., RQ4 recovery partly reflecting RQ2’s self-repair ceiling) are validity concerns, not circularity. The measurement chain is self-contained against external executable oracles.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

The central claims rest on experimental design choices and domain assumptions, not free physical constants. Load-bearing premises are: executable tests as unambiguous correctness oracles; synthetic assertive wrong instructions as stand-ins for human misdiagnosis; five-pass caps and damage filters defining who enters RQ3/RQ4; and API default low-reasoning configs as representative of production. Named constructs (Blind Obedience, Ghost Errors, Irrecoverable Damage Rate) are operational labels for measured behaviors, not new physical entities.

free parameters (4)
  • MAX_PASSES (iterative repair horizon)
    Fixed at 5 for RQ2–RQ4; early-stop on full pass. Irrecoverable Damage Rate and recovery ceilings are defined relative to this horizon, not derived from first principles.
  • RQ3 damage eligibility filter
    Only problems where T2 failed-test count exceeds the original buggy baseline enter RQ3; this hand-chosen filter defines the population on which Ghost Error compounding is measured.
  • sampling / decoding settings (temperature=0.2, top_p=0.95, max_tokens=5000, reasoning_effort low/none)
    Default production-like configs chosen by authors; ablation on reasoning levels uses a stratified 100-sample subset. Outcomes can depend on these knobs.
  • proxy incorrect-instruction generator choice (GPT-5.1 Codex, code-only, no test access)
    RQ3 wrong instructions are dynamically generated by a chosen proxy model and prompt style; different generators could change obedience rates.
axioms (5)
  • domain assumption Deterministic executable test suites are a sufficient and unambiguous oracle for semantic correctness of patches.
    Stated throughout §1 and §4.1; underpins all pass/fail and damage measurements.
  • domain assumption A confident, location-misdirected natural-language instruction that references real code elements is a valid model of human (or agent) misdiagnosis in repair workflows.
    T2 generation prompt and RQ3 proxy design (§3.1, §3.3); ecological validity of Blind Obedience depends on this.
  • domain assumption Self-thinking repair with failing-test feedback and no external instruction is the appropriate upper-bound baseline for recovery capacity (RQ2 vs RQ4).
    §3.2 and §3.4 equate protocols differing only in start state; irrecoverability is defined against the original buggy-patch baseline.
  • standard math McNemar’s test on paired T1/T2 binary outcomes is an appropriate test of systematic Blind Obedience asymmetry.
    Appendix McNemar table; standard paired binary test applied to 538 problems per model.
  • ad hoc to paper Findings on single-function algorithmic Python problems remain informative about production multi-file agent behavior (at least as a lower bound on how hard obedience is to sustain).
    Limitations §8 argues the simple setting should make resistance easiest; transfer remains an unproven bridge to Copilot/Cursor-scale codebases.
invented entities (3)
  • Blind Obedience independent evidence
    purpose: Name the systematic tendency to follow incorrect instructions despite independent recognition that they are wrong.
    Defined from RQ1 generation vs classification asymmetry; operational, not a new physical mechanism.
  • Ghost (Unknown) Errors independent evidence
    purpose: Label structural faults introduced by wrong-location patches beyond the original bug, compounding across iterative wrong instructions.
    Measured via worsening test outcomes and qualitative code drift in RQ3/RQ4 traces.
  • Irrecoverable Damage Rate independent evidence
    purpose: Quantify the share of RQ3-corrupted problems that five-pass self-guided repair cannot restore past the original buggy baseline.
    Defined by RQ4 success semantics; depends on pass horizon and baseline choice.

pith-pipeline@v1.1.0-grok45 · 34269 in / 3986 out tokens · 55493 ms · 2026-07-11T17:44:34.551668+00:00 · methodology

0 comments
read the original abstract

Code language models are now trusted collaborators in production workflows for debugging, refactoring, and iterative repair, and every benchmark that evaluates them assumes the instructions they act on are correct. We study what happens when that assumption breaks. We evaluate code language models across four experiments designed to assess whether models resist or obey incorrect instructions in single-pass and iterative repair settings, using the RunBugRun dataset of algorithmic Python problems with deterministic test cases. Our findings reveal a striking behavioral pattern: models correctly identify an incorrect instruction as wrong, then follow it anyway. This compliance unknowingly introduces errors beyond the original bug, and the corrupted code state cannot be recovered through subsequent self-guided iterative repair, which fails to converge across passes. We term this Blind Obedience, characterize the Ghost (Unknown) Errors it introduces, quantify the proportion of cases where semantic corruption proves irrecoverable, and show that extended reasoning cannot reverse it. These findings surface behavioral properties invisible to pass-rate evaluation, with direct consequences for code language models deployed in production settings.

Figures

Figures reproduced from arXiv: 2607.04537 by Adi Bajpai, Anany Singh Divy, Rajiv Ratn Shah, Raj Jaiswal, Savar Bhasin, Tanuja Ganu.

Figure 1
Figure 1. Figure 1: RQ1 is the full 538-problem baseline. Prob [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The four experimental settings form a progressive chain from blind obedience to irrecoverable semantic [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: T2 (Incorrect Instruction) produces the steepest pass rate degradation across all models, falling below T3 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Models overwhelmingly classify T2 incorrect instructions as wrong in the evaluator role (left panel) yet [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Elevated reasoning does not improve patch correctness — as reasoning level increases, models shift from [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Problems that entered RQ3 damaged never escape across all five passes — obedient problem counts confirm that blind obedience under iterative incorrect guidance is sustained, not momentary. See §5.3. 5.2 RQ2: Self-Guided Code Repair Stalls Beyond the Initial Passes Self-guided iterative repair resolves the majority of recoverable problems within the first two passes ( [PITH_FULL_IMAGE:figures/full_fig_p006… view at source ↗
Figure 7
Figure 7. Figure 7: Recovery is front-loaded — the majority of problems resolve within the first two passes and the resolution [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The majority of problems carrying Ghost Errors from RQ3 never escape across five self-guided recovery [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: All reasoning configurations plateau within [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: The escape rate across RQ3 passes remains [PITH_FULL_IMAGE:figures/full_fig_p008_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: B vs. C problem counts per model under McNemar’s test. B (blind obedience: T1 pass, T2 fail) dominates [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Repository structure of the experimental codebase, organized into four modules: [PITH_FULL_IMAGE:figures/full_fig_p043_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

76 extracted references · 1 canonical work pages

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  2. [2]

    Publications Manual , year = "1983", publisher =

    1983

  3. [3]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    doi:10.1145/322234.322243 1981

  4. [4]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  5. [5]

    Dan Gusfield , title =. 1997

    1997

  6. [6]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  7. [7]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  8. [8]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    TruthfulQA: Measuring How Models Mimic Human Falsehoods , author =. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2022 , publisher =

    2022

  9. [9]

    2021 , eprint=

    Measuring Massive Multitask Language Understanding , author=. 2021 , eprint=

    2021

  10. [10]

    2017 , eprint=

    TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension , author=. 2017 , eprint=

    2017

  11. [11]

    and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle =

    Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle =. 2024 , url =

    2024

  12. [12]

    and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , booktitle =

    Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , booktitle =. 2024 , url =

    2024

  13. [13]

    Agentless: Demystifying

    Xia, Chunqiu Steven and Deng, Yinlin and Dunn, Soren and Zhang, Lingming , year =. Agentless: Demystifying. 2407.01489 , archivePrefix =

    Pith/arXiv arXiv

  14. [14]

    Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for

    Xia, Chunqiu Steven and Zhang, Lingming , year=. Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for. doi:10.1145/3650212.3680323 , booktitle=

    doi:10.1145/3650212.3680323

  15. [15]

    Wang, Zihan and others , year =. Where. 2509.25370 , archivePrefix=

    arXiv

  16. [16]

    2025 , eprint=

    Where LLM Agents Fail and How They can Learn From Failures , author=. 2025 , eprint=

    2025

  17. [17]

    Business Horizons , volume =

    Navigating software development in the ChatGPT and GitHub Copilot era , author =. Business Horizons , volume =. 2024 , publisher =

    2024

  18. [18]

    arXiv preprint arXiv:2509.20353 , year =

    Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study , author =. arXiv preprint arXiv:2509.20353 , year =

    Pith/arXiv arXiv

  19. [19]

    2025 , eprint=

    A Survey on Code Generation with LLM-based Agents , author=. 2025 , eprint=

    2025

  20. [20]

    Beyond Code Generation: LLM-supported Exploration of the Program Design Space , booktitle =

    Andrei Zamfirescu-Pereira and Michael Muller and Steven Tanimoto and Bj. Beyond Code Generation: LLM-supported Exploration of the Program Design Space , booktitle =. 2025 , publisher =. doi:10.1145/3706598.3714154 , url =

    doi:10.1145/3706598.3714154 2025

  21. [21]

    arXiv preprint arXiv:2510.19692 , year =

    Rashina Hoda , title =. arXiv preprint arXiv:2510.19692 , year =

    arXiv

  22. [22]

    arXiv preprint arXiv:2511.04427 , year =

    Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects , author =. arXiv preprint arXiv:2511.04427 , year =

    arXiv

  23. [23]

    2025 , eprint =

    The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering , author =. 2025 , eprint =. doi:10.48550/arXiv.2507.15003 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.15003 2025

  24. [24]

    arXiv preprint arXiv:2507.20439 , year=

    When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptions , author=. arXiv preprint arXiv:2507.20439 , year=

    Pith/arXiv arXiv

  25. [25]

    arXiv preprint arXiv:2504.02733 , year =

    Enhancing LLM Robustness to Perturbed Instructions: An Empirical Study , author =. arXiv preprint arXiv:2504.02733 , year =. doi:10.48550/arXiv.2504.02733 , url =. 2504.02733 , archivePrefix =

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.02733

  26. [26]

    2025 , eprint=

    ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions , author=. 2025 , eprint=

    2025

  27. [27]

    2026 , eprint=

    Kimi K2.5: Visual Agentic Intelligence , author=. 2026 , eprint=

    2026

  28. [28]

    Advances in Neural Information Processing Systems (NeurIPS 2024) , year =

    Code Repair with LLMs gives an Exploration-Exploitation Tradeoff , author =. Advances in Neural Information Processing Systems (NeurIPS 2024) , year =

    2024

  29. [29]

    Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE) , year =

    Islem Bouzenia and Premkumar Devanbu and Michael Pradel , title =. Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE) , year =. doi:10.1109/ICSE55347.2025.00157 , url =

    doi:10.1109/icse55347.2025.00157 2025

  30. [30]

    2021 , eprint=

    Evaluating Large Language Models Trained on Code , author=. 2021 , eprint=

    2021

  31. [31]

    2022 , eprint=

    A Systematic Evaluation of Large Language Models of Code , author=. 2022 , eprint=

    2022

  32. [32]

    arXiv preprint arXiv:2406.00515 , year =

    A Survey on Large Language Models for Code Generation , author =. arXiv preprint arXiv:2406.00515 , year =. doi:10.48550/arXiv.2406.00515 , eprint =

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.00515

  33. [33]

    2025 , eprint=

    CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generation , author=. 2025 , eprint=

    2025

  34. [34]

    2025 , eprint=

    CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generation , author=. 2025 , eprint=

    2025

  35. [35]

    2025 , eprint=

    CIFE: Code Instruction-Following Evaluation , author=. 2025 , eprint=

    2025

  36. [36]

    2025 , eprint=

    Bridging Developer Instructions and Code Completion Through Instruction-Aware Fill-in-the-Middle Paradigm , author=. 2025 , eprint=

    2025

  37. [37]

    2025 , eprint=

    Human-AI collaboration or obedient and often clueless AI in instruct, serve, repeat dynamics? , author=. 2025 , eprint=

    2025

  38. [38]

    arXiv preprint arXiv:2504.16331 , year=

    Can Code Language Models Learn Clarification-Seeking Behaviors? , author=. arXiv preprint arXiv:2504.16331 , year=. 2504.16331 , archivePrefix=

    arXiv

  39. [39]

    arXiv preprint arXiv:2310.10996 , year=

    ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarification , author=. arXiv preprint arXiv:2310.10996 , year=

    Pith/arXiv arXiv

  40. [40]

    arXiv preprint arXiv:2408.13372 , year=

    Understanding Defects in Generated Codes by Language Models , author=. arXiv preprint arXiv:2408.13372 , year=

    Pith/arXiv arXiv

  41. [41]

    2025 , eprint=

    EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits , author=. 2025 , eprint=

    2025

  42. [42]

    2022 , eprint=

    Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor , author=. 2022 , eprint=

    2022

  43. [43]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  44. [44]

    arXiv preprint arXiv:2306.04757 , year=

    INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models , author=. arXiv preprint arXiv:2306.04757 , year=

    Pith/arXiv arXiv

  45. [45]

    Agentic Coding: Fundamentals and Practical Implications of Agentic AI , author=

    Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI , author=. arXiv preprint arXiv:2505.19443 , year=

    Pith/arXiv arXiv

  46. [46]

    arXiv preprint arXiv:2505.16944 , year=

    AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios , author=. arXiv preprint arXiv:2505.16944 , year=

    Pith/arXiv arXiv

  47. [47]

    Le Gall and A

    P. Le Gall and A. Arnould , title =. LNCS 1130: Recent Trends in Data Type Specification (Workshop Proc.) , year =

  48. [48]

    E. T. Barr and M. Harman and P. McMinn and M. Shahbaz and S. Yoo , title =. IEEE Trans. Software Eng. , year =

  49. [49]

    arXiv preprint arXiv:2411.07112 , year =

    ROCODE: Integrating Backtracking Mechanism and Program Analysis in Large Language Models for Code Generation , author=. arXiv preprint arXiv:2411.07112 , year =

    Pith/arXiv arXiv

  50. [50]

    ArXiv preprint , year =

    Debalina Ghosh Paul and Hong Zhu and Ian Bayley , title =. ArXiv preprint , year =

  51. [51]

    Computer Speech & Language , year =

    Chris van der Lee and Albert Gatt and Emiel van Miltenburg and Emiel Krahmer , title =. Computer Speech & Language , year =

  52. [52]

    arXiv preprint arXiv:2312.12450 , year =

    Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions , author=. arXiv preprint arXiv:2312.12450 , year =

    Pith/arXiv arXiv

  53. [53]

    arXiv preprint arXiv:2404.03543 , year =

    CodeEditorBench: Evaluating Code Editing Capability of Large Language Models , author=. arXiv preprint arXiv:2404.03543 , year =

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2403.07974 , year=

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , author=. arXiv preprint arXiv:2403.07974 , year=

    Pith/arXiv arXiv

  55. [55]

    arXiv preprint arXiv:2504.21751 , year =

    CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation , author =. arXiv preprint arXiv:2504.21751 , year =

    Pith/arXiv arXiv

  56. [56]

    2023 , eprint=

    Reflexion: Language Agents with Verbal Reinforcement Learning , author=. 2023 , eprint=

    2023

  57. [57]

    arXiv preprint arXiv:2308.12950 , year =

    Code Llama: Open Foundation Models for Code , author =. arXiv preprint arXiv:2308.12950 , year =

    Pith/arXiv arXiv

  58. [58]

    arXiv preprint arXiv:2406.11931 , year =

    DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence , author =. arXiv preprint arXiv:2406.11931 , year =

    Pith/arXiv arXiv

  59. [59]

    arXiv preprint arXiv:2505.09388 , year =

    Qwen3 Technical Report , author =. arXiv preprint arXiv:2505.09388 , year =

    Pith/arXiv arXiv

  60. [60]

    2026 , eprint=

    GLM-5: from Vibe Coding to Agentic Engineering , author=. 2026 , eprint=

    2026

  61. [61]

    2024 , url =

    Grok Code Fast 1 , author =. 2024 , url =

    2024

  62. [62]

    2026 , url =

    Claude 4.6 Sonnet , author =. 2026 , url =

    2026

  63. [63]

    2026 , url =

    GPT-5.3-Codex , author =. 2026 , url =

    2026

  64. [64]

    2023 , eprint=

    RunBugRun -- An Executable Dataset for Automated Program Repair , author=. 2023 , eprint=

    2023

  65. [65]

    OpenReview , year =

    CoDyn: Dynamic LLM Routing for Coding Tasks , author =. OpenReview , year =

  66. [66]

    Automating the correctness assessment of AI-generated code for security contexts , volume=

    Cotroneo, Domenico and Foggia, Alessio and Improta, Cristina and Liguori, Pietro and Natella, Roberto , year=. Automating the correctness assessment of AI-generated code for security contexts , volume=. doi:10.1016/j.jss.2024.112113 , journal=

    work page doi:10.1016/j.jss.2024.112113 2024

  67. [67]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year =

    CODEJUDGE: Evaluating Code Generation with Large Language Models , author =. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year =

    2024

  68. [68]

    arXiv preprint arXiv:2602.03462 , year =

    Toward Functional and Non-Functional Evaluation of Application-Level Code Generation , author =. arXiv preprint arXiv:2602.03462 , year =

    arXiv

  69. [69]

    arXiv preprint arXiv:2505.13938 , year =

    CLEVER: A Curated Benchmark for Formally Verified Code Generation , author =. arXiv preprint arXiv:2505.13938 , year =

    arXiv

  70. [70]

    2025 , eprint=

    Towards Understanding Sycophancy in Language Models , author=. 2025 , eprint=

    2025

  71. [71]

    Measuring Sycophancy of Language Models in Multi-turn Dialogues , url=

    Hong, Jiseung and Byun, Grace and Kim, Seungone and Shu, Kai , year=. Measuring Sycophancy of Language Models in Multi-turn Dialogues , url=. doi:10.18653/v1/2025.findings-emnlp.121 , booktitle=

    doi:10.18653/v1/2025.findings-emnlp.121 2025

  72. [72]

    2025 , eprint=

    A Hierarchical and Evolvable Benchmark for Fine-Grained Code Instruction Following with Multi-Turn Feedback , author=. 2025 , eprint=

    2025

  73. [73]

    2026 , eprint=

    SciIF: Benchmarking Scientific Instruction Following Towards Rigorous Scientific Intelligence , author=. 2026 , eprint=

    2026

  74. [74]

    2026 , eprint=

    Detect Repair Verify for Securing LLM Generated Code: A Multi-Language Empirical Study , author=. 2026 , eprint=

    2026

  75. [75]

    2025 , eprint=

    When Models Can't Follow: Testing Instruction Adherence Across 256 LLMs , author=. 2025 , eprint=

    2025

  76. [76]

    2024 , eprint=

    CYCLE: Learning to Self-Refine the Code Generation , author=. 2024 , eprint=

    2024