Pith. sign in

REVIEW 3 major objections 6 minor 45 references

When an LLM writes code, it silently commits to assumptions that never appear in the prompt; this paper's AssumptionMiner surfaces those decisions, links each to the code it governs, and regenerates only the affected region when a developer

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 04:10 UTC pith:QMTNBSWR

load-bearing objection A genuinely new framework and benchmark for surfacing implicit assumptions in LLM-generated code, but the headline F1 is carried by an unverified, likely contaminated tier and should be read as upper-bound evidence. the 3 major comments →

arxiv 2607.22898 v1 pith:QMTNBSWR submitted 2026-07-24 cs.SE cs.CL

AssumptionMiner: Extracting, Tracing, and Revising Implicit Assumptions in LLM Code Generation

Jie "JW" Wu This is my paper
classification cs.SE cs.CL
keywords assumption miningLLM code generationimplicit assumptionscode traceabilityAST dependency graphincremental regenerationcode adaptabilitybenchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that LLM-generated code is full of implicit assumptions—design decisions the model makes to fill gaps in underspecified prompts—and that these hidden choices are a major source of 'working-but-wrong' code that passes tests yet violates developer intent. It presents AssumptionMiner, a framework that makes these assumptions an explicit, structured output of code generation, linked to the exact code regions they govern through an AST-based dependency graph. The paper backs this with a benchmark of 180 ambiguous tasks and 676 annotated assumptions, showing that a confidence-weighted ensemble of open-source LLMs can extract assumptions at an F1 of 0.816, far above offline baselines. It also demonstrates that when a developer revises an assumption, targeted regeneration changes far less code than full regeneration, though with a cost in syntactic validity. The sympathetic reader sees a practical path toward more inspectable and controllable AI-generated code: rather than hoping the model guesses right, treat its guesses as first-class artifacts that can be reviewed and corrected.

Core claim

The central discovery is that implicit assumptions in LLM-generated code are not incidental noise but a structural consequence of collapsing requirements-to-design-to-implementation into a single opaque step, and that they can be systematically surfaced as structured AssumptionRecords. Each record captures the assumption's category (from a six-part taxonomy covering input validation, data format, error policy, persistence, performance, and security), a natural-language description, a rationale, at least one realistic alternative, and a confidence score. The records are linked to the code regions they govern via a two-pass dependency mapper (keyword-overlap candidate selection then minimal-su

What carries the argument

The load-bearing machinery is the AssumptionRecord schema combined with the two-phase extraction prompt and the AST-based dependency graph. The extraction prompt first asks the LLM to reason about which choices in the code go beyond what the prompt requires, then to format its discoveries into JSON records with category, description, alternatives, and confidence—the two-phase structure is what outperforms direct generation, clarifying questions, and plain chain-of-thought in preliminary experiments. The dependency mapper then uses a category-to-node-type map to select AST nodes whose identifiers overlap with keywords from the assumption's description, narrows to the minimal enclosing subtree

Load-bearing premise

The benchmark's reference annotations are produced by the same kind of model that is being evaluated (GPT-4o) using the same extraction prompt, and only 30 of the 180 tasks have human-verified references—so the reported F1 scores may measure how well the extractor agrees with an LLM's framing of 'implicit assumption' rather than how well it surfaces decisions that human developers would actually consider important.

What would settle it

Ask a group of professional developers to independently list what they consider the most important hidden decisions in a set of LLM-generated code samples, then run AssumptionMiner on the same samples and compare. If the majority of developer-flagged decisions are absent from the extracted assumption layer, or if the surfaced assumptions are mostly judged trivial or irrelevant, the central claim—that making assumptions explicit improves transparency and controllability—would fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Making assumptions explicit gives developers a way to audit and correct LLM-generated code without re-prompting from scratch, which should reduce the rate of working-but-wrong code shipped to production.
  • The dependency graph turns a revised assumption into a surgical edit: regenerating only the governed region changes 1.3–2.4x less code than full regeneration or manual reprompting, so reviewers can reason about the blast radius of a decision change.
  • The benchmark's strict decision-level metric (F1 = 0.66 for the best cross-model configuration) shows that extracting the exact decision, not just its category, remains an open problem—future systems must improve description fidelity, not just category coverage.
  • Because every revision is traceable to a specific AssumptionRecord, the framework naturally produces a live decision log for generated code, which could serve as automated design documentation and support parallel, independent revisions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If assumption extraction generalized beyond Python and short single-file tasks, it could turn IDE code assistants from black-box generators into tools that surface 'why this code looks this way'—a form of automated design rationale that may reduce debugging time for developers unfamiliar with the generated artifact.
  • The conflict between targeted regeneration and cascading edits suggests a natural next mechanism: a post-revision pass that re-validates all dependency links and flags regions invalidated by a change, rather than the current single-pass splice; such a pass would likely close most of the 17-point syntactic-validity gap.
  • The extraction F1 of 0.816 is measured against an LLM-produced reference; a user study where practicing developers judge the usefulness of the surfaced assumptions would test whether the benchmark's 'implicit assumption' construct maps to what developers actually care about in real code review.
  • Because the confidence-weighted ensemble outperforms either backbone alone by combining a recall-biased and a precision-biased model, the same ensembling trick may transfer to other LLM-augmented engineering tasks where a trade-off between coverage and exactness is the bottleneck.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes AssumptionMiner, an LLM-based framework that takes a natural-language prompt and the code an LLM generated for it, and produces an explicit 'assumption layer' of implicit design decisions (categorized into a six-part taxonomy), links each assumption to the code regions that implement it via a keyword-overlap plus AST-narrowing dependency mapper, and supports targeted regeneration when a developer revises an assumption. It also introduces a benchmark of 180 ambiguous programming tasks with 676 GPT-4o-generated reference assumptions, of which 109 assumptions across 30 tasks are human-verified. The evaluation reports three results: (i) assumption extraction, where a confidence-weighted ensemble of Qwen2.5-Coder-32B and DeepSeek-Coder-V2-Lite reaches F1 = 0.816 under category-level matching (F1 = 0.662 under a stricter decision-level metric); (ii) dependency localization, where AST-guided mapping yields tighter regions than keyword matching, though threshold accuracy is similar; and (iii) assumption-guided regeneration, which produces smaller edits than full regeneration or in-place editing but with lower syntactic validity (83.3% vs 100%).

Significance. If the extraction results are robust, AssumptionMiner is a useful contribution: it turns an invisible side effect of LLM code generation into a structured, inspectable, and revisable artifact, and it provides a public benchmark, source code, and reproducibility package. The paper is unusually careful in several respects: it reports bootstrap confidence intervals, distinguishes lenient category-level from strict decision-level matching, acknowledges the self-consistency risk of using GPT-4o as annotator and evaluator, and performs human verification on a curated subset. The honest disclosure of the strict F1 drop to 0.66 and the RQ3 syntactic-validity gap is a strength. However, because the reference labels for 150 of the 180 tasks are generated by GPT-4o with the same extraction prompt used by AssumptionMiner, and because no per-tier breakdown is reported, the magnitude of the headline 0.816 F1 and the 3.6x improvement over baselines is not yet established. The paper is best read as a framework-plus-benchmark contribution whose central quantitative claim requires additional validation.

major comments (3)
  1. [Table III, §V, §VIII] The headline RQ1 result is reported on the full 180-task benchmark, but 150 of those tasks (567 of 676 reference assumptions) have reference annotations produced by GPT-4o using the same structured extraction prompt as the system under evaluation (Appendix A), and these annotations are not human-verified. The paper acknowledges this in §VIII, but it does not report AR/AP/F1 separately for the 30-task human-verified curated subset and the 150-task extended tier. This is load-bearing because the aggregate F1 = 0.816 is dominated by the unverified, potentially easier and contaminated HumanEval/MBPP tier, and Table IV suggests the curated-only categories T4/T6 have F1 values of only 0.50–0.67 for the open-source backbones. The authors should report per-tier extraction results for every backbone and the ensemble, including confidence intervals, and state explicitly whether the 3.6x improvemen
  2. [§III-B, §V, Definition 1] The reference assumption set A* is generated with the same two-phase extraction prompt (Appendix A) that AssumptionMiner itself uses. The human-verified portion covers only 109 assumptions on 30 tasks, and one of the two annotators is the author. Consequently, the reported F1 scores measure agreement between an extractor and the annotating model's own framing of 'implicit assumptions'; they do not independently establish that the extracted items correspond to what developers would regard as important hidden decisions. The strict decision-level metric mitigates this in part, but the primary abstract/conclusion claims are based on the lenient category-level metric. The authors should either (a) human-verify a representative sample of the extended tier, or (b) explicitly reframe the headline claim as 'agreement with a GPT-4o-derived reference' rather than as fidelity to developer intent, an
  3. [§VI-A, category-level matching] The 'category-level (lenient)' matching rule counts a predicted assumption as correct whenever its taxonomy category matches an unused reference category for the same task. This is essentially a category-coverage metric, not a measure of whether the same design decision was identified. The paper acknowledges this in §VIII and reports the stricter decision-level metric, but the abstract's headline 'F1 = 0.816' is not qualified as category-level at the point where it is introduced. Since the strict decision-level F1 is 0.662, the gap between the two metrics is large and should be foregrounded whenever the 0.816 number is cited. Please add the word 'category-level' to the abstract's numeric claim, or move the strict metric into the headline statement.
minor comments (6)
  1. [References [16] and [27]] Reference [16] contains the literal placeholder 'tODO: complete author list from arXiv:2504.16331' and must be completed. Reference [27] (AVATAR) is cited for 'automated design-rationale extraction' in §VII-D, but AVATAR is a program-repair paper; the citation does not match the claim.
  2. [§I and §III-A] The contribution statement C2 says AssumptionMiner produces 'code and a formalized assumption layer in a single augmented generation step,' but the architecture in §III-A separates code generation (C2) from assumption extraction (C3), which is a second LLM call. Please rephrase to avoid implying joint decoding.
  3. [§VI-A, §VIII] The semantic-similarity threshold τ = 0.56 is calibrated on only 30 author-labeled pairs with 80% agreement. This small, single-annotator calibration is not listed in the threats-to-validity section; please add it, or provide a second annotator for the calibration pairs.
  4. [Table V caption] The 'Calls' column lists '1' for both AM-LLM rows in the full-set columns, which is ambiguous: it presumably means one additional LLM call per assumption, not one call total. Please clarify in the caption or table notes.
  5. [Throughout] The notation alternates between AF1 and F1 (e.g., Tables III–IV vs. the text). Please use a single notation or define both at first use.
  6. [Abstract] The abstract says 'improving on the strongest offline baseline by 3.6x.' This is a ratio of F1 scores (0.816/0.225), not a relative improvement in error rate; since the baseline is near zero and the matching is category-level, the ratio can be misleading. Consider reporting the absolute gain and the strict F1 as well.

Circularity Check

2 steps flagged

The benchmark reference is generated by GPT-4o using the same structured extraction prompt as AssumptionMiner, so the headline F1 partially reduces to agreement with GPT-4o's own labels rather than an independent ground truth.

specific steps
  1. self definitional [§V (Annotation), §VI-B (RQ1), Table III]
    "All 180 tasks use GPT-4o [24] to extract the reference assumption list with the same structured prompt as the extractor (§III-B), followed by programmatic validation (schema conformance, category-vocabulary checks, and deduplication). ... Because GPT-4o produced the reference annotations, results for the GPT-4o backbone carry a self-consistency risk, marked † in Table III."

    For the GPT-4o backbone, the reference A* and the evaluated extractor output are both GPT-4o responses to the identical extraction prompt at temperature 0. The reported F1 = 0.884 therefore measures GPT-4o agreeing with itself, not agreement with an external gold standard. The paper explicitly acknowledges this self-consistency risk by marking the GPT-4o row with †.

  2. fitted input called prediction [§V (Table II, Annotation), §VI-B (Table III, RQ1 answer)]
    "All headline results (Tables III and IV, RQ3) use all 180 tasks. ... All 180 tasks use GPT-4o [24] to extract the reference assumption list with the same structured prompt as the extractor (§III-B). ... The extended tier is not human-verified. This is noted as a residual threat in §VIII."

    The headline open-source F1 = 0.816 is computed over all 180 tasks, but the reference for 567 of 676 assumptions is GPT-4o's own output using the same extraction prompt that AssumptionMiner uses. Human verification covers only the 30-task curated subset (109 assumptions). Thus the central 3.6x improvement claim measures how well open-source models reproduce GPT-4o's rendering of the extraction prompt, not how well they match an independently established definition of implicit assumptions. The paper reports no per-tier breakdown, so the validated portion cannot be separated from the self-referential extended tier.

full rationale

The paper's central evaluation has a partial circularity: the reference annotations for the benchmark are produced by GPT-4o using the same structured extraction prompt that AssumptionMiner applies (Listing 3), and the extended tier contributing 567 of 676 assumptions is not human-verified. For the GPT-4o backbone this is self-agreement by construction, which the paper acknowledges with the † marker. For the open-source backbones that carry the headline result, the F1 still measures agreement with GPT-4o's labels rather than with an independent ground truth; the human-verified subset is too small (109/676) and no per-tier scores are reported. RQ2 and RQ3 are less affected: RQ2 uses human-verified code-region references on the curated subset, and RQ3 measures objective syntactic validity and edit distance. The paper also makes a good-faith effort to mitigate the reference problem through human verification of the curated subset and by reporting a strict decision-level metric with deterministic embedding similarity. However, because the headline aggregate F1 and the 3.6x improvement are anchored to the GPT-4o-generated, mostly unhuman-verified reference, the evaluation is only partially independent of the system's own prompt. I therefore assign a score of 6: partial circularity, concentrated in the benchmark construction, while the localization and regeneration claims retain independent content.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 1 invented entities

The central claim rests on a specific operationalization of 'implicit assumption' (Definition 1) and on reference labels produced by GPT-4o using the same extraction prompt as the proposed system. The dependency mapper and several thresholds are tuned on the evaluation set. These choices are honestly documented but mean the reported numbers should not be read as an unbiased estimate of real-world assumption-mining performance.

free parameters (5)
  • category_to_node_type_map = T1-T6 to AST node types, e.g., T3->[except_clause, with_statement, call]
    Used in the dependency mapper (Algorithm 1); the paper's Table VI shows it was expanded during development on the 30 curated tasks, which are also the RQ2 evaluation set. This is tuning on the test set.
  • IoU threshold for RQ2 = 0.5
    Chosen based on reference span statistics (mean span 2.0 lines) to declare a localization correct. This threshold calibrates the metric and is not independently motivated.
  • semantic similarity threshold tau = 0.56
    Used for strict decision-level matching. Calibrated against 30 human same-decision labels (§VI-B); any tau between 0.55 and 0.58 performs identically, so it is a tuned parameter.
  • ensemble confidence threshold = 0.5
    The CW ensemble includes a category if the mean confidence across both backbones is >= 0.5 (Appendix D). Threshold affects precision/recall trade-off.
  • context width k = 5
    Number of context lines above/below each region in incremental regeneration (§III-E). A small design choice that affects regeneration behavior.
axioms (4)
  • domain assumption Definition 1: an implicit assumption is a proposition not stated in the prompt, instantiated in the code, with an alternative that would produce meaningfully different behavior.
    This operationalization defines the ground truth for the benchmark. If developers' notion of 'assumption' differs, all measured F1 scores change as the concept shifts.
  • domain assumption The six-category taxonomy (T1-T6) is adequate and complete for the studied tasks.
    The taxonomy was developed on 30 curated tasks with LLM assistance and then applied to all 180 tasks. The paper notes it 'does not establish that the taxonomy is complete for other languages, repositories, or development settings.'
  • domain assumption GPT-4o's structured prompt output is a valid proxy for developer-perceived assumptions.
    All reference annotations are generated by GPT-4o using the same extraction prompt as the evaluated system. Human verification covers only 30 tasks, so the extended tier relies entirely on this proxy.
  • domain assumption HumanEval/MBPP tasks, being part of LLM training data, still produce representative ambiguous prompts for evaluating assumption mining.
    The extended 150-task tier is drawn from HumanEval and MBPP. The paper acknowledges contamination risk: the model may generate stereotyped solutions whose assumptions are easier to extract, inflating results.
invented entities (1)
  • AssumptionRecord (assumption layer) no independent evidence
    purpose: Structured representation of an implicit assumption with category, description, alternatives, and code references.
    This is a data structure introduced by the paper, not a physical entity. It has no falsifiable handle outside the paper beyond the software artifact.

pith-pipeline@v1.3.0-alltime-deepseek · 25843 in / 8448 out tokens · 90620 ms · 2026-08-01T04:10:47.052534+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) generate code from natural-language prompts, yet real-world prompts rarely provide complete specifications. When prompts leave input formats, error handling, or design decisions unspecified, LLMs fill these gaps with implicit assumptions that shape the generated code's behavior and correctness. Because these assumptions remain hidden, generated code may satisfy tests while violating developer intent. We present AssumptionMiner, a framework that makes implicit assumptions a first-class artifact of LLM-based code generation. In addition to code, AssumptionMiner produces an explicit assumption layer, a structured representation of inferred constraints and design decisions that developers can inspect, confirm, or revise. An AST-based dependency graph enables targeted regeneration of only the code affected by a revised assumption. We also introduce a benchmark of 180 ambiguous programming tasks with 676 annotated assumptions, including a human-verified subset for evaluating code localization. We evaluate assumption extraction, code localization, and assumption-guided regeneration. Across open-source LLMs, a confidence-weighted ensemble achieves an F1 score of 0.816 for assumption extraction, improving on the strongest offline baseline by 3.6x. On the human-verified localization benchmark, AST-guided localization identifies more precise code regions than keyword-based and whole-file baselines. During assumption revision, targeted regeneration modifies less code than non-targeted alternatives while exposing challenges in handling cascading edits. These results demonstrate that making assumptions explicit improves the transparency and controllability of LLM-based code generation.

Figures

Figures reproduced from arXiv: 2607.22898 by Jie "JW" Wu.

Figure 1
Figure 1. Figure 1: Implicit assumptions in LLM code generation, before and after [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: AssumptionMiner system architecture. Boxes denote com￾ponents. Arrows denote data flow. The dashed region marks the developer interaction loop. C2: Code Generator. Issues p to the backbone LLM and produces the initial code artifact c. Any instruction￾tuned LLM (GPT-4o, Qwen2.5-Coder-7B, or similar) may serve as the backbone. C3: Assumption Extractor. Given (p, c), applies a structured extraction prompt (§I… view at source ↗
Figure 3
Figure 3. Figure 3: Assumption-code dependency graph for the running example. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: AssumptionMiner interactive interface. (1) Highlighted code regions correspond to extracted assumptions. (2) The assumption panel lets developers inspect and edit each assumption record. (3) The revision panel applies changes through incremental regeneration. In the curated subset, 67% of code-region references are single-line (mean span: 2.0 lines), suggesting assumptions typically manifest at specific st… view at source ↗
Figure 5
Figure 5. Figure 5: Representative error cases across six failure modes. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The AssumptionMiner web interface on the running authentication example (screenshots of the released interface/ application). Top: after Analyze, the backbone-generated code (left) is shown beside its five extracted assumption records (right), each labeled with its taxonomy category (T2–T6) and an LLM-assigned severity. Hovering a card highlights the code lines it governs. Bottom: mid-review, the developer… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 10 linked inside Pith

  1. [1]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwardset al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  2. [2]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohanet al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  3. [3]

    Competition-level code generation with AlphaCode,

    Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblondet al., “Competition-level code generation with AlphaCode,”Science, vol. 378, pp. 1092–1097, 2022

    2022

  4. [4]

    SWE-bench: Can language models resolve real-world GitHub issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real-world GitHub issues?” inProc. ICLR, 2024

    2024

  5. [5]

    Productivity assessment of neural code completion,

    A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian, “Productivity assessment of neural code completion,” inProc. MAPS, 2022

    2022

  6. [6]

    Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,

    P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” inProc. CHI EA, 2022

    2022

  7. [7]

    Wiegers and J

    K. Wiegers and J. Beatty,Software Requirements, 3rd ed. Microsoft Press, 2013

    2013

  8. [8]

    An NLP approach for cross-domain ambiguity detection in requirements engineering,

    A. Ferrari and A. Esuli, “An NLP approach for cross-domain ambiguity detection in requirements engineering,”Automated Software Engineering, vol. 26, pp. 559–598, 2019

    2019

  9. [9]

    Correctness, confidence, and context: Framing software assurance in the AI age,

    M. Shaw, “Correctness, confidence, and context: Framing software assurance in the AI age,” 2026. [Online]. Available: https://arxiv.org/abs/2607.04667

    Pith/arXiv arXiv 2026

  10. [10]

    A prompt pattern catalog to enhance prompt engineering with ChatGPT,

    J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert et al., “A prompt pattern catalog to enhance prompt engineering with ChatGPT,”arXiv preprint arXiv:2302.11382, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Specifications: The missing link to making the development of LLM systems an engineering discipline,

    I. Stoica, M. Zaharia, J. Gonzalez, K. Goldberg, K. Sen, H. Zhang, A. Angelopoulos, S. G. Patil, L. Chen, W.-L. Chiang, and J. Q. Davis, “Specifications: The missing link to making the development of LLM systems an engineering discipline,” 2024. [Online]. Available: https://arxiv.org/abs/2412.05299

    Pith/arXiv arXiv 2024

  12. [12]

    ClarifyGPT: A framework for enhancing LLM-based code generation via requirements clarification,

    F. Mu, L. Shi, S. Wang, Z. Yu, B. Zhang, C. Wang, S. Liu, and Q. Wang, “ClarifyGPT: A framework for enhancing LLM-based code generation via requirements clarification,”Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, 2024

    2024

  13. [13]

    Defective task descriptions in LLM-based code generation: Detection and analysis,

    A. Akli, M. Papadakis, M. Cordy, and Y. Le Traon, “Defective task descriptions in LLM-based code generation: Detection and analysis,” arXiv:2604.24703, 2026

    Pith/arXiv arXiv 2026

  14. [14]

    HumanEvalComm: Benchmarking the communication competence of code generation for LLMs and LLM agent,

    J. J. W. Wu and F. H. Fard, “HumanEvalComm: Benchmarking the communication competence of code generation for LLMs and LLM agent,”ACM Transactions on Software Engineering and Methodology, 2025

    2025

  15. [15]

    When prompts go wrong: Eval- uating code model robustness to ambiguous, contradictory, and incomplete task descriptions,

    M. Larbi, A. Akli, M. Papadakis, R. Bouyousfi, M. Cordy, F. Sarro, and Y. Le Traon, “When prompts go wrong: Eval- uating code model robustness to ambiguous, contradictory, and incomplete task descriptions,” arXiv:2507.20439, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    ClarifyCoder: Clarification-aware fine- tuning for programmatic problem solving,

    J. J. W. Wuet al., “ClarifyCoder: Clarification-aware fine- tuning for programmatic problem solving,” 2025, tODO: complete author list from arXiv:2504.16331. [Online]. Available: https://arxiv.org/abs/2504.16331

    arXiv 2025

  17. [17]

    Interac- tive code generation via test-driven user-intent formalization,

    S. K. Lahiri, A. Naik, G. Sakkas, P. Choudhury, C. von Veh, M. Musuvathi, J. P. Inala, C. Wang, and J. Gao, “Interac- tive code generation via test-driven user-intent formalization,” arXiv:2208.05950, 2022

    Pith/arXiv arXiv 2022

  18. [18]

    Self-planning code generation with large language models,

    X. Jiang, Y. Dong, L. Wang, Q. Shang, and G. Li, “Self-planning code generation with large language models,”arXiv preprint arXiv:2303.06689, 2023

    arXiv 2023

  19. [19]

    Is self-repair a silver bullet for code generation?

    T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Solar- Lezama, “Is self-repair a silver bullet for code generation?” in Proc. ICLR, 2024

    2024

  20. [20]

    Self-refine: Iterative refinement with self-feedback,

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegr- effeet al., “Self-refine: Iterative refinement with self-feedback,” inProc. NeurIPS, 2023

    2023

  21. [21]

    Strauss and J

    A. Strauss and J. Corbin,Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory, 2nd ed. Sage Publications, 1998

    1998

  22. [22]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” inProc. NeurIPS, 2022

    2022

  23. [23]

    Conversational automated program repair,

    C. S. Xia and L. Zhang, “Conversational automated program repair,”arXiv preprint arXiv:2301.13246, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    GPT-4o system card,

    OpenAI, “GPT-4o system card,” 2024. [Online]. Available: https://arxiv.org/abs/2410.21276

    Pith/arXiv arXiv 2024

  25. [25]

    Efficient memory manage- ment for large language model serving with PagedAttention,

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory manage- ment for large language model serving with PagedAttention,” in Proc. ACM SOSP, 2023

    2023

  26. [26]

    Self-collaboration code generation via ChatGPT,

    Y. Dong, X. Jiang, Z. Jin, and G. Li, “Self-collaboration code generation via ChatGPT,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 7, pp. 1–38, 2024

    2024

  27. [27]

    AVATAR: Fixing semantic bugs with fix patterns of static analysis viola- tions,

    K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé, “AVATAR: Fixing semantic bugs with fix patterns of static analysis viola- tions,” inProc. SANER, 2019

    2019

  28. [28]

    SequenceR: Sequence-to-sequence learning for end-to-end program repair,

    Z. Chen, S. Kommrusch, M. Tufano, L.-N. Pouchet, D. Poshy- vanyk, and M. Monperrus, “SequenceR: Sequence-to-sequence learning for end-to-end program repair,”IEEE Transactions on Software Engineering, vol. 47, no. 9, pp. 1943–1959, 2021

    1943

  29. [29]

    Neural program repair with execution-based backpropagation,

    H. Ye, M. Martinez, and M. Monperrus, “Neural program repair with execution-based backpropagation,” inProc. ICSE, 2022

    2022

  30. [30]

    Require- ments for tools for ambiguity identification and measurement in natural language requirements specifications,

    N. Kiyavitskaya, N. Zeni, L. Mich, and D. M. Berry, “Require- ments for tools for ambiguity identification and measurement in natural language requirements specifications,” inRequirements Engineering, 2008

    2008

  31. [31]

    Using domain-specific corpora for improved handling of ambiguity in requirements,

    S. Ezzini, S. Abualhaija, C. Arora, M. Sabetzadeh, and L. C. Briand, “Using domain-specific corpora for improved handling of ambiguity in requirements,” inProc. ICSE, 2021, pp. 1485–1497

    2021

  32. [32]

    Traceability transformed: Generating more accurate links with pre-trained BERT models,

    J. Lin, Y. Liu, Q. Zeng, M. Jiang, and J. Cleland-Huang, “Traceability transformed: Generating more accurate links with pre-trained BERT models,” inProc. ICSE, 2021, pp. 324–335

    2021

  33. [33]

    Portfolio: Finding relevant functions and their usages,

    C. McMillan, M. Grechanik, D. Poshyvanyk, C. Fu, and Q. Xie, “Portfolio: Finding relevant functions and their usages,” inProc. ICSE, 2011, pp. 111–120

    2011

  34. [34]

    In defense of soundiness: A manifesto,

    B. Livshits, M. Sridharan, Y. Smaragdakis, O. Lhoták, J. N. Amaral, B.-Y. E. Changet al., “In defense of soundiness: A manifesto,”Communications of the ACM, vol. 58, no. 2, pp. 44–46, 2015

    2015

  35. [35]

    GraphCodeBERT: Pre-training code representations with data flow,

    D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liuet al., “GraphCodeBERT: Pre-training code representations with data flow,” inProc. ICLR, 2021

    2021

  36. [36]

    The Daikon system for dynamic de- tection of likely invariants,

    M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao, “The Daikon system for dynamic de- tection of likely invariants,”Science of Computer Programming, vol. 69, no. 1–3, pp. 35–45, 2007

    2007

  37. [37]

    CodaMosa: Escaping coverage plateaus in test generation with pre-trained large language models,

    C. Lemieux, J. P. Inala, S. K. Lahiri, and K. Sen, “CodaMosa: Escaping coverage plateaus in test generation with pre-trained large language models,” inProc. ICSE, 2023

    2023

  38. [38]

    Fuzz4All: Universal fuzzing with large language models,

    C. S. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L. Zhang, “Fuzz4All: Universal fuzzing with large language models,” in Proc. ICSE, 2024

    2024

  39. [39]

    Software architecture as a set of architectural design decisions,

    A. Jansen and J. Bosch, “Software architecture as a set of architectural design decisions,” inProc. Working IEEE/IFIP Conference on Software Architecture (WICSA), 2005, pp. 109– 120

    2005

  40. [40]

    10 years of software architecture knowledge management: Practice and future,

    R.Capilla,A.Jansen,A.Tang,P.Avgeriou,andM.A.Babar,“10 years of software architecture knowledge management: Practice and future,”Journal of Systems and Software, vol. 116, pp. 191– 205, 2016

    2016

  41. [41]

    Wohlin, P

    C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén,Experimentation in Software Engineering. Springer, 2012

    2012

  42. [42]

    The measurement of observer agreement for categorical data,

    J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,”Biometrics, vol. 33, no. 1, pp. 159–174, 1977. 16

    1977

  43. [43]

    Claude [large language model],

    Anthropic, “Claude [large language model],” 2025. [Online]. Available: https://www.anthropic.com/claude Appendix A Full Extraction Prompt Template ThefullsystemanduserpromptusedbyC3(Assumption Extractor) in all experiments is: SYSTEM: You are an expert software engineer auditing LLM- generated code for implicit design decisions (implicit assumptions). USE...

    2025

  44. [44]

    For each, name at least one realistic alternative

    REASON: List every design decision embedded in the code that is NOT explicitly required by the prompt. For each, name at least one realistic alternative

  45. [45]

    id": "string (e.g. ’A1’)

    FORMAT: Return a JSON array of AssumptionRecord objects matching this schema: {schema} Taxonomy categories: T1 -- Input format / validation T2 -- Return type / output structure T3 -- Edge-case / error-handling policy T4 -- Persistence / storage backend T5 -- Algorithm / performance trade-off T6 -- Security / authentication policy Return ONLY valid JSON af...

    2025