Pith. sign in

REVIEW 4 major objections 7 minor 37 references

Characterizing the Quality Profile of AI-Generated C++ in Production

T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AI-generated C++ in production carries a concentrated quality burden in coupling and copy overhead, and targeted taxonomy-informed feedback measurably reduces it.

desk verdict First production-scale provenance trace of AI-generated C++: the static quality profile is credible and valuable, but the headline 5-8% compute-cost claim is a confounded growth-gap that should be downgraded. read the letter →

arxiv 2608.06640 v1 pith:U4YLKESE submitted 2026-08-06 cs.SE cs.AI

Michael Tran , Fred Lewis , Kun Yang , Saksham Thakur , Aditya Kini , Aditya Patil , Milad Hashemi , Parthasarathy Ranganathan This is my paper · ORCID
classification cs.SEcs.AI
keywords AI-generatedcodeC++softwarequalitystaticanalysisreviewcomputeefficiencyauthoringprovenanceempiricalengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Using authoring-time provenance on 3.52 million submitted changes and a C++ slice of 10.46 million lines, this paper tries to establish that AI-generated C++ has a distinct, measurable quality profile once it reaches production: more interface and coupling burden, more copy and allocation overhead, and more explicit loops instead of optimized standard-library calls. The paper then argues these upstream patterns show up downstream as extra review effort and a 5-8% increase in normalized compute and memory consumption. It also argues the profile is not a hard model ceiling, because adding targeted, taxonomy-informed feedback to generation cut targeted static-analysis warnings by 11.1% and improved an instruction/memory efficiency score by 31%. If correct, the result gives production teams a concrete, prioritized list of code properties to steer AI assistants away from, and an evaluation standard that goes beyond pass rates and task completion.

What carries the argument

The carrying mechanism is authoring-time provenance: every changed byte is tagged during authoring as AI-generated or human-typed, then projected onto the lines, functions, and static findings of the final submitted snapshot. This byte-level attribution is joined to a three-level static taxonomy, with quality attributes, issue categories, and issue types, that collapses raw static checks into developer-facing mechanisms such as missing-move and include-cleaner. The same provenance feeds function-level compute cohorts, so the paper can connect a line-level property like a hand-written loop instead of a standard-library call to a production outcome like normalized CPU and heap growth. The intervention reuses the taxonomy as prompt content: the coding agent is asked to reimplement functions under base, general-feedback, and category-specific-feedback conditions, with static finding counts and an efficiency score as outcomes.

What would settle it

A re-analysis that recomputes AI-versus-human attribution after fully resolving overlapping authoring bytes, then matches AI-heavy and human-typed function cohorts on task type, repository context, model generation, and developer experience, would settle the central claim. If the 5-8% compute and memory gap and the concentrated static-finding gap disappear under that matching, the reported quality profile is an artifact of cohort differences rather than of AI authorship.

Watch

Extended reading notes

Core claim

The paper's central claim is that AI-generated C++ in a large brownfield production codebase carries a concentrated, patterned quality burden that survives code review and shows up in operating costs. In the submitted code, AI changes are larger and more new-code-heavy, and their excess static-finding rate is concentrated in Interface and Coupling Burden and Copy and Allocation Overhead, which account for 82.21% of the positive absolute rate gap. Source-level measures show AI code authors loops at roughly twice the human rate, uses standard-library or API calls 30-40% less often, and has higher rates of missing-move, use-emplace, and inefficient-map warnings. Downstream, AI changes receive 1.92x more blocking review threads and 1.39x more comments, and deployed AI-heavy functions grow to about 5% more normalized compute and 8% more memory than human-typed functions over the study window. The paper also claims that adding category-specific feedback to a reimplementation prompt reduces targeted static findings by 11.1% and raises the benchmark efficiency score by about 31%, showing the observed weaknesses respond to targeted mitigation.

Load-bearing premise

The load-bearing premise is that byte-level authoring-time provenance, after developers edit and revise their code, still correctly attributes the final submitted and deployed lines to AI versus human authorship, and that the compute cohorts differ in authorship rather than in task mix, model generation, or developer skill.

Editorial extensions

If this is right

  • Reviewers and static-analysis tooling can prioritize Interface and Coupling Burden and Copy and Allocation Overhead, which together make up 82.21% of the positive rate gap.
  • Production evaluation of AI coding assistants should track code past submission: the AI-specific cost appears as review friction and compute growth, not as higher revert rates, which stayed below parity.
  • Category-specific prompt feedback is a working mitigation, cutting targeted static findings by 11.1% and improving the benchmark efficiency score by 31% in the controlled regeneration setting.
  • If the compute and memory gap scales with AI share, large organizations adopting AI assistants can expect a persistent 5-8% resource overhead that grows with adoption, making efficiency part of AI sustainability planning.
  • The results suggest an automated loop where static analysis identifies recurring category-level risks and feeds them back into generation, rather than relying on reviewers to catch localized inefficiencies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that the loop-and-avoid-library pattern is likely a brownfield knowledge problem: generalist models trained largely on public code do not know an enterprise's internal optimized libraries, so the pattern may reflect missing repository knowledge more than model capability.
  • If the profile replicates, the same method should produce language-specific taxonomies for Rust, Go, and Java; the C++ categories such as missing-move and use-emplace will not transfer directly, but the imperative-versus-library contrast may.
  • The paper's own null result on review metrics hints that reviewer effort does not intercept these costs, which would make automated, prompt-time or training-time feedback the only scalable mitigation.
  • Because the pooled AI signal spans multiple model generations and interaction modes, the 5-8% estimate likely mixes older and newer models; a per-model breakdown could show the overhead is partly a legacy-model artifact that later models already avoid.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This manuscript reports a one-year observational study of AI-generated C++ code in a large enterprise monorepo, using authoring-time byte-level provenance to attribute code to AI versus human authorship. It addresses four research questions: where AI-generated code appears (RQ1), what upstream structural and static issue profiles look like (RQ2), how those profiles relate to review, reliability, and compute outcomes (RQ3), and whether taxonomy-informed prompt feedback reduces targeted static warnings and improves benchmark efficiency (RQ4). The paper's central claims are that AI-generated C++ exhibits a distinct quality profile (larger changes, concentrated interface/coupling and copy/allocation static findings, more explicit loops, less standard-library use) that translates into increased review effort and an approximate 5-8% relative growth gap in compute and memory consumption, and that targeted feedback yields an 11.1% reduction in targeted static findings. The scale of the dataset (3.52 million changes, 10.46 million C++ LOC) and the provenance instrumentation are notable strengths, but the downstream causal claims are not fully supported by the reported analyses.

Significance. If the central claims hold, this is the first large-scale production evidence that AI-generated C++ carries a concentrated, measurable quality and efficiency burden beyond controlled benchmarks. The taxonomy and the provenance-based measurement approach are valuable contributions, and the RQ4 feedback intervention is a practical step toward mitigation. However, the load-bearing downstream claims about review effort and compute cost outrun the current evidence: the compute figures are unadjusted growth-gap comparisons, the review figures are unadjusted raw contrasts with no stratified results shown, and the RQ4 outcome reuses the same analyzer that defined the target categories. The paper is nonetheless a strong descriptive industrial study with an unusually complete observation pipeline, and with careful reanalysis or tightened claims it could become an important reference point for empirical software engineering research.

major comments (4)
  1. [§4.3, Figure 4 bottom panels] The abstract's '5-8% increase in compute resource consumption' is a relative growth gap between cohort indices: 1.31x versus 1.25x for compute and 1.36x versus 1.25x for memory, i.e., approximately 5% and 8% relative differences in indexed growth. This comparison is adjusted only for edit volume (as stated in §3.4), with no controls for organizational slice, product area, or function task category. Because §4.1 shows AI adoption is highly uneven across organizational slices (e.g., 70.19% in ML/AI versus lower elsewhere), the observed growth gap may reflect cohort mix rather than per-function code authorship. The sentence 'This confirms that the upstream efficiency issues identified in RQ2 translate into measurable compute and memory overheads at scale' asserts a causal interpretation that this design cannot support. The authors should either present adjusted models for this comparison or explicitly label the figures as descriptive and soften the causal language.
  2. [§3.4 and §4.3] The paper promises 'stratified cohort comparisons' with controls for month, change size, coarse organizational slice, author/team, and new-code status, but the review and reliability results in §4.3 present only unadjusted 'descriptive downstream contrasts' (e.g., 1.92x blocking threads, 1.39x comments, 1.24x reviewer iterations). No adjusted effect sizes, confidence intervals, or model outputs are shown for these outcomes. Given that Table 2 shows AI-generated changes are larger and more new-code-heavy, the raw ratios likely overstate the authorship effect. The claim of 'increased review effort' requires the reported stratified estimates, or the claim should be downgraded to a descriptive association.
  3. [§4.4, Table 6] The RQ4 evaluation is partially circular: the 50 benchmark functions were selected because they contain at least one target-category finding from the same static taxonomy used to build the intervention prompt, and the primary outcome is the count of those same targeted static findings. The 11.1% reduction therefore partly reflects the prompt's direct acknowledgement of the analyzer's preferred issue types, not necessarily an improvement in general code quality. The authors should validate the intervention with out-of-taxonomy measures (e.g., independent manual review, different analyzers, or broader efficiency metrics) or explicitly restrict the claim to reductions on the analyzer-defined target categories in the results section rather than only in the threats-to-validity discussion.
  4. [§3.2 and §6] The causal framing of the compute and review comparisons treats AI-heavy versus human-typed function cohorts as if authorship is the only systematic difference. However, the pooled AI signal spans multiple model generations and interaction modes, and model identifiers are masked (§6). Developer editing during authoring can blur the byte-level provenance, as acknowledged in §6. These factors mean the observed differences should be interpreted as specific to the deployment under study, not as a general 'AI effect.' The abstract and conclusion should qualify the claims accordingly, and the authors should report sensitivity analyses that vary the AI-share threshold or restrict to stable sub-cohorts if such analyses are feasible.
minor comments (7)
  1. [Abstract and §4.3] The abstract says '5-8% increase in compute resource consumption,' but the text reports relative growth gaps between cohort indices; the phrase 'relative increase' should appear in the abstract to avoid implying a direct causal increase in absolute resource consumption.
  2. [Figure 4] The bottom panel labels read 'Compute Growth (2025-05 = 1)' and 'Memory Growth (2025-05 = 1)', but the text says the baseline is 'at the start of our study window' (April 1, 2025). Please align the baseline date in the figure with the study window definition.
  3. [§4.4, Table 6] The R_eff score is described as mapping instruction/memory comparison to three values (1, 0.5, 0), but the table reports averages like 0.294; explain how the coarse per-function scores are aggregated and what a score of 0.294 means in practical terms.
  4. [§4.4] In Table 6, the baseline row has no R_eff value; the text explains that baseline is the original code, but the table should either include R_eff = 1.0 by definition or an explicit 'not applicable' note.
  5. [§3.1 and §4.4] The phrase 'A intervention' appears at the beginning of §3.1; it should be 'An intervention.'
  6. [§4.2, Table 4] The row for 'Standard library use' reports a ratio of '~0.4x' with the description 'std::* usages'; the text in §4.2 says 'uses standard-library/API calls 30–40% less often,' which is consistent, but the table should state the exact normalized ratio or range to avoid ambiguity.
  7. [§3.3] The taxonomy validation reports '2 annotators' without giving the inter-rater agreement value; the text says the validation records it, but the number is not reported in the manuscript. Please include the agreement statistic or state that it was computed and used for the final mapping.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: provenance, static findings, review metadata, and compute outcomes are measured on independent surfaces, and the RQ4 feedback result is a controlled benchmark with an independent efficiency score.

full rationale

The paper's load-bearing comparisons do not reduce to their own inputs. RQ2's static taxonomy is constructed from raw static-analysis checks and then compared across AI/human provenance groups; the taxonomy is not defined in terms of the downstream outcomes. RQ3's review and compute outcomes are measured on separate surfaces (review metadata; production CPU/memory monitoring normalized to application share), and the compute comparison is stratified by edit volume. The 5-8% compute-cost gap is an observational cohort contrast, not a fitted parameter renamed as a prediction. RQ4's 11.1% reduction is measured on a synthesized 50-function benchmark against a Stage 1 no-feedback baseline; the secondary R_eff metric from CPU instructions and memory is independent of the static analyzer used to select the functions, so the improvement is not forced by construction. The only overlapping-author citation (ECO, [22]) is motivational context rather than a load-bearing premise or uniqueness theorem. The paper itself discloses model-mix masking and provenance ambiguity, which are validity and confounding concerns rather than circularity. No step in the derivation chain substitutes its conclusion for an assumption.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The central claims rest on four hand-chosen analysis parameters (support threshold, cohort thresholds, Reff weight, benchmark size) and five domain assumptions about provenance accuracy, taxonomy validity, observational cohort comparability, benchmark representativeness, and finding transfer. The two invented entities are measurement artifacts, not physical postulates, and both lack external falsifiable handles.

free parameters (4)
  • Support threshold for issue-type reporting = 1,000 findings
    Issue types with combined support below 1,000 are excluded from reporting, which shapes the taxonomy profile and could hide small but material categories.
  • Cohort definition thresholds for AI-heavy vs human-typed functions = not specified exactly; approximately 25k AI-heavy and 12k human-typed functions, stratified to ~6k matched each
    The definition of AI-heavy versus human-typed is not stated as a precise threshold, and matching by edit volume is described only at the level of approximate cohort sizes; the compute estimates depend on this choice.
  • Reff instruction-to-memory weight = 2:1 (instruction count weighted twice memory)
    The composite efficiency score in RQ4 is computed with a hand-chosen 2x weight because latency is deemed the main cost channel; changing this weight changes the reported improvement.
  • Intervention benchmark size and runs = 50 functions, 3 prompt stages, 3 independent runs
    The intervention's sample is selected from functions with at least one target-category finding; the small size and selection rule bound the generality of the 11.1% and 31% improvements.
assumptions (5)
  • domain assumption Authoring-time byte-level provenance accurately attributes AI versus human authorship after developer edits, revision mapping, and mixed-authorship changes.
    Section 3.2 describes projection of byte-level annotations onto final submitted snapshots; Section 6 acknowledges overlap and revision-mapping measurement choices.
  • domain assumption Static-analysis findings mapped through the 3-level taxonomy are valid proxies for code quality and developer burden.
    Section 3.3 builds the taxonomy from raw checks with annotator validation and expert review, but there is no external ground truth that the categories correspond to real maintenance cost.
  • domain assumption Normalized percent-of-application compute and memory share, stratified by edit volume, isolates authorship effects from unmeasured confounders such as task difficulty, developer skill, and model mix.
    Section 4.3 and Section 6 Internal validity; model identifiers are masked, so the pooled AI signal mixes multiple model generations and interaction modes.
  • domain assumption The 50-function benchmark and the three prompt stages represent production conditions well enough to support claims about mitigation.
    Section 4.4 calls the intervention a controlled generation setting and Section 6 notes it is narrower than an organization-wide randomized rollout.
  • domain assumption Static findings in the original function transfer meaningfully to the synthesized reimplementation benchmark.
    Section 4.4 and Figure 2 describe reimplementation while preserving inputs, outputs, and intended logic, but the re-written bodies may not carry the same findings.
invented entities (2)
  • 3-level static quality taxonomy (quality attributes, issue categories, issue types)
    purpose: Group raw static-analysis checks into developer-facing categories used for the RQ2 profile and RQ4 feedback targets.
    Validated internally with annotators and expert review, but it is constructed from the same tools used to measure outcomes and has no external benchmark or falsifiable prediction outside the paper.
  • R_eff composite efficiency score
    purpose: Quantify improvement in the intervention via weighted CPU instruction count and memory usage relative to the original implementation.
    Defined within the paper with a hand-chosen 2:1 instruction/memory weight; no external validation that it predicts production efficiency.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Characterizing the Quality Profile of AI-Generated C++ in Production." pith.science (2026). https://pith.science/paper/U4YLKESE

@misc{pith2026260806640,
  author       = {Pith},
  title        = {Pith review of: Characterizing the Quality Profile of AI-Generated C++ in Production},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U4YLKESE}},
  note         = {Machine review of arXiv:2608.06640}
}
read the original abstract

The widespread integration of AI coding assistants offers undeniable boosts to engineering velocity. Yet, recent studies point to a growing trade-off, revealing persistent challenges with code quality and maintainability. Industry leaders, including frontier AI labs, echo these concerns. As large language models are increasingly relied upon to author production code, understanding their impact on shipped software quality has become a critical priority. However, assessing these effects in industrial workflows remains difficult due to observability barriers. We study the impact of AI-generated code on production quality within a large enterprise operating global products relied upon by billions of users daily. Driven by this scale and user trust, the organization values code quality and has built thorough observability for every line of code deployed into production, enabling us to overcome measurement barriers to assess these effects. This study presents a large-scale empirical analysis of AI-generated C++ code from April 2025 to April 2026, tracking 3.52 million code changes across this enterprise's brownfield codebase. The core purpose is to understand the quality, performance, and maintenance characteristics of AI-generated code compared to human-written code in a production environment at scale. We find that AI-generated C++ code has a distinct quality profile, showing higher rates of interface and coupling burdens, copy and allocation overheads, and a reliance on explicit loops over optimized standard APIs. These issues translate into tangible downstream costs, including increased review effort and a 5-8% increase in compute resource consumption. However, we demonstrate that providing models with targeted, taxonomy-informed feedback can mitigate these effects, leading to an 11.1% reduction in targeted static analysis warnings and improved computational efficiency.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 14 canonical work pages

  1. [1]

    Al-Fawareh and H

    H. Al-Fawareh and H. M. Al-Shdaifat. Investigating ai-generated code on the impact on soft- ware efficiency code quality factor. InSustainable Data Management: Navigating Big Data, Communication Technology, and Business Digital Leadership. Volume 1, pages 89–97. Springer, 2025

    work page 2025

  2. [3]

    Anderson, G

    E. Anderson, G. Parker, and B. Tan.The hidden costs of coding with generative AI. MIT Sloan Management Review, 2025. 17 Characterizing the Quality Profile of AI-Generated C++ in Production

    work page 2025

  3. [4]

    When ai builds itself, June 2026

    Anthropic. When ai builds itself, June 2026. URL https://www.anthropic.com/ institute/recursive-self-improvement

    work page 2026

  4. [5]

    Generatingenergy- efficient code via large-language models – where are we now?CoRR, abs/2509.10099, 2025

    R.Apsan, V.Stoico, M.Albonico, R.Dhar, K.Vaidhyanathan, andI.Malavolta. Generatingenergy- efficient code via large-language models – where are we now?CoRR, abs/2509.10099, 2025. doi: 10.48550/arXiv.2509.10099. URLhttps://arxiv.org/abs/2509.10099. Accepted at ICSE 2026 Research Track

    arXiv doi:10.48550/arxiv.2509.10099 2025

  5. [6]

    Barke, M

    S. Barke, M. B. James, and N. Polikarpova. Grounded copilot: How programmers interact with code-generating models.Proceedings of the ACM on Programming Languages, 7(OOPSLA1): 85–111, 2023. doi: 10.1145/3586030

    doi:10.1145/3586030 2023

  6. [7]

    Blyth, S

    S. Blyth, S. A. Licorish, C. Treude, and M. Wagner. Static analysis as a feedback loop: Enhancing llm-generated code beyond correctness. In2025 IEEE International Conference on Source Code Analysis & Manipulation ( SCAM), pages 100–109, 2025. doi: 10.1109/SCAM67354.2025.00017

    arXiv 2025

  7. [11]

    Coutinho, L

    M. Coutinho, L. Marques, A. Santos, M. Dahia, C. França, and R. de Souza Santos. The role of generative AI in software development productivity: A pilot case study. InProceedings of the 1st ACM International Conference on AI-Powered Software, AIware ’24, pages 131–138, 2024. doi: 10.1145/3664646.3664773. URLhttps://doi.org/10.1145/3664646.3664773

    arXiv 2024

  8. [14]

    H. He, C. Miller, S. Agarwal, C. Kästner, and B. Vasilescu. Speed at the cost of quality: How cursor ai increases short-term velocity and long-term complexity in open-source projects.arXiv preprint arXiv:2511.04427, 2025. 18 Characterizing the Quality Profile of AI-Generated C++ in Production

    arXiv 2025

Show all 37 references
  1. [15]

    V. J. Hellendoorn, S. Proksch, H. C. Gall, and A. Bacchelli. When code completion fails: A case study on real-world completions. InProceedings of the 41st IEEE/ACM International Conference on Software Engineering, ICSE ’19, pages 960–970, 2019. doi: 10.1109/ICSE.2019.00101

    2019

  2. [16]

    Huynh and B

    N. Huynh and B. Lin. Large language models for code generation: A comprehensive survey of challenges, techniques, evaluation, and applications.arXiv preprint arXiv:2503.01245, 2025

    2025 arXiv

  3. [17]

    Ivanković, G

    M. Ivanković, G. Petrović, Y. Kulizhskaya, M. Lewko, L. Kalinovčić, R. Just, and G. Fraser. Productive coverage: Improving the actionability of code coverage. InProceedings of the International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP 202...

    2024

  4. [18]

    Izadi, J

    M. Izadi, J. Katzy, T. van Dam, M. Otten, R. M. Popescu, and A. van Deursen. Language models for code completion: A practical evaluation. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE ’24, pages 1–13, 2024. doi: 10.1145/3597503. 3639...

    2024

  5. [20]

    Lattner and V

    C. Lattner and V. Adve. Llvm: A compilation framework for lifelong program analysis & transformation. InInternational symposium on code generation and optimization, 2004. CGO 2004., pages 75–86. IEEE, 2004

    2004

  6. [21]

    S. Li, Y. Cheng, J. Chen, J. Xuan, S. He, and W. Shang. Assessing the performance of AI-generated code: A case study on GitHub copilot. In2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE), pages 216–227, 2024. doi: 10.1109/ISSRE62328.2024.00030...

    2024

  7. [22]

    H. Lin, M. Maas, M. Roquemore, A. Hasanzadeh, F. Lewis, Y. Simonson, T.-W. Yang, A. Yazdan- bakhsh, D. Altinbüken, F. Papa, M. N. Edmonds, A. Patil, D. Schwarz, S. Chandra, C. Kennelly, M. Hashemi, and P. Ranganathan. Eco: An llm-driven efficient code optimizer for warehouse s...

    2025 arXiv

  8. [23]

    Y. Liu, T. Le-Cong, R. Widyasari, C. Tantithamthavorn, L. Li, X. D. Le, and D. Lo. Refining ChatGPT-generated code: Characterizing and mitigating code quality issues.ACM Transactions on Software Engineering and Methodology, 33(5):1–26, 2024. doi: 10.1145/3643674. URL https://d...

    2024 doi

  9. [24]

    Mozannar, G

    H. Mozannar, G. Bansal, A. Fourney, and E. Horvitz. Reading between the lines: Modeling user behavior and costs in AI-assisted programming. InProceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–16, 2024. doi: 10.1145/3613904.3641936. URL https://do...

    2024

  10. [25]

    M. A. F. Muñoz, J. C. J. De La Torre, S. P. López, S. Herrera, and C. A. C. Uribe. Comparative study of ai code generation tools: quality assessment and performance analysis.LatIA, (2):21, 2024

    2024

  11. [26]

    S. K. Ngassom, A. M. Dakhel, F. Tambon, and F. Khomh. Chain of targeted verification questions to improve the reliability of code generated by LLMs. InProceedings of the 1st ACM International Conference on AI-Powered Software, AIware ’24, pages 122–130, 2024. doi: 10.1145/3664...

    2024 doi

  12. [27]

    Paradis, K

    E. Paradis, K. Grey, Q. Madison, D. Nam, A. Macvean, V. Meimand, N. Zhang, B. Ferrari-Church, andS.Chandra. HowmuchdoesAIimpactdevelopmentspeed? anenterprise-basedrandomized controlled trial. In2025 IEEE/ACM 47th International Conference on Software Engineering: Soft- ware Eng...

    2025

  13. [28]

    Pearce, B

    H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri. Asleep at the keyboard? assessing the security of GitHub copilot’s code contributions. In2022 IEEE Symposium on Security and Privacy ( SP), pages 754–768, 2022. doi: 10.1109/SP46214.2022.9833571

    2022

  14. [29]

    S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer. The impact of AI on developer productivity: Evidence from GitHub copilot.CoRR, abs/2302.06590, 2023. doi: 10.48550/arXiv.2302.06590

    2023 doi arXiv

  15. [31]

    gradient descent

    R. Pryzant, D. Iter, J. Li, Y. Lee, C. Zhu, and M. Zeng. Automatic prompt optimization with “gradient descent” and beam search. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 7957–7968, 2023

    2023

  16. [32]

    P. K. Rajput, Y. Song, A. A. Bonkoungou, I. E. Olatunji, A. K. Kaboré, J. Klein, and T. F. Bis- syandé. Correctness isn’t efficiency: Runtime memory divergence in LLM-generated code.CoRR, abs/2601.01215, 2026. doi: 10.48550/arXiv.2601.01215. URLhttps://arxiv.org/abs/ 2601.01215

    2026 doi

  17. [33]

    Sadowski, J

    C. Sadowski, J. van Gogh, C. Jaspan, E. Söderberg, and C. Winter. Tricorder: Building a program analysis ecosystem. In2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, pages 598–608, 2015. doi: 10.1109/ICSE.2015.76

    2015 doi

  18. [34]

    Sadowski, E

    C. Sadowski, E. Söderberg, L. Church, M. Sipko, and A. Bacchelli. Modern code review: A case study at google. InProceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP 2018, pages 181–190, 2018. doi: 10.1145/3183519...

    2018 doi

  19. [35]

    Salerno, A

    F. Salerno, A. Al-Kaswan, and M. Izadi. How much do code language models remember? an investigation on data extraction attacks before and after fine-tuning. In2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR), pages 465–477, 2025. doi: 10.1109/M...

    2025

  20. [36]

    Sandoval, H

    G. Sandoval, H. Pearce, T. Nys, R. Karri, S. Garg, and B. Dolan-Gavitt. Lost at C: A user study on the security implications of large language model code assistants. In32nd USENIX Security Symposium (USENIX Security 23), pages 2205–2222, 2023

    2023

  21. [37]

    M. L. Siddiq, S. H. Majumder, M. R. Mim, S. Jajodia, and J. C. S. Santos. An empirical study of code smells in transformer-based code generation techniques. In2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation ( SCAM), pages 71–82, 2022. d...

    2022

  22. [38]

    State of code developer survey report

    Sonar. State of code developer survey report. Technical report, SonarSource, 2026. URL https://www.sonarsource.com/state-of-code-developer-survey-report.pdf. 20 Characterizing the Quality Profile of AI-Generated C++ in Production

    2026

  23. [40]

    Takerngsaksiri, J

    W. Takerngsaksiri, J. Pasuksmit, P. Thongtanunam, C. Tantithamthavorn, R. Zhang, F. Jiang, J. Li, E. Cook, K. Chen, and M. Wu. Human-in-the-loop software development agents. InProceedings of the IEEE/ACM International Conference on Software Engineering: Software Engineering in...

    2025

  24. [41]

    Usman, R

    M. Usman, R. Britto, J. Börstler, and E. Mendes. Taxonomies in software engineering: A systematic mapping study and a revised taxonomy development method.Information and Software Technology, 85:43–59, 2017. doi: 10.1016/j.infsof.2017.01.006

    2017 doi

  25. [42]

    Vijayvergiya, M

    M. Vijayvergiya, M. Salawa, I. Budiselic, D. Zheng, P. Lamblin, M. Ivankovic, J. Carin, M. Lewko, J. Andonov, G. Petrovic, D. Tarlow, P. Maniatis, and R. Just. Ai-assisted assessment of coding practices in modern code review. InProceedings of the 1st ACM International Conferen...

    2024

  26. [44]

    Z. Wang, Z. Zhou, D. Song, Y. Huang, S. Chen, L. Ma, and T. Zhang. Towards understanding the characteristics of code generation errors made by large language models. InProceedings of the 47th IEEE/ACM International Conference on Software Engineering, ICSE ’25, pages 2587–2599,

  27. [45]

    W. Wu, H. Hu, Z. Fan, Y. Qiao, Y. Huang, Y. Li, Z. Zheng, and M. R. Lyu. An empirical study of code clones from commercial AI code generators.Proceedings of the ACM on Software Engineering, 2(FSE):2874–2896, 2025. doi: 10.1145/3729397. URLhttps://doi.org/10. 1145/3729397

    2025 doi

  28. [46]

    Yetistiren, I

    B. Yetistiren, I. Ozsoy, and E. Tuzun. Assessing the quality of GitHub copilot’s code generation. InProceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering, PROMISE ’22, pages 62–71, 2022. doi: 10.1145/3558489.3559072. 21

    2022

  29. [2025]

    doi: 10.1109/ICSE55347.2025.00180

    2025

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.