Pith. sign in

REVIEW 4 major objections 6 minor 93 references

The paper argues that libraries with similar functionality can differ by up to 84% in the quality of code an LLM generates with them, and that technology selection should treat this 'AI coding proficiency' as a first-class decision factor.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 17:02 UTC pith:X57TRUBF

load-bearing objection First large-scale mapping of LLM-vs-library compatibility, but every headline number sits on an unvalidated LLM judge—read the magnitudes with salt, the construct is worth engaging. the 4 major comments →

arxiv 2509.11132 v2 pith:X57TRUBF submitted 2025-09-14 cs.SE

Rethinking Technology Stack Selection with AI Coding Proficiency

Xiaoyu Zhang , Weipeng Jiang , Shiqing Ma , Qingshuang Bao , Chenhao Lin , Chao Shen , Tianlin Li , Juan Zhai This is my paper
classification cs.SE
keywords AI coding proficiencytechnology stack selectionLLM code generationthird-party librariescode quality evaluationLLM-as-a-judgeecosystem diversityprompt engineering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that technology stack selection in the LLM era is missing a dimension: how well an AI model can actually use a given technology. The authors define this as 'AI coding proficiency' and measure it by the quality of code snippets that six LLMs generate across 170 Python libraries and 61 task scenarios. They find that libraries with similar functionality can differ by up to 84% in code-quality score, that the 'better' library often changes depending on the model, and that low proficiency shows up as concrete failure patterns such as incorrect functionality and missing edge-case handling. A sympathetic reader would take this as evidence that choosing a library without considering the model that will write the code invites debugging cost and technical debt, and that selection frameworks should evaluate library-model pairs rather than libraries alone.

Core claim

The central discovery is that AI coding proficiency—a property of a technology, defined as the degree to which a given LLM can accurately understand and invoke its APIs to produce high-quality code snippets—varies widely and systematically across libraries and models. Across 170 Python libraries, six LLMs, and 850 prompts, the authors find that 11.17% of competing library pairs show significant quality-score gaps, with the largest pair difference reaching 84%. The winning library changes with the model for 86.17% of competing pairs, and the ranking of models by overall code quality shows no significant correlation with their general coding-capability ranking. The authors interpret these gaps

What carries the argument

The central object is the AI coding proficiency score P, defined as the average, over all prompts involving library l in scenario s, of the overall quality score S of code generated by model m. The overall quality score is the arithmetic mean of five normalized dimensions: functional suitability (LLM-judged correctness), performance (LLM-estimated time and space complexity), maintainability (Maintainability Index), usability (a readability model), and reliability (LLM-judged edge-case and exception handling). The machinery also includes a dataset pipeline that generates task descriptions with GPT-4o, refines scenarios with expert consensus, and forms 188 competing library pairs, analyzed wit

Load-bearing premise

The paper's measurements rest on the assumption that the automated quality scores—especially the LLM judge's ratings of functional suitability, performance, and reliability, which are not validated against human judgment—are valid and unbiased across libraries; if the judge systematically favors libraries that resemble its training data, the observed 84% gaps could be artifacts of the evaluation rather than real properties of the libraries.

What would settle it

Have independent human experts with Python experience score a random sample of the 25,500 generated code snippets on the same five dimensions and compare their ratings to the automated scores; if agreement is low, or if rerunning the pipeline with human-written task descriptions instead of GPT-4o-generated ones eliminates most of the large library-pair gaps, the central claim would be undercut.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Technology selection frameworks should add AI coding proficiency as a dimension alongside traditional attributes like performance and maintainability.
  • Developers should evaluate library-model combinations jointly, because the 'winner' library changes with the model for 86.17% of competing pairs.
  • Low AI proficiency translates into debugging effort, integration costs, and technical debt that offset the productivity gains of AI-assisted development.
  • AI proficiency gaps can drive a 'winner-takes-all' dynamic in the software ecosystem, reducing diversity and amplifying supply-chain attack risks.
  • Few-shot prompting and regeneration can improve overall code quality and shrink inter-library gaps, though they may reverse the leading library for close pairs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • AI proficiency is a moving target: as new model versions train on newer library documentation and examples, current gaps may shrink or shift, so proficiency measurements would need periodic refresh to stay actionable.
  • The concept generalizes beyond Python libraries—the same methodology could benchmark frameworks, platforms, and agentic workflows, adding companion dimensions like 'AI efficiency' and 'agent proficiency'.
  • Library maintainers can directly influence their AI proficiency by publishing clean, up-to-date examples and minimizing API churn, an implication the paper gestures toward in its future-work discussion.
  • A testable next step is to connect the quality-score proxy to real engineering cost: teams that select high-proficiency library-model pairs could be measured for lower debugging frequency and rework rates in production.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper introduces 'AI coding proficiency' for technology stack selection: the degree to which an LLM can use a given library to produce high-quality code snippets. The authors construct a dataset of 170 Python libraries across 61 task scenarios, generate 850 prompts with GPT-4o, query six LLMs five times per prompt, and score each snippet on five dimensions (functional suitability, performance, maintainability, usability, reliability) using an o1-mini judge for three dimensions, the Maintainability Index, and a readability model. The average of the five scores is the library's AI proficiency for a model. RQ1 reports gaps between competing libraries (11.17% of 188 pairs exceed Cohen's d >= 0.5), model-dependent winners, and no correlation with coding leaderboards; RQ2 builds a human-annotated taxonomy of eight failure patterns; RQ3 tests CoT, few-shot, and regeneration prompting. The paper calls for incorporating AI proficiency into technology selection.

Significance. The paper is timely and is the first large-scale attempt to quantify model-library fit. Strengths include the scale (25,500 responses), repeated runs, bootstrap CIs, public artifacts, human annotation for the failure-pattern taxonomy, and an intervention study. If the measurement pipeline were validated, the concept would be a useful complement to existing selection frameworks. The central weakness is that the numeric scores driving all rankings and gaps are produced by an unvalidated o1-mini judge for three of the five dimensions; this makes the magnitudes (up to 84%) and model rankings (Fig. 5) vulnerable to judge artifacts. The study is publishable after either providing validity evidence or weakening the claims.

major comments (4)
  1. [§4.2; §5.1; §5.3] Functional suitability, performance, and reliability are scored by o1-mini with no validation against human judgments or executable ground truth. These three dimensions enter every AI proficiency score, so the aggregate gaps in Table 1, the model ranking in Fig. 5, and the 84% motivating figure all rest on a single unvalidated judge. The human annotation in RQ2 validates the failure-pattern taxonomy on 814 low-quality snippets, not the continuous scores on the full corpus. Please add (a) a stratified sample of snippets rated by human experts with inter-rater reliability statistics, (b) agreement/calibration between o1-mini and humans, and (c) a test for systematic judge bias across libraries and generating models, especially OpenAI vs non-OpenAI models. Without this, the claimed gaps could reflect judge blind spots rather than library proficiency.
  2. [§3.2; abstract] The 'up to 84% differences' headline is computed from a single prompt and one generated pair (Shapely vs GDAL). The text says repeated runs show the same pattern and the introduction mentions 'up to 20.21% on repeated runs', but no distribution, bootstrap CI, or effect size is given for this pair. As the paper stands, the strongest quantitative claim in the abstract rests on two generated snippets. Please report the full distribution and CI for this motivating case, or reframe it as an anecdote and base the abstract's magnitude claim on the aggregate RQ1 analysis.
  3. [§4.1] The task descriptions are generated by GPT-4o and validated by GPT-4o, with no human check that descriptions are equally natural and achievable for every library in a competing pair. If a generated description aligns better with one library's API conventions, the measured AI proficiency gap is confounded with prompt-library alignment. Please add a human-validated sample of descriptions or a robustness check using multiple independently phrased descriptions per scenario.
  4. [§4.2; Table 1] AI proficiency is an unweighted arithmetic mean of five dimensions, but no sensitivity analysis is provided for alternative weightings. Also, Table 1 labels 'Significant Quality Differences' using Cohen's d >= 0.5, which is an effect-size threshold, not a significance test; the mention of t-tests in §5.1 lacks p-values and multiplicity control across 188 pairs. Please report whether the rankings and gap conclusions are robust to weighting choices, and either report significance tests with correction or relabel the table as effect-size-based.
minor comments (6)
  1. [Table 1] '9,04' should be '9.04'.
  2. [§3.2] 'Shapley' should be 'Shapely' in the phrase 'compared with the code snippet using Shapley'.
  3. [§3.2; §5.2] The relation between 'up to 20.21% on repeated runs' and the 84.89% improvement is not defined; specify which quantity these numbers refer to.
  4. [§4.2] The equation contains 'overal quality score'; should be 'overall'.
  5. [§4.1; §5.1] The counts (170 libraries, 61 scenarios, 850 prompts, 25,500 responses) imply 5 prompts per library and 5 repeats per prompt, but this is never stated explicitly; make the multiplication explicit.
  6. [Header] The ACM template year '2018' in the header should be updated.

Circularity Check

0 steps flagged

No circularity: AI proficiency is explicitly defined as a measured code-quality score; the reported gaps are summaries of that metric, and the LLM-judge validity concern is a measurement threat, not a circular reduction.

full rationale

The paper's derivation chain is definitional rather than circular. AI coding proficiency is explicitly operationalized in §3.1 and §4.2 as the average overall quality score of LLM-generated snippets for a given library and model (P = (1/|P_{l,s}|) Σ S_m^p). The headline findings—quality gaps between competing libraries and model-dependent rankings—are statistical summaries of this defined metric, not predictions derived from it. No equation reduces a predicted quantity to an input: the maintainability score uses the deterministic Maintainability Index, the usability score uses a fixed readability model, and the functional/performance/reliability scores are produced by a fixed o1-mini judge that is not fitted to the reported outcomes. Using an LLM judge raises a genuine measurement-validity concern (potential systematic bias, no human validation), but this is not circularity: the judge is an external, fixed evaluation instrument rather than the target quantity itself. The one self-citation, [92], is used only for scenario collection and prompt-template provenance in §4.1, and the prompt template is reproduced in full; no load-bearing conclusion is imported from the authors' prior work. No uniqueness theorem, fitted parameter renamed as prediction, or ansatz smuggled via citation appears. Therefore the paper is self-contained with respect to circularity; score 0.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central empirical claims rest on an unvalidated LLM-judge scoring pipeline and hand-chosen aggregation thresholds. The only invented entity is the AI coding proficiency construct itself, which lacks external validation. The free parameters are not fitted to data in a regression sense, but their choices directly shape the reported gap statistics.

free parameters (3)
  • Equal weighting of five quality dimensions
    Overall quality score S is the arithmetic mean of functional suitability, performance, maintainability, usability, and reliability (Section 4.2). This hand-chosen weighting directly defines AI proficiency and all gap comparisons.
  • Cohen's d significance threshold = 0.5
    Used in Section 5.2 to flag significant quality differences between competing library pairs; the 11.17% figure depends on this choice.
  • IQR low-quality threshold = Q1 - 1.5*IQR
    Used in Section 5.3 to select 814 low-quality snippets for failure pattern analysis; a standard but hand-chosen cutoff.
axioms (4)
  • domain assumption LLM-as-a-judge (o1-mini) provides valid, unbiased quality scores for functional suitability, performance, and reliability
    Invoked in Section 4.2 and 5.1 with no validation against human raters or ground truth; all central comparisons depend on it.
  • domain assumption GPT-4o-generated and GPT-4o-validated task descriptions are equally fair to all competing libraries
    Section 4.1 relies on a single LLM to create and validate prompts; no human or independent check that prompts do not favor particular library API styles.
  • ad hoc to paper The five-dimension average is a valid operationalization of AI coding proficiency
    Definition 3.1 and Section 4.2 define the construct via the paper's own scoring pipeline; construct validity is assumed, not tested.
  • domain assumption Expert identification of competing library pairs correctly captures functional interchangeability
    Section 4.1 relies on two experts plus a third for consensus; no inter-rater reliability metric is reported.
invented entities (1)
  • AI coding proficiency no independent evidence
    purpose: A new construct proposed as a technology-level property: how effectively an LLM can use a given technology to produce high-quality code.
    Defined in Definition 3.1 and measured entirely through the paper's own pipeline; no external benchmark, dataset, or predictive consequence outside this paper validates the construct.

pith-pipeline@v1.3.0-alltime-deepseek · 25252 in / 15154 out tokens · 142435 ms · 2026-08-04T17:02:46.389909+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are now an integral part of software development workflows and are reshaping the whole process. However, existing technology selection methods mainly focus on the inherent attributes of technologies, overlooking whether the LLM can effectively leverage the chosen technology. Therefore, teams using LLM assistants risk choosing technologies that cannot be used effectively by LLMs, yielding high debugging effort and mounting technical debt. We foresee a practical question in the LLM era, is a technology ready for AI-assisted development? In this paper, we first propose the concept, AI coding proficiency, the degree to which LLMs can utilize a given technology to generate high-quality code snippets. We then conduct the first large-scale empirical study examining AI coding proficiency across 170 third-party libraries and six LLMs. Our findings reveal that libraries with similar functionalities can exhibit up to 84% differences in the quality score of LLM-generated code. These gaps can be translated into real engineering costs and steer developer choices toward a narrow set of technologies, threatening technological diversity in the ecosystem. We call on the community to integrate AI coding proficiency into technology selection frameworks and develop mitigation strategies, preserving competitive balance in AI-driven development.

Figures

Figures reproduced from arXiv: 2509.11132 by Chao Shen, Chenhao Lin, Juan Zhai, Qingshuang Bao, Shiqing Ma, Tianlin Li, Weipeng Jiang, Xiaoyu Zhang.

Figure 1
Figure 1. Figure 1: Evolution of the Technology Stack Selection. • We define AI coding proficiency of technologies, providing a novel perspective for technology stack selection in the LLM era. • We construct a comprehensive evaluation methodology, covering a code quality assessment framework with five dimensions and a comprehensive dataset covering 170 libraries, providing a solid foundation for future AI code proficiency res… view at source ↗
Figure 2
Figure 2. Figure 2: A Motivation Case on GPT-4o Usability measures the ease with which users can understand and apply the code to achieve their goals. Traditional evaluation is mainly human-centric, relying on methods like developer surveys, cognitive complexity studies, and think-aloud protocols to assess a library’s or system’s learning curve and ease of use [31, 65]. However, these methods are not scalable for automaticall… view at source ↗
Figure 3
Figure 3. Figure 3: A Radar Chart for Selected Libraries Across Five Code Quality Dimensions Two technologies with similar functionality or even sim￾ilar popularity can have significant differences in AI pro￾ficiency. Consequently, selecting a technology without considering AI coding proficiency can introduce extra debugging costs and even hidden risks for software devel￾opment. To motivate our study, [PITH_FULL_IMAGE:figure… view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of Code Quality Across Different LLMs metrics in this study use the ‘O1-mini-2024-09-12’ model developed by OpenAI to efficiently obtain high-quality evaluation results. 5.2 RQ1: Proficiency Gap 2.94% 10.00% 2.35% 54.71% 16.47% 13.53% GPT-4o-mini GPT-4o Claude Sonnet 4 Gemini 2.5 Flash Qwen3-Coder-Plus DeepSeek-R1 [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of LLMs that Achieve the Best Quality Scores Across 170 Libraries To systematically study the AI proficiency of different libraries across models, we analyze a total of 25,500 re￾sponses from six LLMs. For each model response, we compute the quality score of its code snippets using the metrics in §4.2. We then conduct two complemen￾tary comparisons. ❶ Model-level. We first compute each model’s… view at source ↗
Figure 6
Figure 6. Figure 6: The Quality of Code Snippets Gen￾erated by Different LLMs For each library, we evaluate the differences between its AI proficiency scores across different LLMs. As shown in [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Code Quality Scores of Competing Libraries on Two LLMs Finding 2: The best choice of LLMs with the highest proficiency score varies across libraries. Gemini 2.5 Flash can achieve the highest score on the most libraries (93/170), while GPT-4o￾mini and Claude Sonnet 4 lead on only 4 libraries. It underscores that the model choice in AI-assisted development should be technology-specific, guided by the AI prof… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

93 extracted references · 1 canonical work pages

  1. [1]

    Federal Trade Commission Act

    1914. Federal Trade Commission Act

    1914

  2. [2]

    What is AI readiness? https://www.ataccama.com/blog/ai-readiness

    2024. What is AI readiness? https://www.ataccama.com/blog/ai-readiness

    2024

  3. [3]

    Our repository

    2025. Our repository. https://anonymous.4open.science/r/AICodingProficiency-15C0/

    2025

  4. [4]

    Aider. 2025. Aider LLM Leaderboards. https://aider.chat/docs/leaderboards/

    2025

  5. [5]

    Anthropic. 2025. Claude 4 Sonnet. https://www.anthropic.com/claude/sonnet

    2025

  6. [6]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732 (2021)

    Pith/arXiv arXiv 2021

  7. [7]

    Cornel Barna, Marin Litoiu, and Hamoun Ghanbari. 2011. Autonomic load-testing framework. InProceedings of the 8th ACM international conference on Autonomic computing. 91–100

    2011

  8. [8]

    2021.Software architecture in practice

    Len Bass, Paul Clements, and Rick Kazman. 2021.Software architecture in practice. Addison-Wesley Professional

    2021

  9. [9]

    Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. 2023. Prompting is programming: A query language for large language models.Proceedings of the ACM on Programming Languages7, PLDI (2023), 1946–1969

    2023

  10. [10]

    1988.A practical handbook for software development

    Nick D Birrell, Martyn A Ould, and MA Ould. 1988.A practical handbook for software development. Cambridge University Press

    1988

  11. [11]

    Briand, John W

    Lionel C. Briand, John W. Daly, and Jurgen K Wust. 2002. A unified framework for coupling measurement in object- oriented systems.IEEE Transactions on software Engineering25, 1 (2002), 91–121

    2002

  12. [12]

    Lei Chai and Ming Li. 2022. Pyramid attention for source code summarization.Advances in Neural Information Processing Systems35 (2022), 20421–20433

    2022

  13. [13]

    Pierre Chambon, Baptiste Roziere, Benoit Sagot, and Gabriel Synnaeve. 2025. BigO (Bench)–Can LLMs Generate Code with Controlled Time and Space Complexity?arXiv preprint arXiv:2503.15242(2025)

    Pith/arXiv arXiv 2025

  14. [14]

    Liguo Chen, Qi Guo, Hongrui Jia, Zhengran Zeng, Xin Wang, Yijiang Xu, Jian Wu, Yidong Wang, Qing Gao, Jindong Wang, et al. 2024. A survey on evaluating large language models in code generation tasks.arXiv preprint arXiv:2408.16498 (2024)

    Pith/arXiv arXiv 2024

  15. [15]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  16. [16]

    Alibaba Cloud. 2025. Qwen3-Coder-Plus. https://modelstudio.console.alibabacloud.com/?spm=a3c0i.29328889. 9901980110.3.3f962b6akYcpTR&tab=doc#/doc/?type=model&url=2840914_2&modelId=qwen3-coder-plus

    2025

  17. [17]

    2013.Statistical power analysis for the behavioral sciences

    Jacob Cohen. 2013.Statistical power analysis for the behavioral sciences. routledge

    2013

  18. [18]

    Tristan Coignion, Clément Quinton, and Romain Rouvoy. 2024. A performance study of llm-generated code on leetcode. InProceedings of the 28th international conference on evaluation and assessment in software engineering. 79–89

    2024

  19. [19]

    2022.Introduction to algorithms

    Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. 2022.Introduction to algorithms. MIT press

    2022

  20. [20]

    Domenico Cotroneo, Luigi De Simone, Pietro Liguori, and Roberto Natella. 2020. Fault injection analytics: A novel approach to discover failure modes in cloud-computing systems.IEEE transactions on dependable and secure computing 19, 3 (2020), 1476–1491

    2020

  21. [21]

    Robert W Crandall and Thomas W Hazlett. 2023. Antitrust reform in the digital era: A skeptical perspective.The University of Chicago Business Law Review2, 2 (2023), 1

    2023

  22. [22]

    Giuseppe Crupi, Rosalia Tufano, Alejandro Velasco, Antonio Mastropaolo, Denys Poshyvanyk, and Gabriele Bavota

  23. [23]

    Guilherme B De Padua and Weiyi Shang. 2017. Revisiting exception handling practices with exception flow analysis. In2017 IEEE 17th International working conference on source code analysis and manipulation (SCAM). IEEE, 11–20

    2017

  24. [24]

    Guilherme B de Pádua and Weiyi Shang. 2018. Studying the relationship between exception handling practices and post-release defects. InProceedings of the 15th International Conference on Mining Software Repositories. 564–575

    2018

  25. [25]

    Alexandre Decan, Tom Mens, and Eleni Constantinou. 2018. On the impact of security vulnerabilities in the npm package dependency network. InProceedings of the 15th international conference on mining software repositories. 181–191

    2018

  26. [26]

    Yashar Deldjoo. 2024. Understanding biases in chatgpt-based recommender systems: Provider fairness, temporal stability, and recency.ACM Transactions on Recommender Systems(2024)

    2024

  27. [27]

    Dirk Draheim, John Grundy, John Hosking, Christof Lutteroth, and Gerald Weber. 2006. Realistic load testing of web applications. InConference on Software Maintenance and Reengineering (CSMR’06). IEEE, 11–pp

    2006

  28. [28]

    Thomas D Gauthier. 2001. Detecting trends using Spearman’s rank correlation coefficient.Environmental forensics2, 4 (2001), 359–362. , Vol. 1, No. 1, Article . Publication date: September 2018. Rethinking Technology Stack Selection with AI Coding Proficiency 21

    2001

  29. [29]

    Bruce L Golden, Edward A Wasil, and Patrick T Harker. 1989. The analytic hierarchy process.Applications and Studies, Berlin, Heidelberg2, 1 (1989), 1–273

    1989

  30. [30]

    Google. 2025. Gemini 2.5 Flash. https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash

    2025

  31. [31]

    Tovi Grossman, George Fitzmaurice, and Ramtin Attar. 2009. A survey of software learnability: metrics, methodologies and guidelines. InProceedings of the sigchi conference on human factors in computing systems. 649–658

    2009

  32. [32]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

    Pith/arXiv arXiv 2025

  33. [33]

    Unstoppable?

    Adam Hales. 2025. "Unstoppable?" ChatGPT Surges to 700 million Weekly Users as Rivals Race to Compete. https://www.windowscentral.com/artificial-intelligence/chatgpt-is-set-to-hit-700-million-weekly-users-but-can- its-rivals-catch-up

    2025

  34. [34]

    1977.Elements of Software Science (Operating and programming systems series)

    Maurice H Halstead. 1977.Elements of Software Science (Operating and programming systems series). Elsevier Science Inc

    1977

  35. [35]

    Ahmed E Hassan. 2009. Predicting faults using the complexity of code changes. In2009 IEEE 31st international conference on software engineering. IEEE, 78–88

    2009

  36. [36]

    Wenpin Hou and Zhicheng Ji. 2025. Comparing large language models and human programmers for generating programming code.Advanced Science12, 8 (2025), 2412279

    2025

  37. [37]

    Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M Zhang. 2024. Effibench: Benchmarking the efficiency of automatically generated code.Advances in Neural Information Processing Systems37 (2024), 11506–11544

    2024

  38. [38]

    International Organization for Standardization. 2011. ISO/IEC 25010:2011 Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models. [Online]. Available: https://www.iso.org/standard/35733.html

    2011

  39. [39]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large language models for code generation.arXiv preprint arXiv:2406.00515(2024)

    Pith/arXiv arXiv 2024

  40. [40]

    Siyuan Jin, Ziyuan Li, Bichao Chen, Bing Zhu, and Yong Xia. 2023. Software code quality measurement: Implications from metric distributions. In2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS). IEEE, 488–496

    2023

  41. [41]

    Jan Jöhnk, Malte Weißert, and Katrin Wyrtki. 2021. Ready or not, AI comes: an interview study of organizational AI readiness factors.Business & information systems engineering63, 1 (2021), 5–20

    2021

  42. [42]

    Rick Kazman, Jai Asundi, and Mark Klein. 2001. Quantifying the costs and benefits of architectural decisions. In Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001. IEEE, 297–306

    2001

  43. [43]

    2000.ATAM: Method for architecture evaluation

    Rick Kazman, Mark Klein, and Paul Clements. 2000.ATAM: Method for architecture evaluation. Technical Report

    2000

  44. [44]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners.Advances in neural information processing systems35 (2022), 22199–22213

    2022

  45. [45]

    Jean-Louis Letouzey and Michel Ilkiewicz. 2012. Managing technical debt with the sqale method.IEEE software29, 6 (2012), 44–51

    2012

  46. [46]

    Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, et al. 2025. Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study. InProceedings of the 31st International Conference on Computational Linguistics. 7511–7531

    2025

  47. [47]

    Jia Li, Chongyang Tao, Jia Li, Ge Li, Zhi Jin, Huangzhao Zhang, Zheng Fang, and Fang Liu. 2023. Large language model-aware in-context learning for code generation.ACM Transactions on Software Engineering and Methodology (2023)

    2023

  48. [48]

    R Li, LB Allal, Y Zi, N Muennighoff, D Kocetkov, C Mou, M Marone, C Akiki, J Li, J Chim, et al. 2023. StarCoder: May the Source be With You!Transactions on machine learning research(2023)

    2023

  49. [49]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode.Science378, 6624 (2022), 1092–1097

    2022

  50. [50]

    Amjad Masad. 2025. Replit CEO says anyone can build an app in an afternoon with AI and it’s powering a new wave of solo startups. Business Insider (online). https://www.businessinsider.com/replit-ceo-vibe-coding-solo-startups-2025-8

    2025

  51. [51]

    Thomas J McCabe. 1976. A complexity measure.IEEE Transactions on software Engineering4 (1976), 308–320

    1976

  52. [52]

    2012.A guide to experimental algorithmics

    Catherine C McGeoch. 2012.A guide to experimental algorithmics. Cambridge University Press

    2012

  53. [53]

    Robert K Merton. 1968. The Matthew effect in science: The reward and communication systems of science are considered.Science159, 3810 (1968), 56–63

    1968

  54. [54]

    1993.Bootstrapping: A nonparametric approach to statistical inference

    Christopher Z Mooney, Robert D Duval, and Robert Duvall. 1993.Bootstrapping: A nonparametric approach to statistical inference. Number 95. sage

    1993

  55. [55]

    Nachiappan Nagappan and Thomas Ball. 2005. Use of relative code churn measures to predict system defect density. InProceedings of the 27th international conference on Software engineering. 284–292. , Vol. 1, No. 1, Article . Publication date: September 2018. 22 Zhang and Jiang, et al

    2005

  56. [56]

    Roberto Natella, Domenico Cotroneo, and Henrique S Madeira. 2016. Assessing dependability with software fault injection: A survey.ACM Computing Surveys (CSUR)48, 3 (2016), 1–55

    2016

  57. [57]

    Meghan Neville. 2025. Why Most CTOs Pick the Wrong Technology Stack (And How to Avoid Their Mistakes). https://softjourn.com/insights/choose-the-right-tech-stac

    2025

  58. [58]

    Paul Oman and Jack Hagemeister. 1992. Metrics for assessing a software system’s maintainability. InProceedings Conference on Software Maintenance 1992. IEEE Computer Society, 337–338

    1992

  59. [59]

    OpenAI. 2024. GPT-4o mini: advancing cost-efficient intelligence. https://openai.com/index/gpt-4o-mini-advancing- cost-efficient-intelligence//

    2024

  60. [60]

    OpenAI. 2024. GPT-4o System Card. https://openai.com/index/gpt-4o-system-card/

    2024

  61. [61]

    OpenAI. 2025. Introducing GPT-5 for developers. OpenAI blog. https://openai.com/index/introducing-gpt-5-for- developers/

    2025

  62. [62]

    Stack Overflow. 2025. Developer Survey. https://survey.stackoverflow.co/2025/ai

    2025

  63. [63]

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? assessing the security of github copilot’s code contributions. In2022 IEEE Symposium on Security and Privacy (SP). IEEE, 754–768

    2022

  64. [64]

    Yun Peng, Jun Wan, Yichen Li, and Xiaoxue Ren. 2025. Coffe: A code efficiency benchmark for code generation. Proceedings of the ACM on Software Engineering2, FSE (2025), 242–265

    2025

  65. [65]

    Marco Piccioni, Carlo A Furia, and Bertrand Meyer. 2013. An empirical study of API usability. In2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE, 5–14

    2013

  66. [66]

    Daryl Posnett, Abram Hindle, and Premkumar Devanbu. 2011. A simpler model of software readability. InProceedings of the 8th working conference on mining software repositories. 73–82

    2011

  67. [67]

    PromptHub. 2025. The Few Shot Prompting Guide. https://www.prompthub.us/blog/the-few-shot-prompting- guide#few-shot-prompting-examples

    2025

  68. [68]

    Yuhao Qing, Boyu Zhu, Mingzhe Du, Zhijiang Guo, Terry Yue Zhuo, Qianru Zhang, Jie M Zhang, Heming Cui, Siu-Ming Yiu, Dong Huang, et al. 2025. EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code.arXiv preprint arXiv:2505.13004(2025)

    Pith/arXiv arXiv 2025

  69. [69]

    Irum Rauf, Elena Troubitsyna, and Ivan Porres. 2019. A systematic mapping study of API usability evaluation methods. Computer Science Review33 (2019), 49–68

    2019

  70. [70]

    Francisco Ribeiro. 2023. Large language models for automated program repair. InCompanion Proceedings of the 2023 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity. 7–9

    2023

  71. [71]

    Ruksit Rojpaisarnkit, Gregorio Robles, Raula Gaikovina Kula, Dong Wang, Chaiyong Ragkhitwetsagul, Jesus M Gonzalez-Barahona, and Kenichi Matsumoto. 2024. Towards Identifying Code Proficiency Through the Analysis of Python Textbooks. In2024 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 288–299

    2024

  72. [72]

    Philip Sedgwick. 2014. Spearman’s rank correlation coefficient.Bmj349 (2014)

    2014

  73. [73]

    Bles Software. 2025. Startup Tech Stack 2025: Choosing the Right Technologies for Your Product. https://www.bles- software.com/blog/startup-tech-stack-2025-choosing-the-right-technologies-for-your-product

    2025

  74. [74]

    Stratoflow. 2023. SDLC Guide: How to Conduct Software Design Phase. Technical guidance. https://stratoflow.com/sdlc- design-phase/

    2023

  75. [75]

    Kunsheng Tang, Wenbo Zhou, Jie Zhang, Aishan Liu, Gelei Deng, Shuai Li, Peigui Qi, Weiming Zhang, Tianwei Zhang, and Nenghai Yu. 2024. GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. 1196–1210

    2024

  76. [76]

    Ali N Tehrani, Subhasis Ray, Sanjit K Roy, Richard L Gruner, and Francesco P Appio. 2024. Decoding AI readiness: An in-depth analysis of key dimensions in multinational corporations.Technovation131 (2024), 102948

    2024

  77. [77]

    TESTINGMIND. 2025. ChatGPT for Automated Testing: From Conversation to Code. https://www.testingmind.com/ chatgpt-for-automated-testing-from-conversation-to-code/

    2025

  78. [78]

    Weixi Tong and Tianyi Zhang. 2024. CodeJudge: Evaluating Code Generation with Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 20032–20051

    2024

  79. [79]

    1977.Exploratory data analysis

    John Wilder Tukey et al. 1977.Exploratory data analysis. Vol. 2. Springer

    1977

  80. [80]

    Lukas Twist, Jie M Zhang, Mark Harman, Don Syme, Joost Noppen, and Detlef Nauck. 2025. LLMs Love Python: A Study of LLMs’ Bias for Programming Languages and Libraries.arXiv preprint arXiv:2503.17181(2025)

    Pith/arXiv arXiv 2025

Showing first 80 references.