"My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistants

David Lo; Jianming Chang; Jieke Shi; Yue Liu; Yunbo Lyu; Zhou Yang

REVIEW 3 major objections 3 minor 75 references

"My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistants

T0 review · 3 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that developers' first-hand marketplace reviews show context-awareness, customizability, and resource efficiency are the major determinants of satisfaction with AI coding assistants.

desk verdict A plausible large-scale review-mining study that deserves peer review, but selection bias toward popular assistants and an unverifiable full text keep me from endorsing the findings yet. read the letter →

arxiv 2508.12285 v1 pith:SMASDHMU submitted 2025-08-17 cs.SE cs.AIcs.HC

Yunbo Lyu , Zhou Yang , Jieke Shi , Jianming Chang , Yue Liu , David Lo This is my paper · ORCID

classification cs.SEcs.AIcs.HC

keywords AIcodingassistantsuserreviewsVSCodeMarketplacetaxonomyofneedscontextawarenesscustomizabilityresourceefficiencydevelopersatisfaction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish what developers truly value and criticize in AI coding assistants by reading their own words in marketplace reviews rather than by running controlled experiments. It identifies 1,085 AI coding assistant extensions on the VS Code Marketplace, notes that over 90% appeared in the past two years, and manually analyzes reviews sampled from 32 assistants with enough installations and reviews. The result is a taxonomy of user needs, with context-awareness, customizability, and resource efficiency emerging as major determinants of satisfaction. A sympathetic reader would care because the data is drawn from real day-to-day work contexts, so the findings describe actual user priorities rather than simulated ones.

What carries the argument

The paper's central object is the taxonomy of user concerns, built by sampling reviews from 32 AI coding assistants with sufficient installations and reviews, then manually annotating each review for attitude toward specific features, concerns, and overall performance. This taxonomy, along with the attitude labels, carries the argument: it converts unstructured marketplace reviews into a structured map of satisfaction and dissatisfaction.

What would settle it

A fresh team of annotators re-codes a random sample of the same reviews without seeing the paper's taxonomy; if the categories and attitude labels do not reproduce with high agreement, the taxonomy is not a stable result. A second check: if a survey of developers using these assistants shows no correlation between context-awareness, customizability, or resource efficiency and their satisfaction ratings, the central claim would lose support.

Watch

Extended reading notes

Core claim

The central discovery is that developers' first-hand reviews of AI coding assistants reveal a taxonomy of needs in which context-awareness, customizability, and resource efficiency are major determinants of satisfaction. Across the sampled reviews, users ask for suggestions that understand their project and recent edits, for tools they can shape to their workflow, and for assistants that do not cost too much in memory, CPU, or battery. The paper also documents a surge: over 90% of the 1,085 identified assistants were released within the past two years, situating these needs in a rapidly expanding ecosystem.

Load-bearing premise

The 32 assistants with enough installations and reviews, and the manual annotation of review sentiment, are assumed to faithfully represent the full population of AI coding assistants and their users.

Editorial extensions

If this is right

Designers who focus only on raw suggestion quality will miss the factors that most affect user satisfaction.
Context-awareness should be treated as a core requirement, meaning assistants need access to project structure, open files, and recent changes.
Customizability and resource efficiency belong in the same priority class as correctness and speed.
The five practical implications from the reviews give assistant builders a concrete user-needs checklist.
The taxonomy can serve as a baseline for comparing future assistants against what users actually ask for.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If context-awareness is as central as the reviews suggest, then benchmarks that evaluate assistants on isolated code snippets may overrate tools that work well in toy tasks but poorly in real projects; a testable extension is to score assistants on context-rich tasks and compare those scores with marketplace review sentiment.
Resource-efficiency complaints are likely to grow as assistants move into enterprise and on-device settings; a natural follow-up is to analyze whether free versus paid tiers change which concerns dominate.
The taxonomy could be operationalized into an automated review-analysis pipeline for marketplace maintainers, something the paper itself does not build.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the abstract describes a plausible and potentially useful review-mining study of AI coding assistants, but the version I have has a garbled full text (it even carries a physics paper's header), so I can only judge the abstract. On that basis, the study is worth taking seriously, but there's a real sampling concern that the authors don't address.

What's new: the scale. Identifying 1,085 AI coding assistants on the VS Code Marketplace and noting the recent surge is a useful empirical fact. The design — mining real user reviews instead of running a controlled lab study — is a legitimate complement to existing user studies of Copilot and similar tools. If the taxonomy of concerns (context-awareness, customizability, resource efficiency) is reliable, it gives tool builders a concrete list of priorities.

What it does well: the manual attitude annotation is a step beyond simple sentiment mining, and the five practical implications show the authors are thinking about use. The abstract is honest about the method and doesn't overclaim — it says 'sampled from 32 assistants,' not 'all assistants.'

Soft spots: the selection threshold is unstated. 'Sufficient installations and reviews' could mean anything, and since 90% of the 1,085 assistants were released in the past two years, most of them will be long-tail with few reviews. If the 32 are the popular, mature tools — which is likely — the taxonomy reflects the concerns of users of established assistants, not the long tail of new entrants. That's not fatal, but it should be flagged as a scope limitation. Second, I can't verify the annotation reliability, codebook, or exclusion criteria because the full text is unusable in the version I have. This is a practical barrier to review, not necessarily a flaw in the work itself — the authors may well have done everything right.

Recommendation: this deserves peer review. The question is timely, the approach is sound in principle, and a serious reviewer should ask for the selection threshold, the distribution of reviews across assistants, and inter-rater reliability. I'd want to see a clean version before citing it.

Referee Report

3 major / 3 minor

Summary. The paper analyzes user reviews of AI coding assistants from the Visual Studio Code Marketplace to construct a taxonomy of user concerns. The authors identify 1,085 AI coding assistant extensions, observe that over 90% were released in the past two years, and manually analyze reviews sampled from 32 assistants that have "sufficient installations and reviews." They manually annotate review attitudes toward specific features and concerns, and from these findings propose five practical implications. The central claim is that developers value context-awareness, customizability, and resource efficiency in AI coding assistants.

Significance. If the empirical findings hold, the paper offers a useful complement to controlled and simulated studies by grounding user needs in authentic, first-hand marketplace reviews. The data source is appropriate, and the taxonomy is derived from external user reviews rather than the authors' prior results, so there is no evident circularity. The reported surge in the release of AI coding assistants is an interesting and credible observation. However, the present manuscript does not yet provide sufficient methodological detail to verify the sampling and annotation claims, and the supplied full text is not in a reviewable state.

major comments (3)

[Abstract / Sampling methodology] The paper does not specify the threshold for "sufficient installations and reviews" used to select the 32 assistants, nor does it report how many of the 1,085 identified assistants met this criterion. This is load-bearing because the taxonomy's generality depends on the sampled assistants representing the population of AI coding assistants; given that over 90% of the assistants were released in the past two years and likely have few reviews, a threshold that selects popular, mature tools could introduce a long-tail selection bias. Please provide the exact inclusion criteria, the number of qualifying assistants, and a comparison of characteristics (e.g., age, install counts, ratings) between included and excluded assistants.
[Manual attitude annotation (abstract and full text)] The abstract states that the authors "manually annotate each review's attitude," but the manuscript as supplied provides no codebook, annotation guidelines, number of annotators, or inter-rater reliability measures such as Cohen's or Fleiss' kappa. Without this information, the attitude annotations cannot be distinguished from anecdotal reading, and the claim of "nuanced insights into user satisfaction and dissatisfaction" is not yet reproducible. Please report the annotation scheme, the annotator setup, and agreement statistics per code.
[Full text / manuscript integrity] The full text of the submitted manuscript is garbled and includes the header "arXiv:2508.12289v4 [physics.flu-dyn]", which belongs to a different paper. As a result, all claims that depend on the detailed empirical sections, including the taxonomy definitions, example review quotes, the attitude-by-category tables, and the five practical implications, cannot be audited. The authors must resupply a clean, readable manuscript so that the methodology and results can be verified.

minor comments (3)

[Abstract] The abstract reports that the 1,085 identified assistants "only account for 1.64% of all extensions," but the denominator (total number of VS Code extensions) is not stated; please provide it explicitly for context.
[Abstract / terminology] The phrase "32 popular assistants" in the abstract does not exactly match the selection criterion "sufficient installations and reviews" described later; please align the wording to avoid ambiguity about how popularity is operationalized.
[Scope] The paper generalizes to "developers" but the data are exclusively from the VS Code Marketplace; please state this limitation explicitly and consider tempering the language to "VS Code users" where appropriate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the taxonomy is an inductive empirical result derived from external user reviews, with no fitted parameters, self-cited load-bearing premises, or definitions that presuppose the conclusion.

full rationale

The paper's central claim is that first-hand user reviews of AI coding assistants, sampled from the VS Code Marketplace and manually annotated, yield a taxonomy of user concerns and that context-awareness, customizability, and resource efficiency emerge as major determinants of satisfaction. This is an inductive empirical pipeline: data collection, sampling, manual annotation, categorization. The abstract contains no equation, no fitted parameter that is later renamed as a prediction, and no definition in which the taxonomy categories are constructed from the conclusions. The paper does not invoke a self-citation or an author-imported uniqueness theorem to justify its choice of categories, and the findings are presented as interpretations of external review text rather than as derivations from prior results. The only concerns visible in the abstract are sampling representativeness (assistants with 'sufficient installations and reviews' may skew toward popular tools) and the audibility of the manual annotation procedure; these are validity, generalizability, and completeness issues, not circularity. The supplied full text is garbled and even mislabeled with a different arXiv identifier, so deeper method-level auditing is impossible from the available material, but unreadable text is not evidence of circular reasoning. Accordingly, no specific circular step can be quoted and exhibited, and the honest finding is no significant circularity with score 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

Abstract-only review; no free parameters or invented entities; the axioms reflect assumptions inherent in the study design.

assumptions (2)

domain assumption User reviews are a valid proxy for developers' authentic perceptions and experiences.
The entire taxonomy is built on review text; this premise is inherent to the abstract's method.
domain assumption The selected 32 assistants with sufficient installations and reviews are representative of the broader ecosystem.
Claims about user needs generalize from this sample, per the abstract.

how reviews work

0 comments

Cite this review

Pith. "Pith review of "My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistants." pith.science (2026). https://pith.science/paper/SMASDHMU

@misc{pith2026250812285,
  author       = {Pith},
  title        = {Pith review of: "My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistants},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SMASDHMU}},
  note         = {Machine review of arXiv:2508.12285}
}

read the original abstract

This paper aims to explore fundamental questions in the era when AI coding assistants like GitHub Copilot are widely adopted: what do developers truly value and criticize in AI coding assistants, and what does this reveal about their needs and expectations in real-world software development? Unlike previous studies that conduct observational research in controlled and simulated environments, we analyze extensive, first-hand user reviews of AI coding assistants, which capture developers' authentic perspectives and experiences drawn directly from their actual day-to-day work contexts. We identify 1,085 AI coding assistants from the Visual Studio Code Marketplace. Although they only account for 1.64% of all extensions, we observe a surge in these assistants: over 90% of them are released within the past two years. We then manually analyze the user reviews sampled from 32 AI coding assistants that have sufficient installations and reviews to construct a comprehensive taxonomy of user concerns and feedback about these assistants. We manually annotate each review's attitude when mentioning certain aspects of coding assistants, yielding nuanced insights into user satisfaction and dissatisfaction regarding specific features, concerns, and overall tool performance. Built on top of the findings-including how users demand not just intelligent suggestions but also context-aware, customizable, and resource-efficient interactions-we propose five practical implications and suggestions to guide the enhancement of AI coding assistants that satisfy user needs.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 51 canonical work pages

[1]

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page
[2]

Mozannar, G

H. Mozannar, G. Bansal, A. Fourney, and E. Horvitz, ``Reading between the lines: Modeling user behavior and costs in ai-assisted programming,'' ser. CHI '24. 1em plus 0.5em minus 0.4em New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3613904.3641936

arXiv 2024
[3]

Sergeyuk, Y

A. Sergeyuk, Y. Golubev, T. Bryksin, and I. Ahmed, ``Using ai-based coding assistants in practice: State of affairs, perceptions, and ways forward,'' Information and Software Technology, vol. 178, 2025

work page 2025
[4]

J. Chen, X. Hu, Z. Li, C. Gao, X. Xia, and D. Lo, ``Code search is all you need? improving code suggestions with code search,'' in Proceedings of the IEEE/ACM 46th international conference on software engineering, 2024, pp. 1--13

work page 2024
[5]

B. Yang, H. Tian, J. Ren, H. Zhang, J. Klein, T. Bissyande, C. Le Goues, and S. Jin, ``Morepair: Teaching llms to repair code via multi-objective fine-tuning,'' ACM Trans. Softw. Eng. Methodol., May 2025, just Accepted. [Online]. Available: https://doi.org/10.1145/3735129

doi:10.1145/3735129 2025
[6]

J. T. Liang, C. Yang, and B. A. Myers, ``A large-scale survey on the usability of ai programming assistants: Successes and challenges,'' in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1--13

work page 2024
[7]

Wilkinson, ``Github copilot drives revenue growth amid subscriber base expansion,'' 2024, accessed: Feb 28, 2025

L. Wilkinson, ``Github copilot drives revenue growth amid subscriber base expansion,'' 2024, accessed: Feb 28, 2025. [Online]. Available: https://www.ciodive.com/news/github-copilot-subscriber-count-revenue-growth/706201/

work page 2024
[8]

A. M. Mcnutt, C. Wang, R. A. Deline, and S. M. Drucker, ``On the design of ai-powered code assistants for notebooks,'' in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, ser. CHI '23. 1em plus 0.5em minus 0.4em New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: https://doi.org/10.1145/3544548.3580940

arXiv 2023

Show all 75 references

[9]

C. Liu, X. Zhang, H. Zhang, Z. Wan, Z. Huang, and M. Yan, ``An empirical study of code search in intelligent coding assistant: Perceptions, expectations, and directions,'' in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineer...

2024
[10]

it’s weird that it knows what i want

J. Prather, B. N. Reeves, P. Denny, B. A. Becker, J. Leinonen, A. Luxton-Reilly, G. Powell, J. Finnie-Ansley, and E. A. Santos, ``“it’s weird that it knows what i want”: Usability and interactions with copilot for novice programmers,'' ACM Trans. Comput.-Hum. Interact., vol. 3...

2023 doi
[11]

C. Wang, J. Hu, C. Gao, Y. Jin, T. Xie, H. Huang, Z. Lei, and Y. Deng, ``How practitioners expect code completion?'' in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2023. 1em ...

2023
[12]

Vaithilingam, T

P. Vaithilingam, T. Zhang, and E. L. Glassman, ``Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,'' in Chi conference on human factors in computing systems extended abstracts, 2022, pp. 1--7

2022
[13]

Barke, M

S. Barke, M. B. James, and N. Polikarpova, ``Grounded copilot: How programmers interact with code-generating models,'' Proc. ACM Program. Lang., vol. 7, no. OOPSLA1, Apr. 2023. [Online]. Available: https://doi.org/10.1145/3586030

2023 doi
[14]

M. C. Davis, E. Aghayi, T. D. Latoza, X. Wang, B. A. Myers, and J. Sunshine, ``What's (not) working in programmer user studies?'' ACM Transactions on Software Engineering and Methodology, vol. 32, no. 5, pp. 1--32, 2023

2023
[15]

Sergeyuk, I

A. Sergeyuk, I. Zakharov, E. Koshchenko, and M. Izadi, ``Human-ai experience in integrated development environments: A systematic literature review,'' arXiv preprint arXiv:2503.06195, 2025

2025
[16]

Pagano and W

D. Pagano and W. Maalej, ``User feedback in the appstore: An empirical study,'' in 2013 21st IEEE International Requirements Engineering Conference (RE), 2013, pp. 125--134

2013
[17]

Martin, F

W. Martin, F. Sarro, Y. Jia, Y. Zhang, and M. Harman, ``A survey of app store analysis for software engineering,'' IEEE transactions on software engineering, vol. 43, no. 9, pp. 817--847, 2016

2016
[18]

A. A. Al-Subaihin, F. Sarro, S. Black, L. Capra, and M. Harman, ``App store effects on software engineering practices,'' IEEE Transactions on Software Engineering, vol. 47, no. 2, pp. 300--319, 2019

2019
[19]

Zhang, P

B. Zhang, P. Liang, X. Zhou, A. Ahmad, and M. Waseem, ``Demystifying practices, challenges and expected features of using github copilot,'' International Journal of Software Engineering and Knowledge Engineering, vol. 33, no. 11n12, pp. 1653--1672, 2023

2023
[20]

X. Zhou, P. Liang, B. Zhang, Z. Li, A. Ahmad, M. Shahin, and M. Waseem, ``Exploring the problems, their causes and solutions of ai pair programming: A study on github and stack overflow,'' Journal of Systems and Software, vol. 219, p. 112204, 2025

2025
[21]

J. Liao, G. Yang, D. Kavaler, V. Filkov, and P. Devanbu, ``Status, identity, and language: A study of issue discussions in github,'' PloS one, vol. 14, no. 6, p. e0215059, 2019

2019
[22]

D. Ford, J. Smith, P. J. Guo, and C. Parnin, ``Paradise unplugged: Identifying barriers for female participation on stack overflow,'' in Proceedings of the 2016 24th ACM SIGSOFT International symposium on foundations of software engineering, 2016, pp. 846--857

2016
[23]

Stack Overflow , `` Stack Overflow Developer Survey 2024 ,'' Online, 2024, available: https://survey.stackoverflow.co/2024/

2024
[24]

Y. Liu, C. Tantithamthavorn, and L. Li, `` Protect Your Secrets: Understanding and Measuring Data Exposure in VSCode Extensions ,'' in 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 1em plus 0.5em minus 0.4em Los Alamitos, CA, USA...

2025
[25]

[Online]

Microsoft , `` Visual Studio Marketplace ,'' 2025, accessed: February 21, 2025. [Online]. Available: https://marketplace.visualstudio.com

2025
[26]

D a browski, E

J. D a browski, E. Letier, A. Perini, and A. Susi, ``Analysing app reviews for software engineering: a systematic literature review,'' Empirical Softw. Engg., vol. 27, no. 2, Mar. 2022. [Online]. Available: https://doi.org/10.1007/s10664-021-10065-7

2022 doi
[27]

Humbatova, G

N. Humbatova, G. Jahangirova, G. Bavota, V. Riccio, A. Stocco, and P. Tonella, ``Taxonomy of real faults in deep learning systems,'' in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ser. ICSE '20. 1em plus 0.5em minus 0.4em New York, NY, US...

2020
[28]

L. Y. Conrad and V. M. Tucker, ``Making it tangible: hybrid card sorting within qualitative interviews,'' Journal of Documentation, vol. 75, no. 2, pp. 397--416, Jan 2019. [Online]. Available: https://doi.org/10.1108/JD-06-2018-0091

2019 doi
[29]

[Online]

GitHub , ``Github copilot: The agent awakens,'' 2025, accessed: February 16 2025. [Online]. Available: https://github.blog/news-insights/product-news/github-copilot-the-agent-awakens/

2025
[30]

Z. Yang, Z. Sun, T. Z. Yue, P. Devanbu, and D. Lo, ``Robustness, security, privacy, explainability, efficiency, and usability of large language models for code,'' arXiv preprint arXiv:2403.07506, 2024

2024 arXiv
[31]

GitHub , `` Survey Reveals AI's Impact on the Developer Experience ,'' https://github.blog/news-insights/research/survey-reveals-ais-impact-on-the-developer-experience/, 2024, accessed: May 25, 2025

2024
[32]

Ziegler, E

A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian, ``Productivity assessment of neural code completion,'' in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, ser. MAPS 2022. 1em plus 0.5...

2022
[33]

A. Shah, A. Chernova, E. Tomson, L. Porter, W. G. Griswold, and A. G. Soosai Raj, ``Students' use of github copilot for working with large code bases,'' in Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1, ser. SIGCSETS 2025. 1em plus 0.5em mi...

2025
[34]

M. V. Phong, T. T. Nguyen, H. V. Pham, and T. T. Nguyen, ``Mining user opinions in mobile app reviews: A keyword-based approach (t),'' in 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2015, pp. 749--759

2015
[35]

N. Chen, J. Lin, S. C. H. Hoi, X. Xiao, and B. Zhang, ``Ar-miner: mining informative reviews for developers from mobile app marketplace,'' ser. ICSE 2014. 1em plus 0.5em minus 0.4em New York, NY, USA: Association for Computing Machinery, 2014, p. 767–778. [Online]. Available: ...

2014
[36]

Guzman and W

E. Guzman and W. Maalej, ``How do users like this feature? a fine grained sentiment analysis of app reviews,'' in 2014 IEEE 22nd International Requirements Engineering Conference (RE), 2014, pp. 153--162

2014
[37]

Maalej, Z

W. Maalej, Z. Kurtanovi \' c , H. Nabil, and C. Stanik, ``On the automatic classification of app reviews,'' Requirements Engineering, vol. 21, no. 3, pp. 311--331, Sep 2016. [Online]. Available: https://doi.org/10.1007/s00766-016-0251-9

2016 doi
[38]

Jha and A

N. Jha and A. Mahmoud, ``Mining non-functional requirements from app store reviews,'' Empirical Software Engineering, vol. 24, no. 6, pp. 3659--3695, Dec 2019. [Online]. Available: https://doi.org/10.1007/s10664-019-09716-7

2019 doi
[39]

GitHub , `` GitHub Copilot ,'' https://marketplace.visualstudio.com/items?itemName=GitHub.copilot, 2025, retrieved Feb 22, 2025

2025
[40]

Visual Studio | Marketplace , `` VS Code Marketplace: AI extensions search results ,'' https://marketplace.visualstudio.com/search?term=AI&target=VSCode&category=All\ 2025, retrieved Feb 21, 2025

2025
[41]

X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, ``Large language models for software engineering: A systematic literature review,'' ACM Trans. Softw. Eng. Methodol., vol. 33, no. 8, Dec. 2024

2024
[42]

[Online]

SurveyMonkey , ``Sample size calculator,'' 2025, accessed: Feb 28, 2025. [Online]. Available: https://www.surveymonkey.com/mp/sample-size-calculator/

2025
[43]

H. Hata, C. Treude, R. G. Kula, and T. Ishio, ``9.6 million links in source code comments: Purpose, evolution, and decay,'' in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 1em plus 0.5em minus 0.4em IEEE, 2019, pp. 1211--1221

2019
[44]

Y. Lyu, H. J. Kang, R. Widyasari, J. Lawall, and D. Lo, ``Evaluating szz implementations: An empirical study on the linux kernel,'' IEEE Trans. Softw. Eng., vol. 50, no. 9, p. 2219–2239, Sep. 2024. [Online]. Available: https://doi.org/10.1109/TSE.2024.3406718

2024
[45]

J. Wang, L. Li, and A. Zeller, ``Restoring execution environments of jupyter notebooks,'' in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 1em plus 0.5em minus 0.4em IEEE, 2021, pp. 1622--1633

2021
[46]

M. L. McHugh, ``Interrater reliability: the kappa statistic,'' Biochemia medica, vol. 22, no. 3, pp. 276--282, 2012

2012
[47]

J. R. Landis and G. G. Koch, ``The measurement of observer agreement for categorical data,'' biometrics, pp. 159--174, 1977

1977
[48]

P. E. McKnight and J. Najab, ``Mann-whitney u test,'' The Corsini encyclopedia of psychology, pp. 1--1, 2010

2010
[49]

Universita de Cagliari, 1912

``Variability and mutability, contribution to the study of statistical distributions and relations,'' Studi Economico-Giuridici della R. Universita de Cagliari, 1912

1912
[50]

Kurtanovi \'c and W

Z. Kurtanovi \'c and W. Maalej, ``Mining user rationale from software reviews,'' in 2017 IEEE 25th international requirements engineering conference (RE). 1em plus 0.5em minus 0.4em IEEE, 2017, pp. 61--70

2017
[51]

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., ``Evaluating large language models trained on code,'' arXiv preprint arXiv:2107.03374, 2021

2021 arXiv
[52]

S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer, ``The impact of ai on developer productivity: Evidence from github copilot,'' arXiv preprint arXiv:2302.06590, 2023

2023 arXiv
[53]

Martinovi \' c and R

B. Martinovi \' c and R. Rozi \' c , ``Perceived impact of ai-based tooling on software development code quality,'' SN Computer Science, vol. 6, no. 1, p. 63, Jan 2025. [Online]. Available: https://doi.org/10.1007/s42979-024-03608-4

2025 doi
[54]

S. Imai, ``Is github copilot a substitute for human pair-programming? an empirical study,'' in Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, ser. ICSE '22. 1em plus 0.5em minus 0.4em New York, NY, USA: Association for...

2022
[55]

Moradi Dakhel , V

A. Moradi Dakhel , V. Majdinasab, A. Nikanjam, F. Khomh, M. C. Desmarais, and Z. M. J. Jiang, ``Github copilot ai pair programmer: Asset or liability?'' Journal of Systems and Software, vol. 203, p. 111734, 2023

2023
[56]

Z. Yang, B. Xu, J. M. Zhang, H. J. Kang, J. Shi, J. He, and D. Lo, ``Stealthy backdoor attack for code models,'' IEEE Transactions on Software Engineering, vol. 50, no. 4, pp. 721--741, 2024

2024
[57]

Z. Yang, Z. Zhao, C. Wang, J. Shi, D. Kim, D. Han, and D. Lo, ``Unveiling memorization in code models,'' ser. ICSE '24. 1em plus 0.5em minus 0.4em New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3597503.3639074

2024
[58]

D. Nam, A. Macvean, V. Hellendoorn, B. Vasilescu, and B. Myers, ``Using an llm to help with code understanding,'' in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE '24. 1em plus 0.5em minus 0.4em New York, NY, USA: ACM, 2024

2024
[59]

Z. Sun, X. Du, F. Song, S. Wang, and L. Li, ``When neural code completion models size up the situation: Attaining cheaper and faster completion through dynamic model inference,'' in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE '2...

2024
[60]

T. Ge, J. Hu, L. Wang, X. Wang, S.-Q. Chen, and F. Wei, ``In-context autoencoder for context compression in a large language model,'' arXiv preprint arXiv:2307.06945, 2023

2023 arXiv
[61]

Pinto, C

G. Pinto, C. De Souza, J. B. Neto, A. Souza, T. Gotto, and E. Monteiro, ``Lessons from building stackspot ai: A contextualized ai coding assistant,'' in Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice, ser. ICSE-SEIP '...

2024
[62]

W. Luo, J. W. Keung, B. Yang, J. Klein, T. F. Bissyande, H. Tian, and B. Le, ``Unlocking llm repair capabilities in low-resource programming languages through cross-language translation and multi-agent refinement,'' arXiv preprint arXiv:2503.22512, 2025

2025
[63]

W. Luo, J. Keung, B. Yang, H. Ye, C. Le Goues, T. F. Bissyand\' e , H. Tian, and X. B. D. Le, ``When fine-tuning llms meets data privacy: An empirical study of federated learning in llm-based program repair,'' ACM Trans. Softw. Eng. Methodol., May 2025, just Accepted. [Online]...

2025 doi
[64]

Mozannar, G

H. Mozannar, G. Bansal, A. Fourney, and E. Horvitz, ``When to show a suggestion? integrating human feedback in ai-assisted programming,'' ser. AAAI'24/IAAI'24/EAAI'24. 1em plus 0.5em minus 0.4em AAAI Press, 2024. [Online]. Available: https://doi.org/10.1609/aaai.v38i9.28878

2024 doi
[65]

Z. Sun, X. Du, F. Song, S. Wang, M. Ni, and L. Li, ``Don't complete it! preventing unhelpful code completion for productive and sustainable neural code completion systems,'' in 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Com...

2023
[66]

J. Shi, Z. Yang, B. Xu, H. J. Kang, and D. Lo, ``Compressing pre-trained models of code into 3 mb,'' in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE '22. 1em plus 0.5em minus 0.4em New York, NY, USA: Association for Comp...

2023
[67]

J. Shi, Z. Yang, H. J. Kang, B. Xu, J. He, and D. Lo, ``Greening large language models of code,'' in Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Society, 2024, pp. 142--153

2024
[68]

J. Shi, Z. Yang, and D. Lo, ``Efficient and green large language models for software engineering: Vision and the road ahead,'' ACM Transactions on Software Engineering and Methodology, 2024

2024
[69]

Mittal, W

V. Mittal, W. T. RossJr., and P. M. Baldasare, ``The asymmetric impact of negative and positive attribute-level performance on overall satisfaction and repurchase intentions,'' Journal of Marketing, vol. 62, no. 1, pp. 33--47, 1998

1998
[70]

B. B. Holloway and S. E. Beatty, ``Satisfiers and dissatisfiers in the online environment: A critical incident assessment,'' Journal of Service Research, vol. 10, no. 4, pp. 347--364, 2008

2008
[71]

Z. Yang, J. Shi, J. He, and D. Lo, ``Natural attack for pre-trained models of code,'' in Proceedings of the 44th International Conference on Software Engineering, ser. ICSE '22. 1em plus 0.5em minus 0.4em New York, NY, USA: Association for Computing Machinery, 2022, p. 1482–14...

2022
[72]

Y. Lyu, T. Le-Cong, H. J. Kang, R. Widyasari, Z. Zhao, X.-B. D. Le, M. Li, and D. Lo, ``Chronos: Time-aware zero-shot identification of libraries from vulnerability reports,'' in Proceedings of the 45th International Conference on Software Engineering, ser. ICSE '23. 1em plus ...

2023
[73]

Y. Lyu, Z. Yang, Y. Niu, J. Jiang, and D. Lo, ``Do existing testing tools really uncover gender bias in text-to-image models?'' arXiv preprint arXiv:2501.15775, 2025

2025 arXiv
[74]

Choudhuri, B

R. Choudhuri, B. Trinkenreich, R. Pandita, E. Kalliamvakou, I. Steinmacher, M. Gerosa, C. Sanchez, and A. Sarma, ``What guides our choices? modeling developers' trust and behavioral intentions towards genai,'' arXiv preprint arXiv:2409.04099, 2024

2024 arXiv
[75]

A. A. Al-Subaihin, F. Sarro, S. Black, L. Capra, and M. Harman, ``App store effects on software engineering practices,'' IEEE Transactions on Software Engineering, vol. 47, no. 2, pp. 300--319, 2021

2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page

[2] [2]

Mozannar, G

H. Mozannar, G. Bansal, A. Fourney, and E. Horvitz, ``Reading between the lines: Modeling user behavior and costs in ai-assisted programming,'' ser. CHI '24. 1em plus 0.5em minus 0.4em New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3613904.3641936

arXiv 2024

[3] [3]

Sergeyuk, Y

A. Sergeyuk, Y. Golubev, T. Bryksin, and I. Ahmed, ``Using ai-based coding assistants in practice: State of affairs, perceptions, and ways forward,'' Information and Software Technology, vol. 178, 2025

work page 2025

[4] [4]

J. Chen, X. Hu, Z. Li, C. Gao, X. Xia, and D. Lo, ``Code search is all you need? improving code suggestions with code search,'' in Proceedings of the IEEE/ACM 46th international conference on software engineering, 2024, pp. 1--13

work page 2024

[5] [5]

B. Yang, H. Tian, J. Ren, H. Zhang, J. Klein, T. Bissyande, C. Le Goues, and S. Jin, ``Morepair: Teaching llms to repair code via multi-objective fine-tuning,'' ACM Trans. Softw. Eng. Methodol., May 2025, just Accepted. [Online]. Available: https://doi.org/10.1145/3735129

doi:10.1145/3735129 2025

[6] [6]

J. T. Liang, C. Yang, and B. A. Myers, ``A large-scale survey on the usability of ai programming assistants: Successes and challenges,'' in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1--13

work page 2024

[7] [7]

Wilkinson, ``Github copilot drives revenue growth amid subscriber base expansion,'' 2024, accessed: Feb 28, 2025

L. Wilkinson, ``Github copilot drives revenue growth amid subscriber base expansion,'' 2024, accessed: Feb 28, 2025. [Online]. Available: https://www.ciodive.com/news/github-copilot-subscriber-count-revenue-growth/706201/

work page 2024

[8] [8]

A. M. Mcnutt, C. Wang, R. A. Deline, and S. M. Drucker, ``On the design of ai-powered code assistants for notebooks,'' in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, ser. CHI '23. 1em plus 0.5em minus 0.4em New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: https://doi.org/10.1145/3544548.3580940

arXiv 2023