Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

Analyzing 82,845 real developer–LLM conversations, this paper establishes that generated code is systematically defect-prone in language-specific ways, with undefined variables plaguing most Python and JavaScript snippets, and that some def

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 17:52 UTC pith:DU5RBK23

load-bearing objection Valuable new dataset and honest descriptive work, but the abstract mislabels a headline number and the linting-from-isolated-snippets threat deserves a serious caveat. the 3 major comments →

arxiv 2509.10402 v1 pith:DU5RBK23 submitted 2025-09-12 cs.SE

Developer-LLM Conversations: An Empirical Study of Interactions and Generated Code Quality

Suzhen Zhong , Ying Zou , Bram Adams This is my paper
classification cs.SE
keywords developer-LLM conversationcode generationcode qualitystatic analysismulti-turn dialogueprompt design gapsCodeChat datasetlanguage-specific defects
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish what real developer–LLM conversations look like and whether the code they produce is good enough to use. Drawing on 82,845 real ChatGPT conversations with 368,506 code snippets, it shows that successes and failures are language-specific: most Python and JavaScript snippets contain undefined variables, most Java snippets lack required comments, and many C++ and C# snippets miss includes or namespaces. It further shows that multi-turn conversation is not a cure-all—Python undefined variables grow over turns while Java documentation quality improves—and that the most reliable repair is a prompt that names the error and asks for a fix. If the findings hold, they argue for routine static checking of LLM output and for context-aware tools that remember prior turns.

Core claim

The paper's central discovery is that real-world LLM-generated code from ChatGPT-style conversations is not just imperfect but systematically defective in language-specific ways: at first turn, 83.4% of Python snippets violate naming conventions, 30.8% contain undefined variables; 75.3% of JavaScript snippets have undefined variables; 75.9% of Java snippets miss required comments; 41.1% of C++ snippets omit #include directives; and 49.2% of C# snippets reference unresolved namespaces. It also finds that these issues evolve unevenly through dialogue: Python undefined-variable proportion rises from 23.5% to 32.8% over five turns, import errors in Python fall, Java's comment violations fall fro

What carries the argument

The explanatory machinery is CodeChat, a filtered dataset of 82,845 real developer–LLM conversations (311,161 turns, 368,506 code snippets) derived from WildChat, combined with a per-turn static-analysis pipeline—Pylint for Python, ESLint for JavaScript, Cppcheck for C++, PMD for Java, Roslyn for C#—and C4 clone detection to group consecutive turns into task sequences. BERTopic labels developer intents; a prompt-gap taxonomy explains why conversations extend. The static-analysis-on-task-sequences design is what lets the paper track quality evolution across turns.

Load-bearing premise

Static analysis of each code snippet in isolation is treated as a valid measure of quality; identifiers defined in earlier turns, surrounding text, or other files are therefore counted as undefined, which can inflate the headline undefined-variable rates.

What would settle it

Re-run the analysis after concatenating each snippet with prior turns' code from the same conversation (or with stub definitions from surrounding text) and recompute the undefined-variable and missing-import percentages. If Python's 30.8% undefined-variable rate or JavaScript's 75.3% rate drops substantially, the per-snippet static-analysis setup is inflating the defect rates; if they stay high, the paper's quality picture holds. The paper's Section IV-C(2)(c) describes snippet-level analysis; no context-joined numbers are reported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If these rates hold, LLM-generated snippets from chat interfaces should not be treated as production-ready; automated linting and post-generation repair should be part of the workflow.
  • Multi-turn dialogue is not a reliable general fix: it improves some defect classes (Python imports, Java comments) while letting others persist or worsen (Python undefined variables, JavaScript/C++ issues).
  • Prompt design matters: explicitly naming the error and requesting a fix is the most common successful correction pattern, so assistants can be trained to elicit such clarifications.
  • Frequent real-world topics (web design, ML training, low-level memory work) should anchor new code-generation benchmarks more than competitive-programming tasks do.
  • Language-specific failure profiles argue for language-aware generation and checking rather than a single generic code-quality fix.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The snippet-isolation methodology likely inflates some headline numbers: identifiers defined in previous turns or in surrounding prose are counted as undefined, so an 'undefined variable' may be a fragmentation artifact rather than a genuine model failure. The paper's threats-to-validity section does not discuss this possibility.
  • The turn-over-turn rise in Python undefined variables may reflect the model losing track of conversational context as discussions grow; a testable implication is that explicitly repeating or resummarizing variable definitions in prompts should flatten that curve.
  • Since the dataset comes from a publicly exposed ChatGPT-style interface, users here may differ from developers in tightly integrated IDE assistants; the distribution of topics and defect rates could shift for Copilot-style tools, so the language-specific numbers are a lower-bound generalization.
  • A practical next step the paper leaves implicit: conversational assistants should maintain artifacts (files, imports, scope) across turns, since the persistent undefined-name pattern suggests the chat format itself hinders context retention.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces CodeChat, a filtered subset of WildChat containing 82,845 real-world developer–LLM conversations with 368,506 code snippets, and uses it to characterize conversational structure (RQ1), prompt topics (RQ2), and static-analysis-based quality of LLM-generated code across Python, JavaScript, C++, Java, and C# (RQ3). The headline descriptive findings are: median token-ratio 14:1, 68% multi-turn conversations, web design and ML training as the most common topics, and high rates of language-specific quality issues such as undefined variables in JavaScript (75.3%) and invalid naming in Python (83.4%). The paper also reports turn-over-turn trends, including an increase in Python undefined-variable occurrences and a decrease in Java comment violations, and analyzes which follow-up prompt categories most often resolve syntax errors.

Significance. If the analysis is accepted, CodeChat is a valuable large-scale resource for studying real, unstructured developer–LLM interactions, and the topic/turn-structure findings are useful for tool builders. The paper's strengths include releasing the dataset and a replication package, comparing LLM output against external baselines such as Jupyter notebooks and Stack Overflow, and reporting inter-annotator agreement for the prompt-gap and syntax-resolution labels. However, the central RQ3 quality numbers rest on an unexamined assumption—linting each extracted code snippet in isolation—which can materially inflate undefined-variable, missing-include, and unresolved-namespace rates in conversational fragments. Combined with the abstract/table inconsistency for the 83.4% Python figure and the E0602 numerical mismatch, the validity of the headline quality claims is not yet established.

major comments (3)
  1. [Abstract; RQ3 summary; Table VII] The abstract states that generated Python code often includes undefined variables, citing 83.4% of snippets; the RQ3 summary and Section V-A likewise present invalid naming as a Python quality issue. However, Table VII attributes 83.4% to Convention C0103 (InvalidName), not to undefined variables. The actual undefined-variable (E0602) row shows 20.8% of snippets, and the RQ3 findings text separately quotes 30.8% for the Error category. This mislabeling must be corrected in the abstract and all associated summaries.
  2. [Section IV-C, findings after Table VII; Table VII (Python Error rows)] The text says: 'Undefined variables (E0602) occur in 46.7% of error-type issues (30.8% of snippets)'. Table VII reports 10,037 Error-level diagnostics (30.8% of snippets) and 6,767 E0602 diagnostics (20.8% of snippets); 6,767/10,037 = 67.4%, not 46.7%. The 46.7% figure also appears to be used for comparison against the human-notebook baseline. This inconsistency undermines the quantitative comparison and needs reconciliation, including a corrected derivation of percentages from the underlying counts.
  3. [Section IV-C(2)(a),(c); Table VII; Figure 9; Section VI-A] The static analyzers (Pylint, ESLint, Cppcheck, PMD, Roslyn) are applied to each extracted code snippet in isolation, without reconstructing the surrounding conversation context, prior turns, or implied file-level definitions. In developer–LLM conversations, responses frequently contain partial code that relies on identifiers, headers, or namespaces defined in an earlier turn or in the user's prompt. This can artificially inflate the undefined-variable, missing-#include, and unresolved-namespace rates that are the paper's headline RQ3 findings. It can also drive the claimed turn-over-turn increase in Python undefined variables (23.5% to 32.8%), because later turns are more likely to be incremental fragments. Section VI-A lists configuration and analyzer-coverage threats but does not acknowledge this fragmentation threat. The authors should either re-analyze snippets with conversational c
minor comments (5)
  1. [Section IV-C(2)(c)] The abstract and Section IV-C3 say changes are 'statistically significant (p<0.05, with a low effect size)', but no effect-size values are reported anywhere. Please provide the actual effect sizes or remove the 'low effect size' characterization.
  2. [Table IV] The grid totals to 648 configurations (3×3×18×4), which is stated, but the selection procedure is only described as using coherence plus manual inspection of the top 10. Please report the final chosen hyperparameter values and the coherence scores of the runners-up to improve reproducibility.
  3. [Section IV-C(2)(b), Table VI] The C4-based filtering is central to the multi-turn evolution analysis, but the paper does not report any validation of C4's clone-detection decisions on this conversational dataset. A small manual sample or precision/recall estimate for the applied threshold would strengthen confidence in the task-sequence construction.
  4. [Section IV-A(2)(b)] The annotation process uses ChatGPT-4o-mini for the first stage and manual annotation for the second stage, with Cohen's Kappa 0.81. It would be clearer to state whether the reported kappa is between the automated and manual labels or between two manual annotators, since the latter is the conventional reliability measure.
  5. [Throughout] There are several typos and placeholder issues: 'exsiting' in Section III, 'aplications' in the HDBSCAN expansion in Section IV-B(2)(a), and reference [8] appears as 'Author or Organization'. A thorough copyedit is needed.

Circularity Check

0 steps flagged

No significant circularity: the paper is an empirical measurement study, not a derivation; its headline rates are direct linter outputs compared against external baselines.

full rationale

The paper makes no first-principles prediction and contains no fitted parameter that is later renamed as a prediction. RQ3's defect rates (Section IV-C, Table VII) are direct outputs of external static-analysis tools (Pylint, ESLint, Cppcheck, PMD, Roslyn) applied to extracted snippets; the trend analysis (Section IV-C(2)(c), Eq. P_{m,N}=C_{m,N}/C_N×100) merely re-expresses measured counts as percentages and then tests for trend. There is no equation in which an output is definitionally equal to an input. The CodeChat dataset [83] is self-released but is the empirical object of study, not a derived result used to justify findings. The only substantive self-citation, [43] for Stack Overflow answer length, is an external benchmark comparison in RQ1 and is not load-bearing for the central quality claims. The possible inflation of 'undefined variable' rates by linting snippets in isolation is a measurement-validity threat, not a circularity: the paper reports measured linter outcomes rather than claiming to derive them from an assumption. Therefore no circular step can be exhibited, and the honest finding is no significant circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The findings rest on the assumption that CodeChat is representative of developer-LLM coding interactions and that tool outputs on isolated snippets are valid quality signals. No free parameters are fit to a mathematical target; the main chosen parameters affect topic modeling and intent-shift detection.

free parameters (2)
  • BERTopic hyperparameters (n_neighbors, n_components, min_cluster_size, min_samples) = n_neighbors in [5,10,15], n_components in [2,5,8], min_cluster_size in [30..200], min_samples in [5,10,15,20]; 648 grid-
    Affects topic definitions and prevalence figures in RQ2; selected partly by qualitative interpretability, so it is a fitted modeling choice.
  • Intent-shift threshold = 5% drop in initial-topic probability
    Chosen by hand to define when a conversation shifts topic; affects topic engagement and transition analysis in RQ2.
axioms (6)
  • domain assumption WildChat is a faithful record of real human-LLM interactions
    The whole study uses WildChat; if it contains synthetic, spam, or non-developer traffic, all CodeChat statistics inherit that bias. Section III-A.
  • domain assumption Triple-backtick code blocks mark code-related conversations and their users are developers
    Section III-A: 'we treat user in a code related conversation as developer'; this filter defines the dataset and the term 'developer' throughout.
  • domain assumption Linting isolated snippets measures code quality
    Section IV-C(2)(c); ignores conversational context and treats style conventions as quality defects, directly affecting the headline percentages.
  • domain assumption C4 clone detection correctly separates task-continuation from task-switch turns
    Section IV-C(2)(b); after filtering, only 79.4% of multi-turn conversations remain, and errors here change the turn-evolution results.
  • domain assumption English-language filtering preserves representativeness of topics
    Section IV-B Step 1; topic modeling uses 52,086 English conversations (62.9% of CodeChat), so non-English developer intents are excluded.
  • domain assumption BERTopic clustering yields meaningful topic labels
    Section IV-B Step 3; hyperparameters chosen by coherence and manual interpretability, which can impose an arbitrary topic granularity.

pith-pipeline@v1.3.0-alltime-deepseek · 25994 in / 14890 out tokens · 144386 ms · 2026-08-04T17:52:34.450882+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) are becoming integral to modern software development workflows, assisting developers with code generation, API explanation, and iterative problem-solving through natural language conversations. Despite widespread adoption, there is limited understanding of how developers interact with LLMs in practice and how these conversational dynamics influence task outcomes, code quality, and software engineering workflows. To address this, we leverage CodeChat, a large dataset comprising 82,845 real-world developer-LLM conversations, containing 368,506 code snippets generated across over 20 programming languages, derived from the WildChat dataset. We find that LLM responses are substantially longer than developer prompts, with a median token-length ratio of 14:1. Multi-turn conversations account for 68% of the dataset and often evolve due to shifting requirements, incomplete prompts, or clarification requests. Topic analysis identifies web design (9.6% of conversations) and neural network training (8.7% of conversations) as the most frequent LLM-assisted tasks. Evaluation across five languages (i.e., Python, JavaScript, C++, Java, and C#) reveals prevalent and language-specific issues in LLM-generated code: generated Python and JavaScript code often include undefined variables (83.4% and 75.3% of code snippets, respectively); Java code lacks required comments (75.9%); C++ code frequently omits headers (41.1%) and C# code shows unresolved namespaces (49.2%). During a conversation, syntax and import errors persist across turns; however, documentation quality in Java improves by up to 14.7%, and import handling in Python improves by 3.7% over 5 turns. Prompts that point out mistakes in code generated in prior turns and explicitly request a fix are most effective for resolving errors.

Figures

Figures reproduced from arXiv: 2509.10402 by Bram Adams, Suzhen Zhong, Ying Zou.

Figure 1
Figure 1. Figure 1: Examples of developer–LLM conversations. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the approach. The term “convo” refers to “conversations”, and “PL” in the figure stands for “Programming [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (a) Token length distributions for developer prompts [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: (a) Distribution of conversation turns measured by [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 7
Figure 7. Figure 7: (a) Top 10 most frequent topics in developer–LLM [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Most frequent 3-gram “chain-of-gap” patterns in [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Distribution of the top three quality issues across turns in multi-turn conversations, ranked by their frequency at the [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CodeChat-Eval: Evaluating Large Language Models in Multi-Turn Code Refinement Dialogues

    cs.SE 2026-06 unverdicted novelty 7.0

    CodeChat-Eval shows LLMs lose 19-69% functional correctness across multi-turn code refinement dialogues, with largest drops on logic changes and additive requests.

  2. CodeChat-Eval: Evaluating Large Language Models in Multi-Turn Code Refinement Dialogues

    cs.SE 2026-06 unverdicted novelty 7.0

    CodeChat-Eval shows LLMs lose 19.2% to 69.2% functional correctness over multi-turn refinement dialogues, with largest drops on logic-level and additive changes.

  3. From Human-Centric to Agentic Code Review: The Impact of Different Generations of Generative AI Technology on Review Quality

    cs.SE 2026-07 conditional novelty 6.0

    AI-agent-led and multi-agent code review correlates with faster decisions in some adoption patterns, while heavy single-LLM adoption correlates with more review smells and no efficiency gain.

Reference graph

Works this paper leans on

83 extracted references · 12 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Analysis of ChatGPT-Generated Codes Across Multiple Programming Languages,

    S. Almanasra and K. Suwais, “Analysis of ChatGPT-Generated Codes Across Multiple Programming Languages,”IEEE Access, vol. 13, pp. 23 580–23 596, 2025

    2025

  2. [2]

    Anthropic economic index: Ai’s impact on software development,

    Anthropic, “Anthropic economic index: Ai’s impact on software development,” https://www.anthropic.com/research/ impact-software-development, 2025, accessed: 2025-09-01

    2025

  3. [3]

    Artifacts are now generally available,

    Anthropic, “Artifacts are now generally available,” https://www. anthropic.com/news/artifacts, 2025, accessed: 2025-09-01

    2025

  4. [4]

    Claude ai: Next-generation ai assistant,

    Anthropic, “Claude ai: Next-generation ai assistant,” https://claude.ai/, 2025, accessed: 2025-09-01

    2025

  5. [5]

    Privacy policy,

    Anthropic, “Privacy policy,” https://www.anthropic.com/legal/privacy, 2025, accessed: 2025-09-01

    2025

  6. [6]

    Cursor: The ai code editor,

    Anysphere Inc., “Cursor: The ai code editor,” https://cursor.com/en, 2025, accessed: 2025-09-01

    2025

  7. [7]

    Evaluation of rust code verbosity, understandability and complexity,

    L. Ardito, L. Barbato, R. Coppola, and M. Valsesia, “Evaluation of rust code verbosity, understandability and complexity,”PeerJ Computer Science, vol. 7, p. e406, 2021

    2021

  8. [8]

    Title or description of the website,

    Author or Organization, “Title or description of the website,” https://x. com/, 2025, accessed: 2025-09-01

    2025

  9. [9]

    Density-Based Clustering Based on Hierarchical Density Estimates,

    R. J. G. B. Campello, D. Moulavi, and J. Sander, “Density-Based Clustering Based on Hierarchical Density Estimates,” inAdvances in Knowledge Discovery and Data Mining, J. Pei, V . S. Tseng, L. Cao, H. Motoda, and G. Xu, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 160–172

    2013

  10. [10]

    Vi- cuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing, “Vi- cuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality,” March 2023

    2023

  11. [11]

    CodePrompt: Task-Agnostic Prefix Tuning for Program and Language Generation,

    Y . Choi and J.-H. Lee, “CodePrompt: Task-Agnostic Prefix Tuning for Program and Language Generation,” inFindings of the Association for Computational Linguistics: ACL 2023, A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, jul 2023, pp. 5282–5297

    2023

  12. [12]

    Chollet, “Keras,” https://github.com/keras-team/keras, 2015, accessed: 2025-09-01

    F. Chollet, “Keras,” https://github.com/keras-team/keras, 2015, accessed: 2025-09-01

    2015

  13. [13]

    Cline: Ai coding, open source and uncompromised,

    Cline Bot Inc., “Cline: Ai coding, open source and uncompromised,” https://cline.bot/, 2025, accessed: 2025-09-01

    2025

  14. [14]

    A Performance Study of LLM-Generated Code on Leetcode,

    T. Coignion, C. Quinton, and R. Rouvoy, “A Performance Study of LLM-Generated Code on Leetcode,” inProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, ser. EASE ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 79–89

    2024

  15. [15]

    Cppcheck: A static analyzer for c/c++ code,

    Cppcheck Team, “Cppcheck: A static analyzer for c/c++ code,” https: //cppcheck.sourceforge.io/, 2025, accessed: 2025-09-01

    2025

  16. [16]

    Multi-task learning with deep neural networks: A survey,

    M. Crawshaw, “Multi-task learning with deep neural networks: A survey,”arXiv preprint arXiv:2009.09796, 2020. [Online]. Available: https://arxiv.org/abs/2009.09796

    Pith/arXiv arXiv 2009

  17. [17]

    Why Do Developers Engage with ChatGPT in Issue-Tracker? Investigating Usage and Reliance on ChatGPT-Generated Code,

    J. K. Das, S. Mondal, and C. K. Roy, “Why Do Developers Engage with ChatGPT in Issue-Tracker? Investigating Usage and Reliance on ChatGPT-Generated Code,”arXiv preprint arXiv:2412.06757, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Evaluating Privacy Questions from Stack Overflow: Can ChatGPT Compete?

    Z. Delile, S. Radel, J. Godinez, G. Engstrom, T. Brucker, K. Young, and S. Ghanavati, “Evaluating Privacy Questions from Stack Overflow: Can ChatGPT Compete?” in2023 IEEE 31st International Requirements Engineering Conference Workshops (REW), 2023, pp. 239–244

    2023

  19. [19]

    Nonparametric pairwise multiple comparisons in indepen- dent groups using dunn’s test,

    A. Dinno, “Nonparametric pairwise multiple comparisons in indepen- dent groups using dunn’s test,”The Stata Journal, vol. 15, no. 1, pp. 292–300, 2015

    2015

  20. [20]

    Getting started with eslint - pluggable javascript linter,

    ESLint Team, “Getting started with eslint - pluggable javascript linter,” https://eslint.org/docs/latest/use/getting-started, 2025, accessed: 2025- 09-01

    2025

  21. [21]

    Investigating Code Generation Performance of ChatGPT with Crowd- sourcing Social Data,

    Y . Feng, S. Vanam, M. Cherukupally, W. Zheng, M. Qiu, and H. Chen, “Investigating Code Generation Performance of ChatGPT with Crowd- sourcing Social Data,” in2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC), 2023, pp. 876–885

    2023

  22. [22]

    Mining rule violations in javascript code snippets,

    U. Ferreira Campos, G. Smethurst, J. P. Moraes, R. Bonifácio, and G. Pinto, “Mining rule violations in javascript code snippets,” in2019 IEEE/ACM 16th International Conference on Mining Software Reposi- tories (MSR), 2019, pp. 195–199

    2019

  23. [23]

    Github: Build and ship software on a single, collaborative platform,

    GitHub, Inc., “Github: Build and ship software on a single, collaborative platform,” https://github.com/, 2025, accessed: 2025-09-01

    2025

  24. [24]

    Bertopic: Neural topic modeling with a class- based tf-idf procedure,

    M. Grootendorst, “Bertopic: Neural topic modeling with a class- based tf-idf procedure,”arXiv preprint arXiv:2203.05794, 2022, https: //maartengr.github.io/BERTopic/index.html Accessed: 2025-09-01

    Pith/arXiv arXiv 2022

  25. [25]

    Bertopic frequently asked questions,

    M. Grootendorst, “Bertopic frequently asked questions,” https:// maartengr.github.io/BERTopic/faq.html, 2025, accessed: 2025-09-01

    2025

  26. [26]

    Bertopic parameter tuning guide,

    M. Grootendorst, “Bertopic parameter tuning guide,” https://maartengr.github.io/BERTopic/getting_started/parameter% 20tuning/parametertuning.html, 2025, accessed: 2025-09-01

    2025

  27. [27]

    On the Effectiveness of Large Language Models in Domain- Specific Code Generation,

    X. Gu, M. Chen, Y . Lin, Y . Hu, H. Zhang, C. Wan, Z. Wei, Y . Xu, and J. Wang, “On the Effectiveness of Large Language Models in Domain- Specific Code Generation,”ACM Trans. Softw. Eng. Methodol., vol. 34, no. 3, feb 2025. JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 16

    2025

  28. [28]

    Prompting and Fine-tuning Large Language Models for Automated Code Review Comment Generation,

    M. A. Haider, A. B. Mostofa, S. S. B. Mosaddek, A. Iqbal, and T. Ahmed, “Prompting and Fine-tuning Large Language Models for Automated Code Review Comment Generation,”arXiv preprint arXiv:2411.10129, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    A modified mann-kendall trend test for autocorrelated data,

    K. H. Hamed and A. R. Rao, “A modified mann-kendall trend test for autocorrelated data,”Journal of Hydrology, vol. 204, no. 1, pp. 182–196, 1998

    1998

  30. [30]

    An empirical study on developers’ shared conversations with ChatGPT in GitHub pull requests and issues,

    H. Hao, K. A. Hasan, H. Qin, M. Macedo, Y . Tian, S. H. Ding, and A. E. Hassan, “An empirical study on developers’ shared conversations with ChatGPT in GitHub pull requests and issues,”Empirical Software Engineering, vol. 29, no. 6, p. 150, 2024

    2024

  31. [31]

    Model context protocol (mcp) at first glance: Studying the security and maintainability of mcp servers,

    M. M. Hasan, H. Li, E. Fallahzadeh, G. K. Rajbahadur, B. Adams, and A. E. Hassan, “Model context protocol (mcp) at first glance: Studying the security and maintainability of mcp servers,” 2025

    2025

  32. [32]

    CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generation,

    D. Huang, Q. Bu, Y . Qing, and H. Cui, “CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generation,”arXiv preprint arXiv:2308.08784, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    A Survey on Large Lan- guage Models for Code Generation,

    J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A Survey on Large Lan- guage Models for Code Generation,”arXiv preprint arXiv:2406.00515, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Who Answers It Better? An In-Depth Analysis of ChatGPT and Stack Overflow Answers to Software Engineering Questions,

    S. Kabir, D. N. Udo-Imeh, B. Kou, and T. Zhang, “Who Answers It Better? An In-Depth Analysis of ChatGPT and Stack Overflow Answers to Software Engineering Questions,”CoRR, vol. abs/2308.02312, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    Openassistant conversations-democratizing large language model align- ment,

    A. Köpf, Y . Kilcher, D. von Rütte, S. Anagnostidis, Z. R. Tam, K. Stevens, A. Barhoum, D. Nguyen, O. Stanley, R. Nagyfiet al., “Openassistant conversations-democratizing large language model align- ment,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  36. [36]

    Does code quality affect pull request acceptance? an empirical study,

    V . Lenarduzzi, V . Nikkola, N. Saarimäki, and D. Taibi, “Does code quality affect pull request acceptance? an empirical study,”Journal of Systems and Software, vol. 171, p. 110806, 2021

    2021

  37. [37]

    Refining chatgpt-generated code: Characterizing and mitigating code quality issues,

    Y . Liu, T. Le-Cong, R. Widyasari, C. Tantithamthavorn, L. Li, X.- B. D. Le, and D. Lo, “Refining chatgpt-generated code: Characterizing and mitigating code quality issues,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 5, pp. 1–26, 2024

    2024

  38. [38]

    UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction,

    L. McInnes, J. Healy, and J. Melville, “UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction,”arXiv preprint arXiv:1802.03426, 2020

    Pith/arXiv arXiv 2020

  39. [39]

    Roslyn analyzers github repository,

    Microsoft, “Roslyn analyzers github repository,” https://github.com/ dotnet/roslyn-analyzers, 2025, accessed: 2025-09-01

    2025

  40. [40]

    Enhancing user interaction in chatgpt: Characterizing and consolidating multiple prompts for issue resolution,

    S. Mondal, S. D. Bappon, and C. K. Roy, “Enhancing user interaction in chatgpt: Characterizing and consolidating multiple prompts for issue resolution,” inProceedings of the 21st International Conference on Mining Software Repositories, ser. MSR ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 222–226

    2024

  41. [41]

    D. C. Montgomery, E. A. Peck, and G. G. Vining,Introduction to linear regression analysis. John Wiley & Sons, 2021

    2021

  42. [42]

    The content division element,

    Mozilla Contributors, “The content division element,” https://developer.mozilla.org/en-US/docs/Web/HTML/Element/div, 2025, accessed: 2025-09-01

    2025

  43. [43]

    Image-based communication on social coding platforms,

    M. Nayebi and B. Adams, “Image-based communication on social coding platforms,”J. Softw. Evol. Process, vol. 36, no. 5, Apr. 2024

    2024

  44. [44]

    A Comparison of the Effectiveness of ChatGPT and Co-Pilot for Generating Quality Python Code Solutions,

    N. Nikolaidis, K. Flamos, K. Gulati, D. Feitosa, A. Ampatzoglou, and A. Chatzigeorgiou, “A Comparison of the Effectiveness of ChatGPT and Co-Pilot for Generating Quality Python Code Solutions,” in2024 IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C), 2024, pp. 93–101

    2024

  45. [45]

    Static analysis warnings and automatic fixing: A replication for C# projects,

    M. Odermatt, D. Marcilio, and C. A. Furia, “Static analysis warnings and automatic fixing: A replication for C# projects,” inProceedings of the 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2022, pp. 805–816

    2022

  46. [46]

    Chatgpt,

    OpenAI, “Chatgpt,” https://chatgpt.com/, 2025, accessed: 2025-09-01

    2025

  47. [47]

    Hello gpt-4o,

    OpenAI, “Hello gpt-4o,” https://openai.com/index/hello-gpt-4o, 2025, accessed: 2025-09-01

    2025

  48. [48]

    Privacy policy,

    OpenAI, “Privacy policy,” https://openai.com/policies/privacy-policy, 2025, accessed: 2025-09-01

    2025

  49. [49]

    tiktoken: A fast bpe tokeniser for use with openai’s models,

    OpenAI, “tiktoken: A fast bpe tokeniser for use with openai’s models,” https://github.com/openai/tiktoken, 2025, accessed: 2025-09-01

    2025

  50. [50]

    Pytorch: An imperative style, high- performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high- performance deep learning library,” inAdvances in Neural Information Processing S...

    2019

  51. [51]

    Pmd - source code analyzer documentation,

    PMD Team, “Pmd - source code analyzer documentation,” https://pmd. github.io/pmd/index.html, 2025, accessed: 2025-09-01

    2025

  52. [52]

    Pylint tutorial and documentation,

    Pylint Development Team, “Pylint tutorial and documentation,” https: //pylint.readthedocs.io/en/stable/tutorial.html, 2025, accessed: 2025-09- 01

    2025

  53. [53]

    Black: The uncompromising python code formatter,

    Python Software Foundation, “Black: The uncompromising python code formatter,” https://github.com/psf/black, 2025, accessed: 2025-09-01

    2025

  54. [54]

    Dynamic Scoring Code Token Tree: A Novel Decoding Strategy for Generating High-Performance Code,

    M. Qu, J. Liu, L. Kang, S. Wang, D. Ye, and T. Huang, “Dynamic Scoring Code Token Tree: A Novel Decoding Strategy for Generating High-Performance Code,” inProceedings of the 39th IEEE/ACM Inter- national Conference on Automated Software Engineering, ser. ASE ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 1308–1318

    2024

  55. [55]

    React - the library for web and native user interfaces,

    React Team, “React - the library for web and native user interfaces,” https://react.dev, 2025, accessed: 2025-09-01

    2025

  56. [56]

    Reddit Inc., “Reddit,” https://www.reddit.com/, 2025, accessed: 2025- 09-01

    2025

  57. [57]

    Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,

    N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP- IJCNLP), K. Inui, J. Jiang, V . Ng, and X. Wan, Eds. Hong Kong, China: Association for Com...

    2019

  58. [58]

    Exploring the Space of Topic Coherence Measures,

    M. Röder, A. Both, and A. Hinneburg, “Exploring the Space of Topic Coherence Measures,” inProceedings of the Eighth ACM Interna- tional Conference on Web Search and Data Mining, ser. WSDM ’15. New York, NY , USA: Association for Computing Machinery, 2015, p. 399–408

    2015

  59. [59]

    The wilcoxon signed rank test for paired comparisons of clustered data,

    B. Rosner, R. J. Glynn, and M.-L. T. Lee, “The wilcoxon signed rank test for paired comparisons of clustered data,”Biometrics, vol. 62, no. 1, pp. 185–192, 07 2005

    2005

  60. [60]

    On the Taxonomy of De- velopers’ Discussion Topics with ChatGPT,

    E. Sagdic, A. Bayram, and M. R. Islam, “On the Taxonomy of De- velopers’ Discussion Topics with ChatGPT,” inProceedings of the 21st International Conference on Mining Software Repositories, ser. MSR ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 197–201

    2024

  61. [61]

    A cluster analysis method for grouping means in the analysis of variance,

    A. J. Scott and M. Knott, “A cluster analysis method for grouping means in the analysis of variance,”Biometrics, pp. 507–512, 1974

    1974

  62. [62]

    Prompt engineering or fine-tuning: An empirical assessment of llms for code,

    J. Shin, C. Tang, T. Mohati, M. Nayebi, S. Wang, and H. Hemmati, “Prompt engineering or fine-tuning: An empirical assessment of llms for code,” in2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR), 2025, pp. 490–502

    2025

  63. [63]

    Do code quality and style issues differ across (non-)machine learning notebooks? yes!

    M. S. Siddik and C.-P. Bezemer, “Do code quality and style issues differ across (non-)machine learning notebooks? yes!” in2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM), 2023, pp. 72–83

    2023

  64. [64]

    Quality assessment of chatgpt generated code and their use by developers,

    M. L. Siddiq, L. Roney, J. Zhang, and J. C. D. S. Santos, “Quality assessment of chatgpt generated code and their use by developers,” in Proceedings of the 21st International Conference on Mining Software Repositories, 2024, pp. 152–156

    2024

  65. [65]

    ChatGPT vs LLaMA: Impact, Reliability, and Challenges in Stack Overflow Discussions,

    L. D. Silva, J. Samhi, and F. Khomh, “ChatGPT vs LLaMA: Impact, Reliability, and Challenges in Stack Overflow Discussions,” 2024

    2024

  66. [66]

    Evaluating Source Code Quality with Large Language Models: a comparative study,

    I. R. d. S. Simões and E. Venson, “Evaluating Source Code Quality with Large Language Models: a comparative study,” inProceedings of the XXIII Brazilian Symposium on Software Quality, ser. SBQS ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 103–113

    2024

  67. [67]

    Codechat repository,

    Software Evolution Analytics Lab, “Codechat repository,” https://github. com/Software-Evolution-Analytics-Lab-SEAL/CodeChat.git, 2025, ac- cessed: 2025-09-01

    2025

  68. [68]

    C4: contrastive cross-language code clone detection,

    C. Tao, Q. Zhan, X. Hu, and X. Xia, “C4: contrastive cross-language code clone detection,” inProceedings of the 30th IEEE/ACM In- ternational Conference on Program Comprehension, ser. ICPC ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 413–424

    2022

  69. [69]

    Unveiling ChatGPT’s Usage in Open Source Projects: A Mining-based Study,

    R. Tufano, A. Mastropaolo, F. Pepe, O. Dabic, M. Di Penta, and G. Bavota, “Unveiling ChatGPT’s Usage in Open Source Projects: A Mining-based Study,” inProceedings of the 21st International Confer- ence on Mining Software Repositories, ser. MSR ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 571–583

    2024

  70. [70]

    A study of llms’ preferences for libraries and programming languages,

    L. Twist, J. M. Zhang, M. Harman, D. Syme, J. Noppen, H. Yannakoudakis, and D. Nauck, “A study of llms’ preferences for libraries and programming languages,” 2025. [Online]. Available: https://arxiv.org/abs/2503.17181

    Pith/arXiv arXiv 2025

  71. [71]

    Pep 8 – style guide for python code,

    G. van Rossum, B. Warsaw, and A. Coghlan, “Pep 8 – style guide for python code,” https://peps.python.org/pep-0008, 2025, accessed: 2025- 09-01. JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 17

    2025

  72. [72]

    The kruskal-wallis test and stochastic homogeneity,

    A. Vargha and H. D. Delaney, “The kruskal-wallis test and stochastic homogeneity,”Journal of Educational and Behavioral Statistics, vol. 23, no. 2, pp. 170–192, 1998

    1998

  73. [73]

    Attention is All you Need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is All you Need,” inAdvances in Neural Information Processing Systems, I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017

    2017

  74. [74]

    Cohen’s kappa coeffi- cient as a performance measure for feature selection,

    S. M. Vieira, U. Kaymak, and J. M. C. Sousa, “Cohen’s kappa coeffi- cient as a performance measure for feature selection,” inInternational Conference on Fuzzy Systems, 2010, pp. 1–8

    2010

  75. [75]

    Wildchat: 1 million chatgpt interaction logs in the wild,

    Wenting Zhao and Xiang Ren and Jack Hessel and Claire Cardie and Yejin Choi and Yuntian Deng, “Wildchat: 1 million chatgpt interaction logs in the wild,” https://huggingface.co/datasets/allenai/WildChat, 2025, accessed: 2025-09-01

    2025

  76. [76]

    Devgpt: Studying developer-chatgpt conversations,

    T. Xiao, C. Treude, H. Hata, and K. Matsumoto, “Devgpt: Studying developer-chatgpt conversations,” inProceedings of the 21st Interna- tional Conference on Mining Software Repositories, 2024, pp. 227–230

    2024

  77. [77]

    Hacker news,

    Y Combinator, “Hacker news,” https://news.ycombinator.com/, 2025, accessed: 2025-09-01

    2025

  78. [78]

    On code reuse from stack- overflow: An exploratory study on jupyter notebook,

    M. Yang, Y . Zhou, B. Li, and Y . Tang, “On code reuse from stack- overflow: An exploratory study on jupyter notebook,”arXiv preprint arXiv:2302.11732, 2023

    Pith/arXiv arXiv 2023

  79. [79]

    A study of c/c++ code weaknesses on stack overflow,

    H. Zhang, S. Wang, H. Li, T.-H. Chen, and A. E. Hassan, “A study of c/c++ code weaknesses on stack overflow,”IEEE Transactions on Software Engineering, vol. 48, no. 7, pp. 2359–2375, 2022

    2022

  80. [80]

    WildChat: 1M ChatGPT Interaction Logs in the Wild,

    W. Zhao, X. Ren, J. Hessel, C. Cardie, Y . Choi, and Y . Deng, “WildChat: 1M ChatGPT Interaction Logs in the Wild,”arXiv preprint arXiv:2405.01470, may 2024

    Pith/arXiv arXiv 2024

Showing first 80 references.