Pith. sign in

REVIEW 4 major objections 6 minor 52 references

How Robust are LLM-Generated Library Imports? An Empirical Study using Stack Overflow

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-generated Python code favors third-party libraries over Python's built-ins, and a small but real share of imports cannot be installed by name.

desk verdict Useful descriptive study with a new taxonomy and dataset, but the headline majority claim overstates a unique-library count and there are several internal number inconsistencies. read the letter →

arxiv 2507.10818 v1 pith:YHMB7MTJ submitted 2025-07-14 cs.SE cs.AIcs.LG

Jasmine Latendresse , SayedHassan Khatoonabadi , Emad Shihab This is my paper
classification cs.SEcs.AIcs.LG
keywords largelanguagemodelslibraryimportsthird-partydependenciesPythondependencymanagementlicenseanalysisaliasresolutioncodegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what happens when large language models choose software libraries while generating code. Prompting six models with real-world Python questions, the authors find that the models consistently import third-party packages more often than Python's standard library, with 54% of distinct imports being third-party versus 41% standard. The recommended third-party libraries tend to be mature, popular, and permissively licensed, which lowers but does not remove integration risk. About 4.6% of distinct imports cannot be resolved automatically, mostly because the import name is an alias for a differently named installable package, and most completions omit installation instructions. If correct, the findings mean developers cannot treat LLM-generated imports as self-contained; they need tooling that maps import names to installable packages.

What carries the argument

The mechanism is a classification pipeline: import statements are extracted from model completions, each top-level library name is labelled standard, third-party, or unknown using the standard-library list and package-index queries, and third-party libraries are characterized by popularity, maintenance, and license metadata. The load-bearing distinction is the alias phenomenon, an import name that differs from the installable package name, because it explains nearly all unresolved cases and motivates the call for dependency-resolution tooling.

What would settle it

Recompute the standard, third-party, and unknown shares by weighting each import by the number of completions in which it appears; if the weighted third-party share falls below half or the unknown share rises above 10%, the paper's headline split does not describe a typical LLM output.

Watch

Extended reading notes

Core claim

Across 87 distinct libraries extracted from 112 completions produced by six models, the paper reports 54.0% third-party, 41.4% standard, and 4.6% unknown imports. The third-party libraries are typically popular (median 6,962 stars), mature (median age above 7 years), lightweight in transitive dependencies (median 1 dependency), and permissively licensed, with 91% of license instances in the permissive category. The unresolved imports are almost all aliases, such as importing cv2 when the installable package is opencv-python, dateutil for python-dateutil, or yaml for PyYAML, plus one module-level import where a submodule was referenced without its parent library. The authors conclude that LLM-generated imports are technically valid but often lack the contextual support needed to run out of the box.

Load-bearing premise

The headline percentages count each distinct library once regardless of how often it appears across models and questions, so a library seen in every completion weighs the same as one seen once.

Editorial extensions

If this is right

  • Developers should expect LLM-suggested Python code to lean on external packages, making dependency review part of using AI assistants.
  • The typical recommended third-party library is stable and permissively licensed, so the default risk profile is moderate rather than alarming.
  • A small but non-negligible share of imports will fail to install by name; mapping aliases like cv2 to opencv-python would resolve most of those cases.
  • Because most completions do not include installation guidance, users should prompt for or independently supply install commands.
  • Behavior on recent, likely unseen questions matched behavior on the original set, suggesting the pattern is not simply memorization of training data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Weighting libraries by how often they actually appear in completions, rather than counting each distinct library once, would give a per-output rate that could differ from the 54/41/4.6 split.
  • An automated alias-expansion tool that rewrites cv2, dateutil, yaml, and similar names into their installable packages would close most of the usability gap identified here.
  • A simple prompt-level intervention, asking the model to include a pip install command, is a testable extension that could be measured against the paper's baseline.
  • The abstract states that only two models provided installation guidance, while the body names three; the intended count should be reconciled.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper evaluates how six LLMs (two proprietary, four open-source) recommend Python libraries when solving 112 Stack Overflow questions from the StackEval benchmark. It classifies imports as standard, third-party, or unknown; analyzes third-party libraries' popularity, maintenance, and licensing; and investigates the causes of unresolvable imports. The headline results are that LLMs favor third-party libraries (54% vs. 41% standard), that these libraries are generally mature, popular, and permissively licensed, and that 4.6% of libraries cannot be resolved by name alone because of alias or module-level import conventions. The authors also report an installation-guidance gap and a robustness check on a separate StackUnseen dataset.

Significance. If the findings hold, the paper addresses an important and under-explored aspect of LLM-based code generation: the dependency-management consequences of library recommendations. The use of real Stack Overflow questions and six current LLMs, together with a released dataset, are concrete strengths. The topic is timely and the results could inform tooling and developer guidance. However, the central quantitative claims rely on a unit-of-analysis choice that is not adequately justified, and several internal inconsistencies in the reported numbers need to be resolved before the claims can be accepted.

major comments (4)
  1. [Section II-B, Table II, RQ1] The headline claim that 'LLMs predominantly favour third-party libraries over standard ones' is computed from unique library names, not from import occurrences. The text in Section II-B states that 'we consider each library as an individual case regardless of how many times it appears across questions or models.' This means a library appearing in every model's output for many questions has the same weight as a library appearing once. Since RQ1 asks 'how frequently' models rely on standard versus third-party libraries, an occurrence-weighted analysis (or at least reporting both unique-library and occurrence-based percentages) is needed. The released dataset makes this check feasible, and the authors should report it before the majority claim is accepted.
  2. [Section VI-A, Table V] The discussion text contradicts Table V. The text states that 'the median number of stars was 3,308 for StackEval and 4,206 for StackUnseen,' while Table V reports Median (Eval) = 6,962 and Median (Unseen) = 10,400. Similarly, the text says 'the median number of dependencies was 2 for StackEval and 3 for StackUnseen,' but Table V shows 1 and 2. These discrepancies undermine the credibility of the robustness analysis and must be corrected and reconciled.
  3. [Abstract vs. Section V (RQ3)] The abstract claims that 'only two models (out of six) provided installation guidance,' but the RQ3 results state that installation steps occurred in outputs from DeepSeek V3, GPT-4 Turbo, and Qwen2.5 — three models. This is a factual inconsistency about a headline summary of the study's usability findings. The authors should correct the count or explain the discrepancy (e.g., if only two models provided guidance in the alias cases specifically, while three did overall).
  4. [Section VI-A] The Mann-Whitney U test compares distributions of library characteristics between StackEval and StackUnseen, using unique third-party libraries across all models. However, the same libraries can appear in both datasets (e.g., popular libraries like requests or numpy likely appear in both), which violates the independence assumption of the Mann-Whitney U test. The paper should quantify the overlap between the two sets or use a statistical approach that accounts for the paired/repeated nature of the data. As written, the 'no statistically significant differences' conclusion is not fully supported.
minor comments (6)
  1. [Section VIII vs. Section II-B] The Threats to Validity section says 'we used a low temperature value (0.1)' while Section II-B specifies 'near-deterministic temperature (0.01).' Please clarify which value was used, as this affects reproducibility.
  2. [Section IV vs. Section VIII] The RQ2 approach states that metadata was collected in February 2024, while Section VIII says the analysis used 'the latest available metadata (as of September 2025).' These dates need to be reconciled.
  3. [Section IV (RQ2)] Two cross-references appear to be off by one section: 'We discuss this occurrence in more detail in RQ2' and 'We discuss the implications of such aliasing further in RQ2' should both refer to RQ3.
  4. [Section V (RQ3) Motivation] The motivation paragraph says 'RQ2 therefore aims to identify and characterize the common sources of non-functional library recommendations,' but this is RQ3's objective, not RQ2's.
  5. [Section VI-A] The sentence 'As seen in RQ2, in StackEval, we identified 24 such cases (4.3% of the recommendations)' has a section-reference error (should be RQ3) and a numerical inconsistency with the 4.6% reported in RQ1; the relationship between occurrence counts and percentages should be clarified.
  6. [General] There are several typographical or grammatical errors, e.g., 'identifid' in the RQ3 summary, 'our analysis points highlights' in Section VI-B, and a missing possessive in 'model's output' in Section V. A careful proofreading pass is recommended.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the headline percentages are observational tabulations of extracted imports, not fitted or predicted quantities.

full rationale

The paper is an empirical measurement study. The central claims (54.0% third-party, 41.4% standard, 4.6% unknown; mature/popular/permissive characteristics; alias and module causes for unresolved imports) are descriptive statistics computed from completions of six LLMs on StackEval questions, classified via stdlib-list and PyPI metadata. No equation defines an output in terms of the claim it is supposed to establish, and no fitted parameter is renamed as a prediction. The StackUnseen robustness check is an external dataset and yields a non-significant Mann-Whitney U comparison of library characteristics; even if the test has an independence concern because libraries may overlap between samples, that is a statistical validity issue, not a circular reduction. The authors cite their previous dependency-management studies (e.g., [4], [40]) to motivate the characteristics examined, but the quantitative results come from new metadata and manual classification, not from those citations. The counting rule of Section II-B ('we consider each library as an individual case regardless of how many times it appears across questions or models') is an operationalization choice; it could understate occurrence-level frequencies, but it does not make the observed majority split equivalent to an input by construction. No enumerated circularity pattern is present.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters or invented entities; the paper is an empirical measurement. Several domain assumptions about dataset representativeness, import extraction, unit of analysis, and statistical test validity are load-bearing.

assumptions (5)
  • domain assumption The 112 Python questions from StackEval, and the 71 from StackUnseen, are representative of real-world Python programming tasks.
    The study generalizes from these question sets to LLM behavior; if unrepresentative, external validity suffers (Section II-A).
  • domain assumption The regex-based extraction of the parent import name captures the library intent correctly.
    Section II-B: 'from import X.Y, we extract X'; this ignores submodule and relative imports, which could misclassify libraries.
  • domain assumption Each unique library is treated as one observation regardless of frequency across models and questions.
    Section II-B: 'we consider each library as an individual case regardless of how many times it appears.' This weighting choice directly shapes RQ1 percentages.
  • domain assumption PyPI and stdlib-list lookups at metadata collection time correctly identify standard vs third-party vs unknown.
    Section III Approach; packages may be added or removed, and name mismatches (e.g., sklearn) are resolved manually.
  • domain assumption Mann-Whitney U test has independent samples.
    Section VI-A compares library characteristics between StackEval and StackUnseen, but the same libraries can appear in both sets, violating independence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Robust are LLM-Generated Library Imports? An Empirical Study using Stack Overflow." pith.science (2026). https://pith.science/paper/YHMB7MTJ

@misc{pith2026250710818,
  author       = {Pith},
  title        = {Pith review of: How Robust are LLM-Generated Library Imports? An Empirical Study using Stack Overflow},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YHMB7MTJ}},
  note         = {Machine review of arXiv:2507.10818}
}
read the original abstract

Software libraries are central to the functionality, security, and maintainability of modern code. As developers increasingly turn to Large Language Models (LLMs) to assist with programming tasks, understanding how these models recommend libraries is essential. In this paper, we conduct an empirical study of six state-of-the-art LLMs, both proprietary and open-source, by prompting them to solve real-world Python problems sourced from Stack Overflow. We analyze the types of libraries they import, the characteristics of those libraries, and the extent to which the recommendations are usable out of the box. Our results show that LLMs predominantly favour third-party libraries over standard ones, and often recommend mature, popular, and permissively licensed dependencies. However, we also identify gaps in usability: 4.6% of the libraries could not be resolved automatically due to structural mismatches between import names and installable packages, and only two models (out of six) provided installation guidance. While the generated code is technically valid, the lack of contextual support places the burden of manually resolving dependencies on the user. Our findings offer actionable insights for both developers and researchers, and highlight opportunities to improve the reliability and usability of LLM-generated code in the context of software dependencies.

Figures

Figures reproduced from arXiv: 2507.10818 by the authors.

Figure 1
Figure 1. An overview of our approach for filtering and curating our dataset. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. License Types in Libraries Recommended by Each Model [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 42 canonical work pages

  1. [1]

    How reuse influences productivity in object-oriented systems,

    V . R. Basili, L. C. Briand, and W. L. Melo, “How reuse influences productivity in object-oriented systems,” Communications of the ACM , vol. 39, no. 10, pp. 104–116, 1996

    work page 1996

  2. [2]

    An empirical comparison of dependency network evolution in seven software packaging ecosystems,

    A. Decan, T. Mens, and P. Grosjean, “An empirical comparison of dependency network evolution in seven software packaging ecosystems,” Empirical Software Engineering , vol. 24, no. 1, pp. 381–416, 2019

    work page 2019

  3. [3]

    What predicts software developers’ productivity?

    E. Murphy-Hill, C. Jaspan, C. Sadowski, D. Shepherd, M. Phillips, C. Winter, A. Knight, E. Smith, and M. Jorde, “What predicts software developers’ productivity?” IEEE Transactions on Software Engineering, vol. 47, no. 3, pp. 582–594, 2019

    work page 2019

  4. [4]

    Not All Depen- dencies are Equal: An Empirical Study on Production Dependencies in NPM,

    J. Latendresse, S. Mujahid, D. E. Costa, and E. Shihab, “Not All Depen- dencies are Equal: An Empirical Study on Production Dependencies in NPM,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE’22) , 2022

    work page 2022

  5. [5]

    Effective dependency management for the javascript soft- ware ecosystem,

    S. Mujahid, “Effective dependency management for the javascript soft- ware ecosystem,” Ph.D. dissertation, Concordia University, 2021. 11

    work page 2021

  6. [6]

    An empirical study of usages, updates and risks of third-party libraries in java projects,

    Y . Wang, B. Chen, K. Huang, B. Shi, C. Xu, X. Peng, Y . Wu, and Y . Liu, “An empirical study of usages, updates and risks of third-party libraries in java projects,” in 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2020, pp. 35–45

    work page 2020

  7. [7]

    What are the characteris- tics of highly-selected packages? a case study on the npm ecosystem,

    S. Mujahid, R. Abdalkareem, and E. Shihab, “What are the characteris- tics of highly-selected packages? a case study on the npm ecosystem,” Journal of Systems and Software , vol. 198, p. 111588, 2023

    work page 2023

  8. [8]

    An empirical study on selection of open source software-preliminary results,

    O. Hauge, T. Osterlie, C.-F. Sorensen, and M. Gerea, “An empirical study on selection of open source software-preliminary results,” in 2009 ICSE Workshop on Emerging Trends in Free/Libre/Open Source Software Research and Development . IEEE, 2009, pp. 42–47

    work page 2009

Show all 52 references
  1. [9]

    Categorizing developer information needs in software ecosystems,

    N. Haenni, M. Lungu, N. Schwarz, and O. Nierstrasz, “Categorizing developer information needs in software ecosystems,” in Proceedings of the 2013 international workshop on ecosystem architectures , 2013, pp. 1–5

    2013

  2. [10]

    An overview and catalogue of dependency challenges in open source software package registries,

    T. Mens and A. Decan, “An overview and catalogue of dependency challenges in open source software package registries,” in The 23rd Belgium-Netherlands Software Evolution Workshop (BENEVOL 2024) , 2024

    2024

  3. [11]

    Selecting third-party libraries: the practitioners’ perspective,

    E. Larios Vargas, M. Aniche, C. Treude, M. Bruntink, and G. Gousios, “Selecting third-party libraries: the practitioners’ perspective,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Enginee...

    2020

  4. [12]

    When and how to make breaking changes: Policies and practices in 18 open source software ecosystems,

    C. Bogart, C. K ¨astner, J. Herbsleb, and F. Thung, “When and how to make breaking changes: Policies and practices in 18 open source software ecosystems,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 30, no. 4, pp. 1–56, 2021

    2021

  5. [13]

    Type regression testing to detect breaking changes in node. js libraries,

    G. Mezzetti, A. Møller, and M. T. Torp, “Type regression testing to detect breaking changes in node. js libraries,” in 32nd european conference on object-oriented programming (ECOOP 2018) . Schloss Dagstuhl–Leibniz-Zentrum f ¨ur Informatik, 2018, pp. 7–1

    2018

  6. [14]

    Model-based testing of breaking changes in node. js libraries,

    A. Møller and M. T. Torp, “Model-based testing of breaking changes in node. js libraries,” in Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering , 2019, pp. 409–419

    2019

  7. [15]

    A com- prehensive study of bloated dependencies in the maven ecosystem,

    C. Soto-Valero, N. Harrand, M. Monperrus, and B. Baudry, “A com- prehensive study of bloated dependencies in the maven ecosystem,” Empirical Software Engineering , vol. 26, no. 3, p. 45, 2021

    2021

  8. [16]

    Towards better dependency management: A first look at dependency smells in python projects,

    Y . Cao, L. Chen, W. Ma, Y . Li, Y . Zhou, and L. Wang, “Towards better dependency management: A first look at dependency smells in python projects,” IEEE Transactions on Software Engineering , vol. 49, no. 4, pp. 1741–1765, 2022

    2022

  9. [17]

    Dependency smells in javascript projects,

    A. J. Jafari, D. E. Costa, R. Abdalkareem, E. Shihab, and N. Tsan- talis, “Dependency smells in javascript projects,” IEEE Transactions on Software Engineering, vol. 48, no. 10, pp. 3790–3807, 2021

    2021

  10. [18]

    Coverage- based debloating for java bytecode,

    C. Soto-Valero, T. Durieux, N. Harrand, and B. Baudry, “Coverage- based debloating for java bytecode,” ACM Transactions on Software Engineering and Methodology , vol. 32, no. 2, pp. 1–34, 2023

    2023

  11. [19]

    Investigating the resolution of vulnerable dependencies with dependabot security updates,

    H. Mohayeji, A. Agaronian, E. Constantinou, N. Zannone, and A. Sere- brenik, “Investigating the resolution of vulnerable dependencies with dependabot security updates,” in 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) . IEEE, 2023, pp. 234–246

    2023

  12. [20]

    On the use of dependabot security pull requests,

    M. Alfadel, D. E. Costa, E. Shihab, and M. Mkhallalati, “On the use of dependabot security pull requests,” in 2021 IEEE/ACM 18th International conference on mining software repositories (MSR). IEEE, 2021, pp. 254–265

    2021

  13. [21]

    Automating dependency updates in practice: An exploratory study on github dependabot,

    R. He, H. He, Y . Zhang, and M. Zhou, “Automating dependency updates in practice: An exploratory study on github dependabot,” IEEE Transactions on Software Engineering , vol. 49, no. 8, pp. 4004–4022, 2023

    2023

  14. [22]

    Vuln4real: A methodology for counting actually vulnerable dependen- cies,

    I. Pashchenko, H. Plate, S. E. Ponta, A. Sabetta, and F. Massacci, “Vuln4real: A methodology for counting actually vulnerable dependen- cies,” IEEE Transactions on Software Engineering , vol. 48, no. 5, pp. 1592–1609, 2020

    2020

  15. [23]

    Depowl: detecting dependency bugs to prevent compatibility failures,

    Z. Jia, S. Li, T. Yu, C. Zeng, E. Xu, X. Liu, J. Wang, and X. Liao, “Depowl: detecting dependency bugs to prevent compatibility failures,” in 2021 IEEE/ACM 43rd International Conference on Software Engi- neering (ICSE). IEEE, 2021, pp. 86–98

    2021

  16. [24]

    A large-scale survey on the usability of ai programming assistants: Successes and challenges,

    J. T. Liang, C. Yang, and B. A. Myers, “A large-scale survey on the usability of ai programming assistants: Successes and challenges,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–13

    2024

  17. [25]

    The programmer’s assistant: Conversational interaction with a large language model for software development,

    S. I. Ross, F. Martinez, S. Houde, M. Muller, and J. D. Weisz, “The programmer’s assistant: Conversational interaction with a large language model for software development,” in Proceedings of the 28th International Conference on Intelligent User Interfaces , 2023, pp. 491– 514

    2023

  18. [26]

    Evaluation of the programming skills of large language models,

    L. B. Heitz, J. Chamas, and C. Scherb, “Evaluation of the programming skills of large language models,”arXiv preprint arXiv:2405.14388, 2024

    2024 arXiv

  19. [27]

    Stack Overflow - where developers learn, share, & build careers

    “Stack Overflow - where developers learn, share, & build careers.” [Online]. Available: https://stackoverflow.com/

  20. [28]

    Stackeval: Benchmarking llms in coding assistance,

    N. Shah, Z. Genc, and D. Araci, “Stackeval: Benchmarking llms in coding assistance,” Advances in Neural Information Processing Systems, vol. 37, pp. 36 976–36 994, 2024

    2024

  21. [29]

    Popular python libraries and their application domains,

    A. S. Saabith, T. Vinothraj, and M. Fareez, “Popular python libraries and their application domains,” International Journal of Advance Engi- neering and Research Development , vol. 7, no. 11, 2020

    2020

  22. [30]

    Python–the fastest growing programming language,

    K. Srinath, “Python–the fastest growing programming language,” Inter- national Research Journal of Engineering and Technology, vol. 4, no. 12, pp. 354–357, 2017

    2017

  23. [31]

    Assessing the effectiveness of chatgpt in generating python code,

    V . Adamson and J. B ¨agerfeldt, “Assessing the effectiveness of chatgpt in generating python code,” 2023

    2023

  24. [32]

    Chatgpt and python programming homework,

    M. E. Ellis, K. M. Casey, and G. Hill, “Chatgpt and python programming homework,” Decision Sciences Journal of Innovative Education, vol. 22, no. 2, pp. 74–87, 2024

    2024

  25. [33]

    Evaluation of openai o1: Opportunities and challenges of agi,

    T. Zhong, Z. Liu, Y . Pan, Y . Zhang, Y . Zhou, S. Liang, Z. Wu, Y . Lyu, P. Shu, X. Yu et al. , “Evaluation of openai o1: Opportunities and challenges of agi,” arXiv preprint arXiv:2409.18486 , 2024

    2024

  26. [34]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  27. [35]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  28. [36]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  29. [37]

    Wizardlm: Empowering large language models to follow complex instructions,

    C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang, “Wizardlm: Empowering large language models to follow complex instructions,” arXiv preprint arXiv:2304.12244 , 2023

    2023 arXiv

  30. [38]

    Qwen2. 5-coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu et al. , “Qwen2. 5-coder technical report,” arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  31. [39]

    How do software developers use chatgpt? an ex- ploratory study on github pull requests,

    M. Chouchen, N. Bessghaier, M. Begoug, A. Ouni, E. Alomar, and M. W. Mkaouer, “How do software developers use chatgpt? an ex- ploratory study on github pull requests,” in Proceedings of the 21st International Conference on Mining Software Repositories , 2024, pp. 212–216

    2024

  32. [40]

    Dependency update strategies and package characteristics,

    A. Javan Jafari, D. E. Costa, E. Shihab, and R. Abdalkareem, “Dependency update strategies and package characteristics,” ACM Trans. Softw. Eng. Methodol. , vol. 32, no. 6, sep 2023. [Online]. Available: https://doi.org/10.1145/3603110

    2023 doi

  33. [41]

    Surviving software dependencies,

    R. Cox, “Surviving software dependencies,” Communications of the ACM, vol. 62, no. 9, pp. 36–43, 2019

    2019

  34. [42]

    A qualitative study of dependency management and its security implications,

    I. Pashchenko, D.-L. Vu, and F. Massacci, “A qualitative study of dependency management and its security implications,” in Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, 2020, pp. 1513–1531

    2020

  35. [43]

    Open source license inconsistencies on github,

    T. Wolter, A. Barcomb, D. Riehle, and N. Harutyunyan, “Open source license inconsistencies on github,” ACM Transactions on Software Engineering and Methodology , vol. 32, no. 5, pp. 1–23, 2023

    2023

  36. [44]

    Exploring risks in the usage of third-party libraries,

    S. Raemaekers, A. van Deursen, and J. Visser, “Exploring risks in the usage of third-party libraries,” in of the BElgian-NEtherlands software eVOLution seminar, vol. 31, 2011

    2011

  37. [45]

    Can you trust ChatGPT’s package recom- mendations?

    B. Lanyado, “Can you trust ChatGPT’s package recom- mendations?” 3 2023. [Online]. Available: https://vulcan.io/blog/ ai-hallucinations-package-risk

    2023

  38. [46]

    Escaping dependency hell: finding build dependency errors with the unified depen- dency graph,

    G. Fan, C. Wang, R. Wu, X. Xiao, Q. Shi, and C. Zhang, “Escaping dependency hell: finding build dependency errors with the unified depen- dency graph,” in Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis , 2020, pp. 463–474

    2020

  39. [47]

    A context-oriented program- ming approach to dependency hell,

    Y . Tanabe, T. Aotani, and H. Masuhara, “A context-oriented program- ming approach to dependency hell,” in Proceedings of the 10th ACM International Workshop on Context-Oriented Programming: Advanced Modularity for Run-time Composition , 2018, pp. 8–14

    2018

  40. [48]

    On the impact of using trivial packages: An empirical case study on npm and pypi,

    R. Abdalkareem, V . Oda, S. Mujahid, and E. Shihab, “On the impact of using trivial packages: An empirical case study on npm and pypi,” Empirical Software Engineering , vol. 25, pp. 1168–1204, 2020. 12

    2020

  41. [49]

    Helping or not helping? why and how trivial packages impact the npm ecosys- tem,

    X. Chen, R. Abdalkareem, S. Mujahid, E. Shihab, and X. Xia, “Helping or not helping? why and how trivial packages impact the npm ecosys- tem,” Empirical Software Engineering , vol. 26, pp. 1–24, 2021

    2021

  42. [50]

    Welcome to the era of chatgpt et al. the prospects of large language models,

    T. Teubner, C. M. Flath, C. Weinhardt, W. van der Aalst, and O. Hinz, “Welcome to the era of chatgpt et al. the prospects of large language models,” Business & Information Systems Engineering , vol. 65, no. 2, pp. 95–101, 2023

    2023

  43. [51]

    Can chatgpt replace stackoverflow? a study on robustness and reliability of large language model code generation,

    L. Zhong and Z. Wang, “Can chatgpt replace stackoverflow? a study on robustness and reliability of large language model code generation,” 2024

    2024

  44. [52]

    Can chatgpt support developers? an empirical evaluation of large language models for code generation,

    K. Jin, C.-Y . Wang, H. V . Pham, and H. Hemmati, “Can chatgpt support developers? an empirical evaluation of large language models for code generation,” arXiv preprint arXiv:2402.11702 , 2024. 13 APPENDIX Table VI presents the descriptive statistics (mean, standard deviation,...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.