Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

The Foundation Cracks: A Comprehensive Study on Bugs and Testing Practices in LLM Libraries

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An empirical study of 313 bug-fixing commits from HuggingFace Transformers and vLLM finds API misuse is the dominant root cause of bugs, and that existing test suites fail to detect most of them.

desk verdict A solid, genuinely new empirical study of LLM-library bugs; the headline API-misuse claim holds qualitatively but the test-effectiveness numbers need rework before they are cited. read the letter →

arxiv 2506.12320 v1 pith:KF4Y4QEO submitted 2025-06-14 cs.SE cs.AI

Weipeng Jiang , Xiaoyu Zhang , Xiaofei Xie , Jiongchi Yu , Yuhan Zhi , Shiqing Ma , Chao Shen This is my paper · ORCID
classification cs.SEcs.AI
keywords LLMlibrariesbugtaxonomyAPImisusetestoracleeffectivenessempiricalstudyHuggingFaceTransformersv
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is the first systematic empirical study of what breaks in modern LLM libraries and why. By manually classifying 313 bug-fixing commits from HuggingFace Transformers and vLLM, it builds taxonomies of 5 bug symptoms and 14 root causes, and it analyzes 7,748 test functions to characterize current testing practice. Its central finding is that API misuse—wrong parameters, missing conditions, wrong or missing calls—has become the leading root cause (32.17%–48.19%), displacing the algorithm-implementation errors that dominated traditional deep learning frameworks. It further claims that most bugs escape detection by the libraries' own test suites, with the gaps traced to missing test cases (41.73%), missing test drivers (32.37%), and weak test oracles (25.90%). A sympathetic reader should care because these libraries are the backbone of LLM deployment, so knowing where the bugs concentrate points testing effort where it can actually pay off.

What carries the argument

The argument rests on two pieces of machinery built by the authors. The first is a grounded-theory classification pipeline that manually labels each bug-fix commit by symptom and root cause, with two independent annotators and Cohen's Kappa above 0.80 used to ensure reliability. The second is a coverage-based effectiveness analysis: for 139 environment-compatible Transformers bugs, the paper extracts the functions modified in the fix commit, compares them against the functions executed by the test suite, and classifies each bug as undetected due to a missing driver, a missing test case, or a missing oracle, depending on whether the function is executed and whether the buggy branch is covered.

What would settle it

Run the 139 environment-compatible bug-fixing commits against the test suite at the parent commit and record actual test failures; if the real failure rate is substantially above 6.10%, or if tests that cover a buggy branch do fail where the paper classifies them as oracle-limited, the reported percentages for missing drivers, cases, and oracles would need revision.

Watch

Extended reading notes

Core claim

The paper's central claim is that the bug landscape in LLM libraries has shifted from algorithm- and type-centric faults to interface-centric ones. API misuse—defined as incorrect parameters, missing or redundant conditions, wrong API calls, and missing or redundant API calls—accounts for 32.17% of the bugs in Transformers and 48.19% in vLLM, while incorrect algorithm implementation, the historical leader in deep learning frameworks, is second at 29.57% and 32.53%. The paper also claims that existing test suites at the time a bug is introduced detect only 6.10% of the studied bugs, and that the reasons are systematically identifiable: 41.73% of bugs sit in functions that tests execute but without reaching the buggy branch, 32.37% are in functions no test invokes, and 25.90% are covered by tests whose oracles are too weak to notice the wrong behavior.

Load-bearing premise

The paper infers that a bug is undetected when its modified function is not executed by the test suite or when the executed tests do not cover the buggy branch, rather than actually running the buggy version of the library and observing whether the tests fail.

Editorial extensions

If this is right

  • If the shift to API misuse is real, quality effort in LLM libraries should focus on interface validation, better parameter checks, and documentation of valid call sequences rather than only on low-level numerical correctness.
  • The 32.37% of bugs in functions no test invokes implies that expanding test drivers to cover newly added functions is a direct and measurable improvement lever.
  • The 41.73% of bugs missed because tests execute the function but not the triggering branch implies that tests should systematically explore parameter combinations and edge cases, especially for interfaces with many parameters.
  • The 25.90% of bugs that escape despite full code coverage implies that oracle design, such as checking semantic properties of outputs rather than shapes and types, is a separate and necessary axis of testing investment.
  • The paper's warning that vLLM's differential oracles inherit Transformers' errors implies that gold-standard references should themselves be independently validated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The API-misuse dominance may extend beyond Transformers and vLLM to other LLM infrastructure; if so, test-generation tools that mine API signatures and documentation could catch a large share of the bugs before release.
  • Because the effectiveness numbers are inferred from coverage rather than from actually running the buggy versions, the reported percentages are best read as approximate; the true detection gap could differ once runtime failures are measured.
  • The finding that only 6.10% of bugs are caught by test failures at introduction time implies that many defects surface only in real deployments, which suggests complementing test suites with production monitoring and differential checks against reference implementations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents an empirical study of bugs and testing practices in two LLM libraries, HuggingFace Transformers and vLLM, based on 313 bug-fixing commits collected over January–December 2024 and 7,748 test functions. The authors construct taxonomies of bug symptoms (5 categories), root causes (14 categories), test granularity levels, and test oracle strategies. The headline findings are that API misuse is the dominant root cause of bugs (32.17% in Transformers and 48.19% in vLLM) and that most bugs remain undetected by current test suites due to lack of test drivers (32.37%), lack of test cases (41.73%), and weak test oracles (25.90%). The paper promises dataset release and offers recommendations for library developers, users, and researchers.

Significance. The study addresses a timely and important topic and appears to be the first systematic empirical investigation of bug characteristics and testing practices specifically for LLM libraries. Its strengths include a substantial manually labeled dataset, an explicit inter-rater reliability protocol, and a direct comparison with prior DL-framework studies such as Chen et al. [11]. If the quantitative findings are validated, the taxonomy and released dataset would be valuable resources for the software engineering community. However, the headline claims currently rest on a fragile category boundary in the root-cause taxonomy, a coverage-based proxy rather than actual execution of buggy code in the test-effectiveness analysis, and an unresolved inconsistency in the reported inadequacy percentages. These issues are fixable but require additional validation before the central conclusions can be taken at face value.

major comments (3)
  1. [Section V.A and V.B] The claim that API Misuse is the predominant root cause is sensitive to the classification boundary between API Misuse (A) and Incorrect Algorithm Implementation (D). In Transformers, A accounts for 74 bugs and D for 68 (32.17% vs. 29.57%), so a net reclassification of just six commits reverses the ordering. The definition of A includes 'missing or redundant conditions' and 'wrong API calls,' which can also be naturally described as control-flow or algorithm-implementation errors under D, yet no decision rule is provided for such borderline cases. The reported Cohen's Kappa >0.80 is a global value on a 20% sample; per-category confusion matrices are needed to rule out systematic boundary drift. Without this evidence, the 'significant departure from DL frameworks' conclusion is not yet established.
  2. [Section VI.C] The test-effectiveness analysis does not actually execute the buggy versions of the library. Instead, it infers non-detection from whether buggy functions are executed by the test suite and whether the specific buggy branches are covered. Coverage is a necessary but not sufficient condition for detection: a test can execute the faulty line and still pass if the oracle is weak, which is precisely the 'lack of test oracles' category the paper claims to measure. The reported percentages of 32.37%, 41.73%, and 25.90% therefore rest on an unvalidated proxy. I recommend running the pre-fix commits against the test suite for the 139 locally compatible bugs (or a representative sample) and reporting actual detection rates, or alternatively reframing the results as a coverage-gap analysis rather than a measurement of test effectiveness. The analysis is also limited to Transformers and to 139 of 230 bugs, so Finding 6 should be scoped accordingly.
  3. [Section VI.C] The three inadequacy categories are reported as 32.37%, 41.73%, and 25.90%, which sum to 100%, yet the text states that test oracle limitations 'can co-exist with missing test drivers or cases' and that the actual proportion of oracle issues 'may be higher.' If the categories are mutually exclusive, the co-existence statement is inconsistent with the data; if they overlap, the percentages cannot sum to 100% and the interpretation of the categories changes. The assignment rule for these categories should be clarified, and any overlaps should be reported explicitly.
minor comments (6)
  1. [Throughout (Figures 2-5 and Section III.A.3)] The manuscript contains numerous '/uni...' encoding artifacts, for example after Section III.A.3 and in the captions of Figures 2, 3, 4, and 5. These appear to be PDF-extraction noise and should be removed or replaced with the intended figure content.
  2. [Section I] The phrase 'aiming to answers three key research questions' should be corrected to 'aiming to answer three key research questions.'
  3. [Section II] The typo 'e/g.,' should be 'e.g.,.'
  4. [Section VI.C] The sentence 'It is important to note that test oracle limitations can co-exist with missing test drivers or cases. suggesting that the actual proportion of lack of oracle issues may be higher.' contains a capitalization error after the period and should be split into two proper sentences.
  5. [Finding 6] Finding 6 states that 'Most bugs can not be identified by existing tests' without qualifying that the effectiveness analysis was performed on a subset of 139 locally compatible bugs in a single library (Transformers). Please scope the finding or provide equivalent evidence for vLLM.
  6. [References] Reference [18] is listed as 'Our repository' with an anonymous URL; the camera-ready version should provide a permanent, non-anonymous public link to the released dataset.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an empirical taxonomy and measurement effort with no derivation chain that reduces to its own inputs.

full rationale

This paper makes no formal derivation or prediction claim. Its central results are empirical counts and manually annotated taxonomies (5 symptom types, 14 root causes, 7 oracle categories) over 313 bug-fixing commits and 7,748 test functions. The headline claim that API misuse is the dominant root cause (32.17%-48.19%) is a direct annotation count, not a quantity derived from a fitted parameter or from a definition that presupposes the outcome. The taxonomy categories can overlap (e.g., 'API Misuse' includes missing conditions that might also be described as control-flow or algorithm errors), but the paper reports an inter-rater reliability protocol with Cohen's Kappa > 0.80, and the possibility of annotation boundary ambiguity is a threat-to-validity concern, not circularity. The testing-effectiveness analysis in Section VI.C uses function-level coverage and branch coverage as a proxy for bug detection rather than executing the buggy versions and observing test failures; this is an unvalidated proxy that weakens the precision of the 32.37%/41.73%/25.90% figures, but it is not a case of a fitted input being renamed as a prediction or a result being forced by construction. Self-citations appear only for methodology (bug-collection practices, test-level classifications) and are not load-bearing for the main findings, which are compared against external studies such as Chen et al. [11]. No equation or definition in the paper makes a claimed output identical to its input. Therefore the paper is not circular; it is an empirical study whose limitations are validity concerns rather than circular reasoning.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted parameters or invented entities; the paper's numbers are empirical counts from manual annotation. The central claims rest on modeling assumptions: representativeness of the commit sample, coverage as a proxy for detection, validity of the manual taxonomy, and generalizability from two libraries.

assumptions (4)
  • domain assumption Bug-fixing commits identified by keyword filtering and manual inspection (Section III.A.2) are representative of the real bug population in LLM libraries.
    The study bases all conclusions on 313 commits selected through a keyword list and manual review; if this sample is biased, the distribution percentages change.
  • domain assumption Coverage of modified functions by the current test suite is a valid proxy for whether the test suite could have detected the bug (Section VI.C).
    The RQ3 analysis infers detection failure from execution coverage rather than from observing test failures against the buggy code.
  • domain assumption The taxonomy of 14 root causes and 5 symptoms, adapted from prior DL studies and extended by the authors, captures the meaningful distinctions in LLM library bugs (Section III.B.1).
    The headline finding about API misuse depends on how the taxonomy groups heterogeneous defects into a single category.
  • domain assumption The two selected libraries, Transformers and vLLM, together represent the broader LLM library ecosystem (Section III.A.1).
    Generalizations such as 'LLM libraries shift from algorithm-centric to interface-centric bugs' rest on this sample.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Foundation Cracks: A Comprehensive Study on Bugs and Testing Practices in LLM Libraries." pith.science (2026). https://pith.science/paper/KF4Y4QEO

@misc{pith2026250612320,
  author       = {Pith},
  title        = {Pith review of: The Foundation Cracks: A Comprehensive Study on Bugs and Testing Practices in LLM Libraries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KF4Y4QEO}},
  note         = {Machine review of arXiv:2506.12320}
}
read the original abstract

Large Language Model (LLM) libraries have emerged as the foundational infrastructure powering today's AI revolution, serving as the backbone for LLM deployment, inference optimization, fine-tuning, and production serving across diverse applications. Despite their critical role in the LLM ecosystem, these libraries face frequent quality issues and bugs that threaten the reliability of AI systems built upon them. To address this knowledge gap, we present the first comprehensive empirical investigation into bug characteristics and testing practices in modern LLM libraries. We examine 313 bug-fixing commits extracted across two widely-adopted LLM libraries: HuggingFace Transformers and vLLM.Through rigorous manual analysis, we establish comprehensive taxonomies categorizing bug symptoms into 5 types and root causes into 14 distinct categories.Our primary discovery shows that API misuse has emerged as the predominant root cause (32.17%-48.19%), representing a notable transition from algorithm-focused defects in conventional deep learning frameworks toward interface-oriented problems. Additionally, we examine 7,748 test functions to identify 7 distinct test oracle categories employed in current testing approaches, with predefined expected outputs (such as specific tensors and text strings) being the most common strategy. Our assessment of existing testing effectiveness demonstrates that the majority of bugs escape detection due to inadequate test cases (41.73%), lack of test drivers (32.37%), and weak test oracles (25.90%). Drawing from these findings, we offer some recommendations for enhancing LLM library quality assurance.

Figures

Figures reproduced from arXiv: 2506.12320 by the authors.

Figure 1
Figure 1. Typical Architecture of Modern LLM Libraries [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Bug Distribution by Symptoms B. Classification and Labeling Process This study employs a systematic multi-phase classification methodology to ensure scientific rigor and reproducibility in bug and test characterization. 1) Taxonomy Development: The classification framework was developed through an iterative grounded theory approach. Initial taxonomies were grounded in established frameworks from prior DL library stu… view at source ↗
Figure 3
Figure 3. Bug Distribution by Root Causes (X-axis Shows the Abbreviations of Each Category) sistent state management across parallel processes, potentially leading to data corruption or erroneous outputs. Dependent Module Issues (K). The complex and often evolving dependency landscape of LLM libraries introduces a class of bugs related to external packages. This includes missing imports, incompatible version combinations, and… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Test Distribution by Levels logical defects in the underlying implementation, which can silently corrupt the model outputs. The ‘Poor Performance’ symptom is rare overall. It is related to two main categories (i.e., ‘API Misuse’ and ‘Incorrect Algorithm Implementation’…
Figure 5
Figure 5. Figure 5: Test Distribution by Oracles different hardware platforms (e.g., CPU vs. GPU); (2) Intra￾library differential testing compares different internal APIs within the same library that should produce equivalent results; (3) Cross-library differential testing compares output…
Figure 6
Figure 6. Figure 6: Distribution between Test Levels and Oracles [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomy

    cs.SE 2026-03 conditional novelty 6.0 of 10

    MCP server faults form five empirical categories—server setting, server/tool configuration, server/host configuration, documentation, and general programming—confirmed by a 41-practitioner survey.

Reference graph

Works this paper leans on

43 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [11]

    Toward understanding deep learning framework bugs,

    J. Chen, Y . Liang, Q. Shen, J. Jiang, and S. Li, “Toward understanding deep learning framework bugs,”ACM Transactions on Software Engi- neering and Methodology, vol. 32, no. 6, pp. 1–31, 2023

    work page 2023

  2. [1]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. K ¨opf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Baiet al., “Pytorch: An imperative style, high-performance deep learning library,” inAdvances in neural information processing systems, 20...

    work page 2019

  3. [2]

    TensorFlow: Large-scale machine learning on heterogeneous systems,

    M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y . Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Man ´e, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V . Vanho...

    work page 2015

  4. [3]

    Transformers: State-of- the-art natural language processing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y . Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, “Transformers: State-of- the-art natural language processing,” inProceedings of the 2020 Conference on Empirical Me...

    2020

  5. [4]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  6. [5]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,

    J. Rasley, S. Rajbhandari, O. Ruwase, and Y . He, “Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,” inProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ser. KDD ’20. New York, NY , USA: Association for Computing Machinery, 2020, p. 3505–3506. [Online]. A...

    arXiv 2020

  7. [6]

    Architectural foundations for the large language model infrastructures,

    H. Zhu, “Architectural foundations for the large language model infrastructures,” 2024. [Online]. Available: https://arxiv.org/abs/2408. 09205

    work page 2024

  8. [7]

    Efficient training of large language models on distributed infrastructures: A survey,

    J. Duan, S. Zhang, Z. Wang, L. Jiang, W. Qu, Q. Hu, G. Wang, Q. Weng, H. Yan, X. Zhang, X. Qiu, D. Lin, Y . Wen, X. Jin, T. Zhang, and P. Sun, “Efficient training of large language models on distributed infrastructures: A survey,” 2024. [Online]. Available: https://arxiv.org/abs/2407.20018

    arXiv 2024

Show all 43 references
  1. [8]

    Bytedance intern fired for planting malicious code in ai models,

    “Bytedance intern fired for planting malicious code in ai models,” https://arstechnica.com/tech-policy/2024/10/bytedance-intern-fired-for -planting-malicious-code-in-ai-models/, Oct 2024

    2024

  2. [9]

    Critical bugs which affected everyone’s llm training,

    “Critical bugs which affected everyone’s llm training,” https://www.reddit.com/r/singularity/comments/1g9pcbo/i fixed cri tical bugs which affected everyones/, Nov 2024

    2024

  3. [10]

    A comprehensive study on deep learning bug characteristics,

    M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, “A comprehensive study on deep learning bug characteristics,” inProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2019...

    2019

  4. [12]

    Characterizing perfor- mance bugs in deep learning systems,

    J. Cao, B. Chen, C. Sun, L. Hu, and X. Peng, “Characterizing perfor- mance bugs in deep learning systems,”arXiv preprint arXiv:2112.01771, p. 59, 2021

    2021 arXiv

  5. [13]

    Towards understanding the faults of javascript-based deep learning systems,

    L. Quan, Q. Guo, X. Xie, S. Chen, X. Li, and Y . Liu, “Towards understanding the faults of javascript-based deep learning systems,” inProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–13

    2022

  6. [14]

    Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,

    Y . Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang, “Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,” 2023. [Online]. Available: https: //arxiv.org/abs/2212.14834

    2023 arXiv

  7. [15]

    Jitfuzz: Coverage-guided fuzzing for jvm just-in-time compilers,

    M. Wu, M. Lu, H. Cui, J. Chen, Y . Zhang, and L. Zhang, “Jitfuzz: Coverage-guided fuzzing for jvm just-in-time compilers,” inProceedings of the 45th International Conference on Software Engineering, ser. ICSE ’23. IEEE Press, 2023, p. 56–68. [Online]. Available: https://doi.or...

    2023

  8. [16]

    Nnsmith: Generating diverse and valid test cases for deep learning compilers,

    J. Liu, J. Lin, F. Ruffy, C. Tan, J. Li, A. Panda, and L. Zhang, “Nnsmith: Generating diverse and valid test cases for deep learning compilers,” inProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Vol...

    2023

  9. [17]

    A comprehensive study on challenges in deploying deep learning based software,

    Z. Chen, Y . Cao, Y . Liu, H. Wang, T. Xie, and X. Liu, “A comprehensive study on challenges in deploying deep learning based software,” in Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engine...

    2020

  10. [18]

    Our repository,

    “Our repository,” 2025, https://anonymous.4open.science/r/EmpricalLL MLib-E53C

    2025

  11. [19]

    Lora: Low-rank adaptation of large language models

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chenet al., “Lora: Low-rank adaptation of large language models.” ICLR, vol. 1, no. 2, p. 3, 2022

    2022

  12. [20]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,”Advances in neural information processing systems, vol. 36, pp. 10 088–10 115, 2023

    2023

  13. [21]

    Bugs in pods: Understanding bugs in container runtime systems,

    J. Yu, X. Xie, C. Zhang, S. Chen, Y . Li, and W. Shen, “Bugs in pods: Understanding bugs in container runtime systems,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1364–1376

    2024

  14. [22]

    Large-scale analysis of non-termination bugs in real-world oss projects,

    X. Shi, X. Xie, Y . Li, Y . Zhang, S. Chen, and X. Li, “Large-scale analysis of non-termination bugs in real-world oss projects,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 256– 268

    2022

  15. [23]

    A comprehensive study of deep learning compiler bugs,

    Q. Shen, H. Ma, J. Chen, Y . Tian, S.-C. Cheung, and X. Chen, “A comprehensive study of deep learning compiler bugs,” inProceedings of the 29th ACM Joint meeting on european software engineering conference and symposium on the foundations of software engineering, 2021, pp. 968–980

    2021

  16. [24]

    The measurement of observer agreement for categorical data,

    J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,”biometrics, pp. 159–174, 1977

    1977

  17. [25]

    Error happens when using siglipvisionmodel.forward with interpo- late pos encoding=true,

    “Error happens when using siglipvisionmodel.forward with interpo- late pos encoding=true,” https://github.com/huggingface/transformers/ issues/33993, October 2024

    2024

  18. [26]

    Mistral ’sentencepiecetokenizer’ object has no attribute ’id to byte piece’,

    “Mistral ’sentencepiecetokenizer’ object has no attribute ’id to byte piece’,” https://github.com/vllm-project/vllm/issues/9907, November 2024

    2024

  19. [27]

    [v1] fix when max model len is not divisible by block size,

    “[v1] fix when max model len is not divisible by block size,” https: //github.com/vllm-project/vllm/pull/10903, December 2024

    2024

  20. [28]

    Performance bug example in transformers,

    “Performance bug example in transformers,” https://github.com/huggi ngface/transformers/pull/34890, November 2024

    2024

  21. [29]

    Replace torch.tensor() with torch.from numpy() when process- ing numpy arrays,

    “Replace torch.tensor() with torch.from numpy() when process- ing numpy arrays,” https://github.com/huggingface/transformers/issues /33185, August 2024

    2024

  22. [30]

    Rename phi-3 rope scaling type,

    “Rename phi-3 rope scaling type,” https://github.com/huggingface/tra nsformers/pull/31436, June 2024

    2024

  23. [31]

    Llava: fix checkpoint doc,

    “Llava: fix checkpoint doc,” https://github.com/huggingface/transform ers/pull/32458, August 2024

    2024

  24. [32]

    A survey of deep learning library testing methods,

    X. Zhang, W. Jiang, C. Shen, Q. Li, Q. Wang, C. Lin, and X. Guan, “A survey of deep learning library testing methods,”arXiv e-prints, pp. arXiv–2404, 2024

    2024

  25. [33]

    Audee: Automated testing for deep learning frameworks,

    Q. Guo, X. Xie, Y . Li, X. Zhang, Y . Liu, X. Li, and C. Shen, “Audee: Automated testing for deep learning frameworks,” inProceedings of 11 the 35th IEEE/ACM international conference on automated software engineering, 2020, pp. 486–498

    2020

  26. [34]

    Writing unit tests,

    vLLM, “Writing unit tests,” https://docs.vllm.ai/en/v0.7.0/contributing /model/tests.html, 2024

    2024

  27. [35]

    Fixes for issue #33763 in idefics2 model,

    “Fixes for issue #33763 in idefics2 model,” https://github.com/hugging face/transformers/pull/33766, October 2024

    2024

  28. [36]

    Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt,

    Y . Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang, “Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt,”arXiv preprint arXiv:2304.02014, 2023

    2023 arXiv

  29. [37]

    Silent bugs in deep learning frameworks: an empirical study of keras and tensorflow,

    F. Tambon, A. Nikanjam, L. An, F. Khomh, and G. Antoniol, “Silent bugs in deep learning frameworks: an empirical study of keras and tensorflow,”Empirical Software Engineering, vol. 29, no. 1, p. 10, 2024

    2024

  30. [38]

    Understanding performance problems in deep learning systems,

    J. Cao, B. Chen, C. Sun, L. Hu, S. Wu, and X. Peng, “Understanding performance problems in deep learning systems,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 357– 369

    2022

  31. [39]

    To- wards neural architecture-aware exploration of compiler optimizations in a deep learning{graph}compiler,

    G. Verma, S. Finviya, A. M. Malik, M. Emani, and B. Chapman, “To- wards neural architecture-aware exploration of compiler optimizations in a deep learning{graph}compiler,” inProceedings of the 19th ACM International Conference on Computing Frontiers, 2022, pp. 244–250

    2022

  32. [40]

    Moving deep learning into web browser: How far can we go?

    Y . Ma, D. Xiang, S. Zheng, D. Tian, and X. Liu, “Moving deep learning into web browser: How far can we go?” inThe World Wide Web Conference, 2019, pp. 1234–1244

    2019

  33. [41]

    Beyond accuracy: An empirical study on unit testing in open-source deep learning projects,

    H. Wang, S. Yu, C. Chen, B. Turhan, and X. Zhu, “Beyond accuracy: An empirical study on unit testing in open-source deep learning projects,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 4, pp. 1–22, 2024

    2024

  34. [42]

    The seeds of the future sprout from history: Fuzzing for unveiling vulnerabilities in prospective deep-learning libraries,

    Z. Li, J. Wu, X. Ling, T. Luo, Z. Rui, and Y . Wu, “The seeds of the future sprout from history: Fuzzing for unveiling vulnerabilities in prospective deep-learning libraries,”arXiv preprint arXiv:2412.01317, 2024

    2024 arXiv

  35. [43]

    Free lunch for testing: fuzzing deep-learning libraries from open source,

    A. Wei, Y . Deng, C. Yang, and L. Zhang, “Free lunch for testing: fuzzing deep-learning libraries from open source,” in Proceedings of the 44th International Conference on Software Engineering, ser. ICSE ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 99...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.