Pith. sign in

REVIEW 3 major objections 4 minor 78 references

Demystifying Deep Learning Compiler Frontend Bugs: An LLM-Aided Empirical Study

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TorchDynamo's frontend bugs cluster into seven root-cause categories, and an LLM prompted with those causes reports 23 new defects in newer PyTorch releases, with 15 confirmed.

desk verdict First serious frontend-specific taxonomy of TorchDynamo bugs, with a useful LLM-aided pipeline; the RQ3 efficacy claim needs a baseline before it should be quoted as demonstrated. read the letter →

arxiv 2607.25651 v1 pith:SSD736VL submitted 2026-07-28 cs.PL cs.SE

Xinyi Yuan , Wei Chen , Jinyi Liu , Pengyu Chen , Jun Wei , Guoquan Wu , Jiaxin Zhu , Tao Huang This is my paper · ORCID
classification cs.PLcs.SE
keywords deeplearningcompilerTorchDynamofrontendbugsrootcausetaxonomyLLM-aidedtestingsymbolicexecutionPy2empiricalstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes a root-cause taxonomy for bugs in TorchDynamo, the graph-capturing frontend of PyTorch 2, based on 123 real, fix-linked issues from a one-year window. It finds that 57 of the 123 bugs trace to symbolic execution of Python objects and that the top categories are wrong handling of execution context/scope and wrong modeling of Python objects, not low-level operator or backend issues. It then uses the taxonomy as prompts for an LLM to synthesize self-contained tests that run under eager mode and torch.compile; 23 of 170 tests expose previously unknown failures, 15 of them confirmed by maintainers across 8 of 15 subcategories. If true, the work gives developers a prioritized map of where frontend tracing breaks and shows a cheap, repeatable way to convert historical bug causes into regression tests for new versions.

What carries the argument

The load-bearing mechanism is a domain-knowledge-enhanced LLM pipeline whose pivot is a hand-built map of TorchDynamo's four core tasks: symbolic bytecode execution, guard generation, graph-break handling, and side-effect deferral. That map is injected into LLM prompts to annotate each bug report with related task, trigger, symptom, and summary, and the verified annotations are clustered into the 7/15 taxonomy. In the second half, each root cause description is turned into a test-generation prompt with constraints (self-contained, minimal, executable with and without torch.compile, with assertions), and results are checked by differential testing between eager and compile modes. The taxonomy

What would settle it

Take the paper's taxonomy and apply the same LLM test-generation prompts to a fresh one-year window of TorchDynamo issues (2025-07 through 2026-07). If the newly reported bugs do not cluster in the same 7 categories, or if the 15 'confirmed' issues are later reclassified by maintainers as expected behavior or non-compiler defects, the central claims are weakened. A direct check is to rerun the 170 generated tests on the next PyTorch release and record how many of the 23 failures still reproduce and whether the 8 pending reports become confirmed or rejected.

Watch

Extended reading notes

Core claim

The central claim is that TorchDynamo's frontend failures are not a random assortment of isolated defects but concentrate around a small set of recurring root causes, with the largest being the impedance mismatch between Python's dynamic object model and TorchDynamo's symbolic bytecode execution. From the 123 bug dataset, the authors build a 7-category/15-subcategory taxonomy: wrong modeling of Python objects (23), wrong modeling of containers (15), desynchronized iterator state (4), missing type conversions (8), wrong handling of execution context and scope (24), uncaptured side effects (15), and guard overspecialization/deficiencies (18), plus 16 unclassified. They further claim that this

Load-bearing premise

The entire argument depends on the 123 closed, fix-linked issues labeled 'module: dynamo' from a single year being a fair sample of all real TorchDynamo frontend bugs; if that sample skews toward crash-prone, easily reported cases, both the taxonomy's distribution and the categories targeted by new tests would not generalize.

Editorial extensions

If this is right

  • Developers should prioritize symbolic execution paths that handle custom classes and containers, since these account for the largest share of frontend bugs and fixes in the dataset are often case-specific patches rather than root-cause repairs.
  • Guard bugs split into two failure modes — overspecialization (which causes excessive recompilation) and missing guards (which cause invalid graph reuse) — implying guard design should be tuned for a correctness-reuse balance.
  • Root-cause-aware LLM test generation found new bugs in 8 of 15 subcategories (47%), all in recent releases, including two tagged by maintainers as silent-correctness issues where the wrong output is produced with no error.
  • The task-entity correlations (e.g., iterator issues only in symbolic execution, global-state and in-place-op issues almost only in side-effect handling) can serve as bug-localization indicators during frontend development.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the taxonomy reflects stable failure mechanisms rather than a single-year reporting artifact, the same root-cause patterns likely appear in other Python-based graph-capture frontends (e.g., JAX tracing or TensorFlow Autograph), and the prompt-injection methodology could transfer with retuned domain knowledge.
  • The effectiveness claim would be cleaner if tested against a control: an LLM prompted with random bug categories or a generic fuzzing prompt, to separate root-cause guidance from the model's general knowledge of torch.compile idioms.
  • Eight of the 23 new bugs are still unconfirmed, so the true hit-rate could shift either way; a later check of whether maintainers close them as real defects is a direct, public test of the method's precision.
  • Because 16 of 123 historical bugs were unclassifiable, the 7/15 taxonomy should be treated as version-specific and likely to grow new categories as TorchDynamo evolves.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports an empirical study of 123 TorchDynamo frontend bugs collected from PyTorch GitHub issues (2024-06-30 to 2025-06-30). Using a domain-knowledge-enhanced LLM-aided workflow with two LLMs and two-author manual verification, it constructs a taxonomy of 7 root-cause categories and 15 subcategories, and analyzes bug distribution across TorchDynamo's symbolic execution, guards, graph breaks, and side-effect handling. The paper then uses GPT-5 to generate root-cause-aware test cases, reporting that 23 of 170 generated tests trigger failures on PyTorch 2.10, with 15 confirmed by maintainers across eight (sub)categories, and claims this demonstrates the efficacy of the methodology.

Significance. If the taxonomy is accepted, this is a useful first systematic study of TorchDynamo frontend bugs, with concrete, linked issue/PR examples and a public dataset. The strengths are the grounded corpus of 123 real bugs, the use of code listings and fixing PRs to support root-cause claims, the two-LLM plus two-author validation protocol, and the public reproducibility artifact. The RQ3 bug-detection result, if properly controlled, would be a significant contribution for DLC testing. However, as reported, the detection experiment lacks the control condition needed to support the abstract's efficacy claim, so the paper's strongest validation claim is currently under-supported.

major comments (3)
  1. [§4.3, §5.3, Table 3] The abstract, Finding #9, and Implication #9 attribute the discovery of 23 previously unknown bugs (15 confirmed) to root-cause-aware LLM test generation. However, the experiment has no baseline or ablation: all 170 tests are generated with root-cause prompts, and no comparison is made against generic LLM prompts, random generation, or existing fuzzers. The data are equally compatible with the hypothesis that GPT-5 can produce plausible differential tests for torch.compile on demand and that PyTorch 2.10 has many easily triggerable frontend defects. The paper should add a controlled comparison, such as the same LLM with non-root-cause prompts, or a coverage/random baseline matched in test count and validation effort, and report the differential yield. Without this, the efficacy claim should be substantially weakened.
  2. [§4.1.1] The dataset is restricted to closed GitHub issues labeled 'module: dynamo' that link to at least one fixing PR, with additional manual exclusions of non-defects, duplicates, feature requests, and performance discussions. This selection likely over-represents bugs that are easy to report, label, and fix, and under-represents silent correctness bugs, performance issues, and bugs that lack clean PR linkage. Therefore distributional claims such as 'crashes dominate (88/123)' and 'symbolic execution accounts for 57/123' may reflect reporting and fixability biases rather than the true fBug population. The paper should either analyze this selection bias (e.g., compare characteristics of excluded issues) or explicitly temper the generalization claims in RQ1/RQ2 and in Section 6.3.
  3. [§5.3, Table 3] The confirmed-bug count is not stable: 8 of the 23 reported issues are still pending, and some may be closed as duplicates or not bugs, which would shrink the confirmed count. The paper should distinguish confirmed-only results from total failures in the text and tables, and should not count pending issues as evidence of efficacy. In addition, the paper says 'ten test cases for each (sub)category' and reports 170 tests, but the abstract and taxonomy state 15 subcategories; clarifying whether the 17 (sub)categories include the single-category C and D and the Others bin would remove a confusing inconsistency.
minor comments (4)
  1. [§6.1] The claimed 80% reduction in analysis time is anecdotal ('approximately one hour' vs. '<30 seconds + <10 minutes verification'). Please either report the measurement protocol or soften the claim.
  2. [Table 2] The 'Others' category contains 16/123 (13%) bugs described as too varied to group. This is a substantial residual; it would help to state explicitly how this category was excluded from root-cause-level test generation and whether the taxonomy's completeness was assessed beyond inspection.
  3. [§5.3, Table 3] The table uses gray boxes to indicate confirmed issues, which may be hard to read in monochrome print or for color-blind readers. Please add a separate status column or use symbols.
  4. [§4.2] No inter-rater agreement metric is reported for the two-author manual review of LLM annotations. Since disagreements were resolved by discussion, reporting a simple agreement measure (even a count) would strengthen the reliability claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: taxonomy is built from external issue reports and the 15 confirmed new bugs are independent evidence; RQ3's missing baseline is an experimental-design question, not a circular reduction.

full rationale

The paper's derivation chain is empirical rather than formal: (i) 123 TorchDynamo issue reports are collected from GitHub under explicit filters; (ii) an LLM aided by TorchDynamo task knowledge produces structured annotations, which two authors manually review; (iii) the annotations are clustered into a 7-category taxonomy; (iv) for each category the LLM generates 170 tests that are run on PyTorch 2.10 in eager-vs-compile differential mode; (v) 23 tests fail and 15 resulting issue reports are confirmed by PyTorch maintainers. No step defines an output in terms of the quantity it claims to predict: the taxonomy is a summary of the 123 input reports, not an assumption that entails the new failures; the 23 new issues carry distinct issue numbers and are externally confirmed, so they are not the original reports renamed or a fitted parameter rescaled as a prediction. The paper uses no self-citation as load-bearing support, and prior-work citations (e.g., symptom categories [26,28,50]) are external. The strongest non-circularity concern is RQ3: 'We follow the method in Section 4.3 to generate ten test cases for each (sub)category' and then Finding #9 says the approach is 'underscored by' discovering bugs in 8/17 subcategories; because no ablation or baseline is reported, the data do not prove that root-cause-aware prompting, rather than generic LLM differential testing or the bug density of PyTorch 2.10, caused the detections. That is a missing-control/validity threat, not a circular reduction: the 15 confirmations are independent and the failures are not logically entailed by the taxonomy. The paper's own Section 6.4 limitation ('for unseen bugs that fall outside the identified categories, the effectiveness of root-cause-aware generation is a concern') also concerns scope, not circularity. Score 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claims rest on study-design assumptions rather than fitted scientific parameters. The two free parameters (10 tests per subcategory; one-year issue window) shape the detection and distribution results. No new physical or conceptual entities are introduced; the 'fBug' term is just a label for frontend bugs.

free parameters (2)
  • test_cases_per_subcategory = 10
    Section 5.3: 'generate ten test cases for each (sub)category'; the RQ3 detection count (23 failures, 15 confirmed) depends on this hand-chosen sample size, with no power analysis or saturation argument.
  • issue time window = 2024-06-30 to 2025-06-30
    Section 4.1.1: inclusion criterion for GitHub issues; the taxonomy's distribution (57/123 symbolic execution etc.) is conditional on this one-year sample and on the closed+PR filter.
assumptions (3)
  • domain assumption GitHub issues labeled 'module: dynamo' with a linked fixing PR are valid, representative instances of TorchDynamo frontend bugs.
    Section 4.1.1 applies this filter automatically; if labels or PR linkages are noisy, incomplete, or biased toward user-reportable fixable bugs, the distribution and taxonomy will be skewed.
  • domain assumption Human-reviewed LLM annotations are accurate enough to cluster 123 bugs into a stable taxonomy.
    Section 4.2 uses LLM outputs plus two-author consensus; no inter-rater reliability measure or independent gold-standard validation is reported.
  • domain assumption A differential failure between eager and torch.compile on PyTorch 2.10 indicates a compiler bug rather than a user error or environment issue.
    Section 4.3/5.3: all 23 failures are reported as bugs, but 8 remain pending confirmation, so the assumption can produce false positives.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Demystifying Deep Learning Compiler Frontend Bugs: An LLM-Aided Empirical Study." pith.science (2026). https://pith.science/paper/SSD736VL

@misc{pith2026260725651,
  author       = {Pith},
  title        = {Pith review of: Demystifying Deep Learning Compiler Frontend Bugs: An LLM-Aided Empirical Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SSD736VL}},
  note         = {Machine review of arXiv:2607.25651}
}
read the original abstract

Deep learning compilers (DLCs) are designed to translate deep learning programs into optimized, hardware-specific code. Typically, DLC frontends translate programs into graph-based intermediate representations (IRs) to enable optimizations. Defects introduced during this stage (termed \emph{fBug}s) are severe yet understudied, as prior work predominantly focuses on low-level APIs and operators or treats DLCs as monolithic entities. To bridge this gap, we conduct the first systematic empirical study of \emph{fBug}s in TorchDynamo, the default DLC frontend for PyTorch 2, the most popular DL framework. Leveraging a domain-knowledge-enhanced LLM-aided methodology, we analyze 123 \emph{fBug}s and construct a taxonomy comprising 7 root cause categories and 15 subcategories. Our findings provide actionable insights for DLC development and testing. Furthermore, we leverage the LLM to generate targeted, root cause-aware test cases to detect new bugs. We uncovered 23 previously unknown \emph{fBug}s in recent releases (15 confirmed) across eight (sub)categories, demonstrating the efficacy of our methodology in testing and hardening DLC frontends.

Figures

Figures reproduced from arXiv: 2607.25651 by the authors.

Figure 1
Figure 1. An Issue of PyTorch Related to DLC Frontend [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. TorchDynamo Workflow 4 Methodology We attempt to answer the following three research questions. • RQ1 (Distribution): Are fBugs prevalent? What are their char￾acteristics in terms of distributions and symptoms? • RQ2 (Root cause): What are the root causes of fBugs? • RQ3 (Bug detection): Are these root causes helpful in discover￾ing new bugs? We design an LLM-aided empirical study methodology. As illus￾trated in [P… view at source ↗
Figure 3
Figure 3. Methodology Overview of Our LLM-Aided Empirical Study [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 5 canonical work pages

  1. [1]

    Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng

    Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Man- junath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System f...

    2016

  2. [2]

    Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael L...

    2024

  3. [3]

    ANTHROPIC. 2025. System Card: Claude Opus 4 & Claude Sonnet 4. https:// www-cdn.anthropic.com/6d8a8055020700718b0c49369f60816ba2a7c285.pdf. Ac- cessed: 2026-03-19

    2025

  4. [4]

    Junjie Chen, Yihua Liang, Qingchao Shen, Jiajun Jiang, and Shuochuan Li. 2023. Toward Understanding Deep Learning Framework Bugs.ACM Trans. Softw. Eng. Methodol.32, 6, Article 135 (Sept. 2023), 31 pages. doi:10.1145/3587155

    doi:10.1145/3587155 2023

  5. [5]

    Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. In13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsba...

    2018

  6. [6]

    PyTorch Contributors. 2024. Issue #128319: [dynamo] Recompilation on a counter- like attribute of nn module. https://github.com/pytorch/pytorch/issues/128319. Accessed: 2026-03-19

    2024

  7. [7]

    PyTorch Contributors. 2024. Issue #128944: torch.compile graph break due to unsupported builtin filter function. https://github.com/pytorch/pytorch/issues/ 128944. Accessed: 2026-03-19

    2024

  8. [8]

    PyTorch Contributors. 2024. Issue #132165: Mutating global variable during inlining a function from imported module breaks in dynamo. https://github.com/ pytorch/pytorch/issues/132165. Accessed: 2026-03-19

    2024

Show all 78 references
  1. [9]

    PyTorch Contributors. 2024. Issue #133063: torch.compile Parsing error results in error. https://github.com/pytorch/pytorch/issues/133063. Accessed: 2026-03-19

    2024

  2. [10]

    PyTorch Contributors. 2024. Issue #134820: [Dynamo] propagate required_grad info while applying autograd function. https://github.com/pytorch/pytorch/ issues/134820. Accessed: 2026-03-19

    2024

  3. [11]

    PyTorch Contributors. 2024. Issue #134844: [Dynamo] Handle tensor attributes. https://github.com/pytorch/pytorch/issues/134844. Accessed: 2026-03-19

    2024

  4. [12]

    PyTorch Contributors. 2024. Issue #136814: Dynamo inlining errors with some calls to nested functions that use captured variables. https://github.com/pytorch/ pytorch/issues/136814. Accessed: 2026-03-19

    2024

  5. [13]

    PyTorch Contributors. 2024. Issue #141118: Dynamo: how to deal with multiple inheritance (nn.Module/MutableMapping). https://github.com/pytorch/pytorch/ issues/141118. Accessed: 2026-03-19

    2024

  6. [14]

    PyTorch Contributors. 2024. Issue #143756: self.__dict__[...] = ... produces a graph break. https://github.com/pytorch/pytorch/issues/143756. Accessed: 2026-03-19

    2024

  7. [15]

    PyTorch Contributors. 2025. Issue #142055: Dynamo doesn’t support != when the compared objects have custom __eq__. https://github.com/pytorch/pytorch/ issues/142055. Accessed: 2026-03-19

    2025

  8. [16]

    PyTorch Contributors. 2025. Issue #144461: ThroughputBenchmark incorrectly change autocast dtype on CPU. https://github.com/pytorch/pytorch/issues/ 144461. Accessed: 2026-03-19

    2025

  9. [17]

    PyTorch Contributors. 2025. Issue #145284: [dynamo] torch.compile ICE on using a sourceless unspecialized NN module as branching condition. https: //github.com/pytorch/pytorch/issues/145284. Accessed: 2026-03-19

    2025

  10. [18]

    PyTorch Contributors. 2025. Issue #150265: Graph break on Ten- sor._make_subclass. https://github.com/pytorch/pytorch/issues/150265. Ac- cessed: 2026-03-19

    2025

  11. [19]

    PyTorch Contributors. 2025. Issue #150540: PropagateUnbackedSymInts does not know about shape checks in guards. https://github.com/pytorch/pytorch/ issues/150540. Accessed: 2026-03-19

    2025

  12. [20]

    PyTorch Contributors. 2025. Issue #153605: [dynamo] aot_eager can’t process try...except when meeting AttributeError. https://github.com/pytorch/pytorch/ issues/153605. Accessed: 2026-03-19

    2025

  13. [21]

    PyTorch Contributors. 2025. Issue #155841: torch.compile fails to trace methods decorated with @lru_cache. https://github.com/pytorch/pytorch/issues/155841. Accessed: 2026-03-19

    2025

  14. [22]

    PyTorch Contributors. 2025. Issue #156720: torch.compile fails with dynamic shape parameters in view operations. https://github.com/pytorch/pytorch/issues/ 156720. Accessed: 2026-03-19. Conference’17, July 2017, Washington, DC, USA Xinyi Yuan, Wei Chen, Jinyi Liu, Pengyu Chen,...

    2025

  15. [23]

    PyTorch Contributors. 2025. Pull Request #142078: [dynamo] Properly handle != under user-defined __eq__. https://github.com/pytorch/pytorch/pull/142078. Accessed: 2026-03-19

    2025

  16. [24]

    Scott Cyphers, Arjun K. Bansal, Anahita Bhiwandiwalla, Jayaram Bobba, Matthew Brookhart, Avijit Chakraborty, William Constable, Christian Convey, Leona Cook, Omar Kanawi, Robert Kimball, Jason Knight, Nikolay Korovaiko, Varun Kumar Vijay, Yixing Lao, Christopher R. Lishka, Jai...

    2018 arXiv

  17. [25]

    Yinlin Deng, Chenyuan Yang, Anjiang Wei, and Lingming Zhang. 2022. Fuzzing deep-learning libraries via automated relational API inference. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering(Sing...

    2022

  18. [26]

    Xiaoting Du, Zheng Zheng, Lei Ma, and Jianjun Zhao. 2021. An Empirical Study on Common Bugs in Deep Learning Compilers. In2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE). 184–195. doi:10.1109/ ISSRE52982.2021.00030

    2021

  19. [28]

    Lili Huang, Qingchao Shen, Dong Wang, Yunping Wu, Meng Wang, and Junjie Chen. 2025. False-Positive Bug Reports in Deep Learning Compilers: Stages, Root Causes, and Mitigation.ACM Trans. Softw. Eng. Methodol.(Nov. 2025). doi:10.1145/3774889 Just Accepted

    2025 doi

  20. [29]

    Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. 2020. Taxonomy of real faults in deep learning systems. InProceedings of the ACM/IEEE 42nd International Conference on Software Engi- neering(Seoul, South Korea)(ICSE ’20)....

    2020

  21. [30]

    Md Johirul Islam, Giang Nguyen, Rangeet Pan, and Hridesh Rajan. 2019. A comprehensive study on deep learning bug characteristics. InProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineeri...

    2019

  22. [31]

    Jaeseong Kwon, Bongjun Jang, Juneyoung Lee, and Kihong Heo. 2025. Optimization-Directed Compiler Fuzzing for Continuous Translation Valida- tion.Proc. ACM Program. Lang.9, PLDI, Article 172 (June 2025), 24 pages. doi:10.1145/3729275

    2025 doi

  23. [32]

    Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler validation via equiv- alence modulo inputs. InProceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation(Edinburgh, United Kingdom) (PLDI ’14). Association for Computing Machinery, Ne...

    2014

  24. [33]

    Cong Li, Yanyan Jiang, Chang Xu, and Zhendong Su. 2025. Validating JIT Com- pilers via Compilation Space Exploration.ACM Trans. Comput. Syst.43, 3, Article 6 (July 2025), 37 pages. doi:10.1145/3715102

    2025 doi

  25. [34]

    Meiziniu Li, Jialun Cao, Yongqiang Tian, Tsz On Li, Ming Wen, and Shing-Chi Cheung. 2023. COMET: Coverage-guided Model Generation For Deep Learning Library Testing.ACM Trans. Softw. Eng. Methodol.32, 5, Article 127 (July 2023), 34 pages. doi:10.1145/3583566

    2023 doi

  26. [35]

    Liang, Carmen Badea, Christian Bird, Robert DeLine, Denae Ford, Nicole Forsgren, and Thomas Zimmermann

    Jenny T. Liang, Carmen Badea, Christian Bird, Robert DeLine, Denae Ford, Nicole Forsgren, and Thomas Zimmermann. 2024. Can GPT-4 Replicate Empirical Software Engineering Research?Proc. ACM Softw. Eng.1, FSE, Article 60 (July 2024), 24 pages. doi:10.1145/3660767

    2024 doi

  27. [36]

    Yuchen Ling, Shengcheng Yu, Chunrong Fang, Quan Zhou, and Zhenyu Chen

  28. [37]

    Jiawei Liu, Jinkun Lin, Fabian Ruffy, Cheng Tan, Jinyang Li, Aurojit Panda, and Lingming Zhang. 2023. NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers. InProceedings of the 28th ACM International Conference on Architectural Support for Programming L...

    2023

  29. [38]

    Jiawei Liu, Yuxiang Wei, Sen Yang, Yinlin Deng, and Lingming Zhang. 2022. Coverage-guided tensor compiler fuzzing with joint IR-pass mutation.Proc. ACM Program. Lang.6, OOPSLA1, Article 73 (April 2022), 26 pages. doi:10.1145/3527317

    2022 doi

  30. [39]

    Vsevolod Livinskii, Dmitry Babokin, and John Regehr. 2020. Random testing for C and C++ compilers with YARPGen. 4, OOPSLA, Article 196 (Nov. 2020), 25 pages. doi:10.1145/3428264

    2020 doi

  31. [40]

    Lopes, Juneyoung Lee, Chung-Kil Hur, Zhengyang Liu, and John Regehr

    Nuno P. Lopes, Juneyoung Lee, Chung-Kil Hur, Zhengyang Liu, and John Regehr

  32. [41]

    McKeeman

    William M. McKeeman. 1998. Differential Testing for Software.Digit. Tech. J.10 (1998), 100–107. https://api.semanticscholar.org/CorpusID:14018070

    1998

  33. [42]

    Mohammad Mehdi Morovati, Amin Nikanjam, Florian Tambon, Foutse Khomh, and Zhen Ming (Jack) Jiang. 2024. Bug characterization in machine learning- based systems.Empir. Softw. Eng.29, 1 (2024), 14. doi:10.1007/S10664-023-10400-0

    2024 doi

  34. [43]

    OpenAI. 2025. GPT-5 System Card. arXiv:2601.03267 [cs.CL] https://arxiv.org/ abs/2601.03267

    2025 arXiv

  35. [44]

    David Pankratz. 2020. TVMFuzz: Fuzzing Tensor-level Intermediate Representa- tion in TVM. https://github.com/dpankratz/TVMFuzz. Accessed: 2026-03-19

    2020

  36. [45]

    2019.PyTorch: an imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fan...

    2019

  37. [46]

    Hung Viet Pham, Thibaud Lutellier, Weizhen Qi, and Lin Tan. 2019. CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 1027–1038. doi:10.1109/ICSE.2019.00107

    2019

  38. [47]

    plaidML Team. 2025. plaidML: A platform for making deep learning work every- where. https://github.com/plaidml/plaidml. Accessed: 2025-12-31

    2025

  39. [48]

    James Reed, Zachary DeVito, Horace He, Ansley Ussery, and Jason Ansel. 2022. torch. fx: Practical program capture and transformation for deep learning in python.Proceedings of Machine Learning and Systems4 (2022), 638–651

    2022

  40. [49]

    Nadav Rotem, Jordan Fix, Saleem Abdulrasool, Garret Catron, Summer Deng, Roman Dzhabarov, Nick Gibson, James Hegeman, Meghan Lele, Roman Leven- stein, et al. 2018. Glow: Graph lowering compiler techniques for neural networks. arXiv preprint arXiv:1805.00907(2018)

    2018 arXiv

  41. [50]

    Qingchao Shen, Haoyang Ma, Junjie Chen, Yongqiang Tian, Shing-Chi Cheung, and Xiang Chen. 2021. A comprehensive study of deep learning compiler bugs. InProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Sof...

    2021

  42. [51]

    Qingchao Shen, Yongqiang Tian, Haoyang Ma, Junjie Chen, Lili Huang, Ruifeng Fu, Shing-Chi Cheung, and Zan Wang. 2025. A Tale of Two DL Cities: When Library Tests Meet Compiler. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society,...

    2025

  43. [52]

    Qingchao Shen, Zan Wang, Haoyang Ma, Yongqiang Tian, Lili Huang, Zibo Xiao, Junjie Chen, and Shing-Chi Cheung. 2025. Optimization-Aware Test Generation for Deep Learning Compilers. arXiv:2511.18918 [cs.SE] https://arxiv.org/abs/ 2511.18918

    2025

  44. [53]

    Jingyi Shi, Yang Xiao, Yuekang Li, Yeting Li, Dongsong Yu, Chendong Yu, Hui Su, Yufeng Chen, and Wei Huo. 2023. ACETest: Automated Constraint Extraction for Testing Deep Learning Operators. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and A...

    2023

  45. [54]

    Florian Tambon, Amin Nikanjam, Le An, Foutse Khomh, and Giuliano Antoniol

  46. [55]

    NVIDIA Team. 2023. Fuser: A Fusion Code Generator for NVIDIA GPUs (com- monly known as nvFuser). https://github.com/NVIDIA/Fuser

    2023

  47. [56]

    OpenVINO Team. 2025. OpenVINO: Open-source software toolkit for optimiz- ing and deploying deep learning models. https://github.com/openvinotoolkit/ openvino. Accessed: 2025-12-31

    2025

  48. [57]

    XLA Team. 2017. XLA (Accelerated Linear Algebra). https://openxla.org/xla TensorFlow Dev Summit

    2017

  49. [58]

    Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Program- ming Languages. 10–19

    2019

  50. [59]

    Nicolas Vasilache, Oleksandr Zinenko, Theodoros Theodoridis, Priya Goyal, Zachary DeVito, William S Moses, Sven Verdoolaege, Andrew Adams, and Albert Cohen. 2018. Tensor comprehensions: Framework-agnostic high-performance machine learning abstractions.arXiv preprint arXiv:1802...

    2018 arXiv

  51. [60]

    Zihan Wang, Pengbo Nie, Xinyuan Miao, Yuting Chen, Chengcheng Wan, Lei Bu, and Jianjun Zhao. 2023. GenCoG: A DSL-Based Approach to Generating Computation Graphs for TVM Testing. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis(Seat...

    2023

  52. [61]

    Zan Wang, Ming Yan, Junjie Chen, Shuang Liu, and Dongdi Zhang. 2020. Deep learning library testing via effective model generation. InProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium Demystifying Deep Learning Compiler Frontend...

    2020

  53. [62]

    AI Is Crazy Website. 2023. Pytorch vs Tensorflow: A Head-to-Head Compari- son. https://aiiscrazy.com/pytorch-vs-tensorflow-a-head-to-head-comparison/. Accessed: 2025-06-28

    2023

  54. [63]

    Anjiang Wei, Yinlin Deng, Chenyuan Yang, and Lingming Zhang. 2022. Free lunch for testing: fuzzing deep-learning libraries from open source. InProceedings of the 44th International Conference on Software Engineering(Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computi...

    2022

  55. [64]

    Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024. Fuzz4All: Universal Fuzzing with Large Language Models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal)(ICSE ’24). Association f...

    2024

  56. [65]

    Danning Xie, Yitong Li, Mijung Kim, Hung Viet Pham, Lin Tan, Xiangyu Zhang, and Michael W. Godfrey. 2022. DocTer: documentation-guided fuzzing for testing deep learning API functions. InProceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysi...

    2022

  57. [66]

    Xiaoyuan Xie, Yan Song, Songqiang Chen, and Jinfu Chen. 2026. Subgraph- Oriented Testing for Deep Learning Libraries.IEEE Transactions on Software Engineering52, 3 (2026), 908–922. doi:10.1109/TSE.2026.3655712

    2026

  58. [67]

    Chenyuan Yang, Yinlin Deng, Runyu Lu, Jiayi Yao, Jiawei Liu, Reyhaneh Jab- barvand, and Lingming Zhang. 2024. WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models.Proc. ACM Program. Lang.8, OOPSLA2, Article 296 (Oct. 2024), 27 pages. doi:10.1145/3689736

    2024 doi

  59. [68]

    Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and under- standing bugs in C compilers(PLDI ’11). Association for Computing Machinery, New York, NY, USA, 283–294. doi:10.1145/1993498.1993532

    2011

  60. [69]

    Sharon Chee Yin Ho, Vahid Majdinasab, Mohayeminul Islam, Diego Elias Costa, Emad Shihab, Foutse Khomh, Sarah Nadi, and Muhammad Raza. 2023. An Empirical Study on Bugs Inside PyTorch: A Replication Study. In2023 IEEE International Conference on Software Maintenance and Evolutio...

    2023

  61. [70]

    Brian Yu, Rubayet Rahman Rongon, Chen Cao, and Xuechen Zhang. 2024. A Study of PyTorch Bug Patterns and Memory-Related Challenges. In2024 IEEE Interna- tional Conference on Big Data (BigData). 7586–7591. doi:10.1109/BigData62323. 2024.10824945

    2024

  62. [71]

    Jiongchi Yu, Weipeng Jiang, Xiaoyu Zhang, Qiang Hu, Xiaofei Xie, and Chao Shen

  63. [72]

    Jie Zhao, Bojie Li, Wang Nie, Zhen Geng, Renwei Zhang, Xiong Gao, Bin Cheng, Chen Wu, Yun Cheng, Zheng Li, Peng Di, Kun Zhang, and Xuefeng Jin. 2021. AKG: automatic kernel generation for neural processing units using polyhedral trans- formations. InProceedings of the 42nd ACM ...

    2021

  64. [73]

    Chijin Zhou, Bingzhou Qian, Gwihwan Go, Quan Zhang, Shanshan Li, and Yu Jiang. 2024. PolyJuice: Detecting Mis-compilation Bugs in Tensor Compilers with Equality Saturation Based Rewriting.Proc. ACM Program. Lang.8, OOPSLA2, Article 317 (Oct. 2024), 27 pages. doi:10.1145/3689757

    2024 doi

  65. [77]

    arXiv:2510.04997 [cs.SE] https://arxiv.org/abs/2510.04997

    AutoEmpirical: LLM-Based Automated Research for Empirical Software Fault Analysis. arXiv:2510.04997 [cs.SE] https://arxiv.org/abs/2510.04997

  66. [231]

    doi:10.1109/ICSME58846.2023.00031

    2023

  67. [594]

    https://www.usenix.org/conference/osdi18/presentation/chen

  68. [2021]

    InProceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation(Virtual, Canada)(PLDI 2021)

    Alive2: bounded translation validation for LLVM. InProceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation(Virtual, Canada)(PLDI 2021). Association for Computing Machinery, New York, NY, USA, 65–79. doi:10.1145/3453483.3454030

    2021

  69. [2023]

    Engg.29, 1 (Nov

    Silent bugs in deep learning frameworks: an empirical study of Keras and TensorFlow.Empirical Softw. Engg.29, 1 (Nov. 2023), 34 pages. doi:10.1007/s10664- 023-10389-6

    2023 doi

  70. [2025]

    LLM-based Crowdsourced Test Report Clustering.ACM Trans. Softw. Eng. Methodol.(Sept. 2025). doi:10.1145/3765756 Just Accepted

    2025 doi

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.