Pith. sign in

REVIEW 17 cited by

Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.02014 v1 pith:CRBLUHXA submitted 2023-04-04 cs.SE

Yinlin Deng , Chunqiu Steven Xia , Chenyuan Yang , Shizhuo Dylan Zhang , Shujing Yang , Lingming Zhang This is my paper · ORCID
classification cs.SE
keywords fuzzgptprogramsfuzzingllmsbugslanguagelearninglibraries
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Deep Learning (DL) library bugs affect downstream DL applications, emphasizing the need for reliable systems. Generating valid input programs for fuzzing DL libraries is challenging due to the need for satisfying both language syntax/semantics and constraints for constructing valid computational graphs. Recently, the TitanFuzz work demonstrates that modern Large Language Models (LLMs) can be directly leveraged to implicitly learn all the constraints to generate valid DL programs for fuzzing. However, LLMs tend to generate ordinary programs following similar patterns seen in their massive training corpora, while fuzzing favors unusual inputs that cover edge cases or are unlikely to be manually produced. To fill this gap, this paper proposes FuzzGPT, the first technique to prime LLMs to synthesize unusual programs for fuzzing. FuzzGPT is built on the well-known hypothesis that historical bug-triggering programs may include rare/valuable code ingredients important for bug finding. Traditional techniques leveraging such historical information require intensive human efforts to design dedicated generators and ensure the validity of generated programs. FuzzGPT demonstrates that this process can be fully automated via the intrinsic capabilities of LLMs (including fine-tuning and in-context learning), while being generalizable and applicable to challenging domains. While FuzzGPT can be applied with different LLMs, this paper focuses on the powerful GPT-style models: Codex and CodeGen. Moreover, FuzzGPT also shows the potential of directly leveraging the instruct-following capability of the recent ChatGPT for effective fuzzing. Evaluation on two popular DL libraries (PyTorch and TensorFlow) shows that FuzzGPT can substantially outperform TitanFuzz, detecting 76 bugs, with 49 already confirmed as previously unknown bugs, including 11 high-priority bugs or security vulnerabilities.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 23 citations worldwide. Full citation record

  1. EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code

    cs.CL 2025-05 conditional novelty 7.0 of 10

    A new six-language benchmark for code efficiency shows that LLMs produce correct but measurably slower code than expert baselines, with the best model at about 62 percent of human runtime efficiency.

  2. GapForge: Directed Compiler Fuzzing via Coverage-Gap Analysis

    cs.SE 2026-07 conditional novelty 6.0 of 10

    GapForge targets uncovered compiler code regions by having an LLM infer what program structures and compiler flags reach them, improving GCC/LLVM line coverage and finding 12 real failures.

  3. When Fuzzing Meets Understanding: LLM-Driven Semantic Test Generation for RTL Verification

    cs.AR 2026-07 conditional novelty 6.0 of 10

    LLM-guided dual-stage hardware fuzzing raises condition coverage by 5.8 points and bug detection by 21.1 points over the strongest baseline on three open-source CPUs.

  4. SAGE: Semantic-Aware Gray-Box Game Regression Testing with Large Language Models

    cs.SE 2025-11 conditional novelty 6.0 of 10

    SAGE generates gray-box game regression tests via LLM-guided RL, compresses them with Pareto optimization, and prioritizes them via LLM-analyzed update logs, beating automated baselines on unique-bug detection at lowe...

  5. Benchmarking LLMs for Unit Test Generation from Real-World Functions

    cs.SE 2025-08 conditional novelty 6.0 of 10

    A new decontaminated benchmark of complex Python functions shows LLMs generate far fewer correct, covering, and bug-killing unit tests than on older benchmarks.

  6. LLMCup: Ranking-Enhanced Comment Updating with LLMs

    cs.SE 2025-07 conditional novelty 6.0 of 10

    LLMCup combines multi-prompt LLM comment generation with a learned ranker, achieving 0.383 exact-match accuracy on the CUP test set versus 0.257 for HebCup and 0.177 for CUP.

  7. Deep Learning Framework Testing via Model Mutation: How Far Are We?

    cs.SE 2025-06 conditional novelty 6.0 of 10

    Existing mutation-based deep learning framework testing tools largely miss high-priority defects such as performance and memory issues, and targeted optimizations to mutation settings can uncover new developer-confirmed bugs.

  8. Understanding LLM-Centric Challenges for Deep Learning Frameworks: An Empirical Analysis

    cs.SE 2025-06 conditional novelty 6.0 of 10

    An empirical study of 38,742 issue reports and 19 interviews produces a 20-theme, 75-sub-theme taxonomy of LLM-centric framework challenges and five recommendations.

  9. The Foundation Cracks: A Comprehensive Study on Bugs and Testing Practices in LLM Libraries

    cs.SE 2025-06 conditional novelty 6.0 of 10

    API misuse is the dominant root cause of bugs in HuggingFace Transformers and vLLM, and most bugs are missed by existing tests because of missing drivers, missing cases, and weak oracles.

  10. Finding Missed Code Size Optimizations in Compilers using LLMs

    cs.SE 2024-12 conditional novelty 6.0 of 10

    An LLM-guided mutation and differential testing tool found 24 confirmed missed code size optimization bugs in production C/C++, Rust, and Swift compilers.

  11. The Seeds of the FUTURE Sprout from History: Fuzzing for Unveiling Vulnerabilities in Prospective Deep-Learning Libraries

    cs.SE 2024-12 conditional novelty 6.0 of 10

    FUTURE fine-tunes CodeLlama to convert historical bug-triggering code from PyTorch and TensorFlow into seed programs for new DL libraries, finding 148 bugs across MLX, MindSpore, and OneFlow.

  12. XAMT: Cross-Framework API Matching for Testing Deep Learning Libraries

    cs.SE 2025-08 conditional novelty 5.0 of 10

    XAMT matches functionally equivalent APIs across five deep learning frameworks and uses variance-guided differential testing to detect 17 bugs, 12 confirmed, that intra-framework tests miss.

  13. MultiFuzz: A Dense Retrieval-based Multi-Agent System for Network Protocol Fuzzing

    cs.CR 2025-08 reject novelty 4.0 of 10

    MultiFuzz combines retrieval-augmented generation and multiple LLM agents within the ChatAFL protocol fuzzer, reporting marginal and statistically unsupported gains in branch coverage and state exploration for RTSP.

  14. Pixels to Play: A Foundation Model for 3D Gameplay

    cs.CV 2025-08 unverdicted novelty 4.0 of 10

    Pixels2Play-0.1 is a decoder-only transformer trained via behavior cloning and inverse-dynamics-imputed actions to play 3D games from pixels, with only qualitative results reported.

  15. Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques

    cs.CR 2025-07 conditional novelty 4.0 of 10

    A survey that maps LLM applications, vulnerabilities, and defenses across eight cybersecurity domains, but with significant citation and rigor problems.

  16. The Current Challenges of Software Engineering in the Era of Large Language Models

    cs.SE 2024-12 conditional novelty 4.0 of 10

    The paper reports 26 challenges in LLM-based software engineering, grouped into seven aspects, derived from a structured discussion among 24 academics and practitioners.

  17. A Contemporary Survey of Large Language Model Assisted Program Analysis

    cs.SE 2025-02 conditional novelty 1.0 of 10

    A review that catalogs how large language models are used in static, dynamic, and hybrid program analysis, and outlines open challenges.

Pith tools