REVIEW 10 cited by
Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT
read the original abstract
Deep Learning (DL) library bugs affect downstream DL applications, emphasizing the need for reliable systems. Generating valid input programs for fuzzing DL libraries is challenging due to the need for satisfying both language syntax/semantics and constraints for constructing valid computational graphs. Recently, the TitanFuzz work demonstrates that modern Large Language Models (LLMs) can be directly leveraged to implicitly learn all the constraints to generate valid DL programs for fuzzing. However, LLMs tend to generate ordinary programs following similar patterns seen in their massive training corpora, while fuzzing favors unusual inputs that cover edge cases or are unlikely to be manually produced. To fill this gap, this paper proposes FuzzGPT, the first technique to prime LLMs to synthesize unusual programs for fuzzing. FuzzGPT is built on the well-known hypothesis that historical bug-triggering programs may include rare/valuable code ingredients important for bug finding. Traditional techniques leveraging such historical information require intensive human efforts to design dedicated generators and ensure the validity of generated programs. FuzzGPT demonstrates that this process can be fully automated via the intrinsic capabilities of LLMs (including fine-tuning and in-context learning), while being generalizable and applicable to challenging domains. While FuzzGPT can be applied with different LLMs, this paper focuses on the powerful GPT-style models: Codex and CodeGen. Moreover, FuzzGPT also shows the potential of directly leveraging the instruct-following capability of the recent ChatGPT for effective fuzzing. Evaluation on two popular DL libraries (PyTorch and TensorFlow) shows that FuzzGPT can substantially outperform TitanFuzz, detecting 76 bugs, with 49 already confirmed as previously unknown bugs, including 11 high-priority bugs or security vulnerabilities.
Forward citations
Cited by 10 Pith papers
-
ClozeMaster: Fuzzing Rust Compiler by Harnessing LLMs for Infilling Masked Real Programs
ClozeMaster masks bracketed structures in historical Rust bug code and uses LLMs to infill them, generating test programs that discovered 27 confirmed bugs in rustc and mrustc while outperforming existing fuzzers.
-
Once4All: Skeleton-Guided SMT Solver Fuzzing with LLM-Synthesized Generators
Once4All synthesizes LLM-based generators from extracted SMT grammars and populates formula skeletons to fuzz Z3 and cvc5, discovering 43 confirmed bugs with 40 fixed.
-
Like a Hammer, It Can Build, It Can Break: Large Language Model Uses, Perceptions, and Adoption in Cybersecurity Operations on Reddit
Reddit analysis of 892 posts shows SOC practitioners adopt LLMs mainly for controllable productivity work and constrain autonomy due to reliability, security, and cost concerns.
-
GapForge: Directed Compiler Fuzzing via Coverage-Gap Analysis
GapForge targets uncovered compiler code regions by having an LLM infer what program structures and compiler flags reach them, improving GCC/LLVM line coverage and finding 12 real failures.
-
When Fuzzing Meets Understanding: LLM-Driven Semantic Test Generation for RTL Verification
LLM-guided dual-stage hardware fuzzing raises condition coverage by 5.8 points and bug detection by 21.1 points over the strongest baseline on three open-source CPUs.
-
ParityFuzz: Finding Inconsistencies across Solidity Compilers via Fine-Grained Mutation and Differential Analysis
ParityFuzz finds 64 new inconsistencies across six Solidity compilers by combining fine-grained mutation rules with reinforcement learning for differential testing.
-
TEMPLATEFUZZ: Fine-Grained Chat Template Fuzzing for Jailbreaking and Red Teaming LLMs
TEMPLATEFUZZ mutates chat templates with element-level rules and heuristic search to reach 98.2% average jailbreak success rate on twelve open-source LLMs while degrading accuracy by only 1.1%.
-
SAGE: Semantic-Aware Gray-Box Game Regression Testing with Large Language Models
SAGE generates gray-box game regression tests via LLM-guided RL, compresses them with Pareto optimization, and prioritizes them via LLM-analyzed update logs, beating automated baselines on unique-bug detection at lowe...
-
SDLLMFuzz: Dynamic-static LLM-assisted greybox fuzzing for structured input programs
SDLLMFuzz combines LLM-based generation of syntactically valid inputs with a dynamic-static feedback loop from crash artifacts to improve bug discovery and time-to-bug on structured-input programs compared to traditio...
-
Like a Hammer, It Can Build, It Can Break: Large Language Model Uses, Perceptions, and Adoption in Cybersecurity Operations on Reddit
Security practitioners use LLMs independently for low-risk productivity tasks while showing interest in enterprise platforms, but reliability, verification needs, and security risks limit broader autonomy.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.