REVIEW 4 major objections 6 minor 37 references
Scheduzz: Constraint-based Fuzz Driver Generation with Dual Scheduling
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Constraint-based API grouping beats manual and AI fuzz drivers
desk verdict A genuinely new combination of LLM constraints and dual scheduling with a substantial but not yet airtight evaluation; deserves peer review with a required artifact and repeated runs. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the constraint pair plus dual scheduler. Explicit constraints are extracted by parsing the library's source with an AST visitor, recording each function's parameter and return types. Implicit constraints are extracted by prompting a large language model to read header files and return JSON-encoded imply and conflict relations. Both are encoded as Prolog facts; the solver's rules eliminate any API group that has an orphan function (no type dependency on another group member) or that violates an imply or conflict relation. On top sits the dual scheduler: the Group Scheduler ranks API groups by four objectives using nondominated sorting, and the Driver Scheduler scores each compiled driver by coverage over execution time and remaining energy, suspending saturated drivers. Together they turn driver generation and execution into one online optimization loop.
What would settle it
Label all 362 implicit constraints by reading each library's documentation and maintainer-approved usage; if the false-positive rate among the unsampled constraints is much higher than the reported 31 of 228, or if removing all implicit constraints does not lower coverage and bug count, the rationality filter is not doing the claimed work.
Extended reading notes
Core claim
The central claim is that rationality of API combinations, not just validity, can be captured by two constraint types and automated. Explicit constraints record shared parameter and return types so that each API in a group depends on another; implicit constraints, inferred from headers by a large language model, encode imply and conflict relations, such as "GetUncompressedLength must precede RawUncompress." A Prolog solver filters all API permutations to those satisfying both constraint types, yielding rational groups. The paper then shows that scheduling both group-to-driver generation and driver execution — using coverage feedback, group similarity, group length, group entropy, and driver saturation — substantially improves fuzzing: higher region coverage than a unit-test-based tool on 16 of 21 libraries, branch coverage of 53.54% versus 33.12%, 35.69%, and 28.22% for two LLM-based tools and a handcrafted fuzzing project, and 33 new bugs with 3 CVEs.
Load-bearing premise
The whole pipeline rests on the trust that the LLM's extracted "must call" and "must not call" constraints are mostly correct, since only 228 of 362 were manually checked and a faulty constraint makes many supposedly rational API groups irrational.
Editorial extensions
If this is right
- If Scheduzz's results hold, library fuzzing can start from source code alone and still beat handcrafted drivers, removing a major human bottleneck.
- Rationality filtering via implicit constraints reduces stillborn and false-positive-prone drivers, so compute time goes to useful campaigns rather than crash-heavy irrational ones.
- The dual scheduling formulation makes fuzzing a library an online optimization problem; future work can apply bandit or online-learning algorithms to the same loop.
- Coverage gains on well-tested libraries translate directly into vulnerability discovery: 33 new bugs, 3 with CVEs, in under 24-hour campaigns.
Reading between the lines
- A natural extension is to let the LLM also propose structured input layouts when the driver needs them; the paper's three underperforming libraries suggest such a component would close the gap.
- The scheduling metrics could be tested in isolation: a randomized trial with different seeds or a multi-armed-bandit selector would show whether the specific Pareto ranking is essential.
- If the implicit-constraint false-positive rate (31 of 228 sampled) is representative, roughly one in eight API groups is labeled rational when it is not; measuring how coverage and bug reports change as the constraint set is cleaned could quantify the ceiling of the approach.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. Scheduzz proposes an LLM-based automatic library fuzzing technique that extracts explicit type constraints via AST analysis and implicit usage constraints (imply/conflict) via LLM prompts, solves them with Prolog to obtain rational API groups, and then applies two asynchronous schedulers (group scheduler and driver scheduler) to prioritize API combinations and fuzz drivers during a fuzzing campaign. The paper evaluates Scheduzz on 21 Utopia-benchmark libraries and 12 PromptFuzz-benchmark libraries, comparing region/branch coverage against Utopia, CKGFuzzer, PromptFuzz, and OSS-Fuzz. It reports higher coverage than Utopia on 16 out of 21 libraries, overall branch-coverage improvements of 1.62x, 1.50x, and 1.89x over CKGFuzzer, PromptFuzz, and OSS-Fuzz, and 33 confirmed previously unknown bugs, 3 of which received CVEs. A component ablation shows that implicit constraints, group scheduling, and driver scheduling each contribute to coverage.
Significance. If the empirical results survive scrutiny, Scheduzz would be a practical advance: it formalizes API rationality as solver constraints, introduces a dual-scheduling view of driver generation and execution, and demonstrates the approach on a large set of real libraries with four baselines and a component ablation. The paper also reports concrete bug findings, including three CVEs, which is a strong external indicator. The strengths are the breadth of the benchmark, the inclusion of an ablation for each major component, and the new formulation of driver generation as an online optimization problem. However, the current evaluation has several load-bearing weaknesses that prevent the central comparative claims from being accepted as stated: inconsistent resource budgets, unstated provenance of baseline numbers, lack of repeated trials, and only partial validation of the LLM-extracted constraints. These issues are addressable by additional experiments and analysis, so the contribution is not fundamentally flawed, but the evidence base needs substantial strengthening.
major comments (4)
- [Section 4, Evaluation Setup] The resource budgets for the comparison are internally contradictory. The text first states: 'while Utopia were run for 100 per-core hours on 112-core CPUs, we limit Scheduzz to 20 per-core hours on 5 cores,' which is 100 core-hours total, but then says 'we configure Scheduzz's fuzzing campaign to 16 cores × 24 hours per library,' which is 384 core-hours per library. These are different budgets by a factor of nearly four, and the latter is not 'stricter' than Utopia's budget. Since the paper's core claim is that Scheduzz reduces computational overhead while achieving higher coverage, the exact per-library core-hour budget for Scheduzz and for every baseline must be stated unambiguously, and the comparison should be made under equalized compute where possible.
- [Section 4, Tables 2 and 3] The manuscript does not state whether Utopia, CKGFuzzer, PromptFuzz, and OSS-Fuzz were rerun in the same environment (hardware, compiler, sanitizer, fuzzing duration) or whether the numbers were taken from prior publications. The note under Table 2 — 'Since Utopia does not publicly disclose its total region coverage, we leave it empty' — is puzzling because per-library Utopia region-coverage values are shown in the same table, suggesting mixed data sources. Coverage differences are only meaningful if all tools are measured under identical conditions; otherwise the reported 16/21 win and the aggregate ratios 1.62x, 1.50x, and 1.89x are not attributable to Scheduzz's design. The paper should either rerun all baselines in a controlled environment or transparently report the exact source and conditions for each baseline number.
- [Section 4, Figures 7–9 and Tables 2–3] All coverage curves and reported coverage numbers appear to be single-run measurements. Fuzzing is randomized and the LLM is queried with temperature 1, so any single trace can be atypical; without repeated trials, error bars, or significance testing, the claims that Scheduzz 'outperforms Utopia on 16 out of 21 libraries' and achieves '1.62x, 1.50x, and 1.89x higher coverage' are not statistically established. At minimum, the paper should report multiple independent runs per configuration (e.g., 5 runs) and provide a statistical comparison such as a paired test or a confidence interval for the per-library differences.
- [Section 4.1 and Section 6] The implicit constraints, which are the key mechanism for ensuring 'rational' API groups, are only validated on a sample: 228 of 362 extracted constraints are inspected, and 31 are classified as false positives, with 15 of those involving non-function entities. All constraints, including the unvalidated ones, are used by the solver. Section 6 acknowledges the hallucination risk and the lack of saturation checks, but the central evaluation does not quantify how sensitive the coverage and bug results are to constraint errors. A concrete test would be to validate all extracted constraints, or to ablate the constraint set by removing low-confidence or sampled-false constraints and re-measuring coverage; otherwise the 'rationality' contribution remains a correctness risk rather than an established property of the approach.
minor comments (6)
- [Section 1, first paragraph] There is a typo in 'Thechallenge 1' — it should be 'The challenge 1'.
- [Figure 4 prompt] The prompt text contains 'contraint' instead of 'constraint' in two places; this is a presentation issue but should be fixed for clarity.
- [Section 2, Definition 2.3] The formal notation for implication and conflict constraints is hard to parse; the arrow symbols are typeset in a way that obscures the intended meaning, and the quantifier structure of the rationality definition should be rewritten in standard notation or explained in prose.
- [Table 2 note] The note about Utopia's total region coverage being unavailable is contradicted by the per-library Utopia coverage values shown in the same table; please clarify which values come from public reports and which were measured or computed by the authors.
- [Section 4.2, Utopia comparison] The sentence 'Scheduzz outperforms it in 17 out of 21 libraries' refers to driver counts, while the later sentence reports 16/21 for region coverage; the two statements are easy to confuse and should be explicitly distinguished.
- [Section 4.3, Table 4] The ablation table shows that the 'without GS' condition generates more drivers than the full system in many rows, which is plausible, but the paper should explain why the full system produces fewer drivers while achieving higher coverage; a sentence on the interaction between scheduling and driver count would help.
Circularity Check
No significant circularity: coverage and bug findings are externally anchored measurements, and the only self-tuned parameter targets failure rate, not the headline metrics.
full rationale
Scheduzz's central claims—higher region/branch coverage than Utopia/CKGFuzzer/PromptFuzz/OSS-Fuzz and 33 bugs with 3 CVEs—are measured outcomes anchored to external libraries and baseline tools, not quantities used to fit the method's parameters. The only parameter chosen from the paper's own data, the maximum of four repeat LLM queries, was tuned to failure rate rather than to the headline coverage numbers (Section 4, Appendix A). The implicit constraints used by the solver are validated on a manually inspected sample (31/228 false positives, Section 4.1), and the contribution of constraints is tested by ablation (Section 4.3), so the rationality claim is not true by definition. The scheduler uses coverage feedback, but this is standard coverage-guided fuzzing feedback, not a tautology: coverage is an observed objective, not an input fitted to a target. The only overlapping-author citations (CEREBRO [16] for nondominated sorting, APICraft [34] in related work) are not load-bearing: the sorting algorithm is standard and externally published, and APICraft is background. Section 6 explicitly acknowledges LLM hallucination risk and lack of saturation checks; this is a robustness limitation, not circularity. The skeptic's resource-budget ambiguity is an experimental-control concern, not a circular-derivation concern.
Assumptions & free parameters
free parameters (4)
- short-term fuzzing threshold =
15 seconds
- max repeated LLM queries =
4
- API group length limit =
5
- query cost limit per library =
$5
assumptions (5)
- domain assumption LLM-extracted implicit constraints (imply/conflict) accurately capture rational library usage conventions.
- domain assumption Type-based explicit constraints, treating every parameter as both input and output, define valid API combinations.
- domain assumption Unit-level library functionality rarely requires more than five APIs.
- domain assumption A driver that crashes within 15 seconds is irrational (a false positive) rather than a real bug.
- domain assumption Code coverage is an appropriate proxy for fuzzing effectiveness.
Cite this review
Pith. "Pith review of Scheduzz: Constraint-based Fuzz Driver Generation with Dual Scheduling." pith.science (2026). https://pith.science/paper/T35DMY3X
@misc{pith2026250718289,
author = {Pith},
title = {Pith review of: Scheduzz: Constraint-based Fuzz Driver Generation with Dual Scheduling},
year = {2026},
howpublished = {\url{https://pith.science/paper/T35DMY3X}},
note = {Machine review of arXiv:2507.18289}
}
read the original abstract
Fuzzing a library requires experts to understand the library usage well and craft high-quality fuzz drivers, which is tricky and tedious. Therefore, many techniques have been proposed to automatically generate fuzz drivers. However, they fail to generate rational fuzz drivers due to the lack of adherence to proper library usage conventions, such as ensuring a resource is closed after being opened. To make things worse, existing library fuzzing techniques unconditionally execute each driver, resulting in numerous irrational drivers that waste computational resources while contributing little coverage and generating false positive bug reports. To tackle these challenges, we propose a novel automatic library fuzzing technique, Scheduzz, an LLM-based library fuzzing technique. It leverages LLMs to understand rational usage of libraries and extract API combination constraints. To optimize computational resource utilization, a dual scheduling framework is implemented to efficiently manage API combinations and fuzz drivers. The framework models driver generation and the corresponding fuzzing campaign as an online optimization problem. Within the scheduling loop, multiple API combinations are selected to generate fuzz drivers, while simultaneously, various optimized fuzz drivers are scheduled for execution or suspension. We implemented Scheduzz and evaluated it in 33 real-world libraries. Compared to baseline approaches, Scheduzz significantly reduces computational overhead and outperforms UTopia on 16 out of 21 libraries. It achieves 1.62x, 1.50x, and 1.89x higher overall coverage than the state-of-the-art techniques CKGFuzzer, Promptfuzz, and the handcrafted project OSS-Fuzz, respectively. In addition, Scheduzz discovered 33 previously unknown bugs in these well-tested libraries, 3 of which have been assigned CVEs.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[1]
Domagoj Babić, Stefan Bucur, Yaohui Chen, Franjo Ivančić, Tim King, Markus Kusano, Caroline Lemieux, László Szekeres, and Wei Wang. 2019. Fudge: fuzz driver generation at scale. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 975–985
work page 2019
-
[2]
Peng Chen, Yuxuan Xie, Yunlong Lyu, Yuxiao Wang, and Hao Chen. 2023. Hop- per: Interpretative fuzzing for libraries. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security . 1600–1614
work page 2023
-
[3]
Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2023. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In Proceedings of the 32nd ACM SIGSOFT international symposium on software testing and analysis . 423–435
2023
-
[4]
Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shu- jing Yang, and Lingming Zhang. 2024. Large Language Models are Edge-Case Generators: Crafting Unusual Programs for Fuzzing Deep Learning Libraries. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineer- ing (Lisbon, Portugal) (ICSE ’24). Association for C...
arXiv 2024
-
[5]
Google. 2023. Google Test Framework. https://github.com/google/googletest
work page 2023
-
[6]
Harrison Green and Thanassis Avgerinos. 2022. GraphFuzz: library API fuzzing with lifetime-aware dataflow graphs. In Proceedings of the 44th International Conference on Software Engineering . 1070–1081
work page 2022
-
[7]
Elad Hazan. 2019. Introduction to Online Convex Optimization. CoRR abs/1909.05207 (2019). arXiv:1909.05207 http://arxiv.org/abs/1909.05207
arXiv 2019
-
[8]
Kyriakos Ispoglou, Daniel Austin, Vishwath Mohan, and Mathias Payer. 2020. {FuzzGen}: Automatic fuzzer generation. In 29th USENIX Security Symposium (USENIX Security 20). 2271–2287
work page 2020
Show all 37 references
-
[9]
Bokdeuk Jeong, Joonun Jang, Hayoon Yi, Jiin Moon, Junsik Kim, Intae Jeon, Taesoo Kim, WooChul Shim, and Yong Ho Hwang. 2023. Utopia: Automatic generation of fuzz driver using unit tests. In 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2676–2692
2023
-
[10]
Jianfeng Jiang, Hui Xu, and Yangfan Zhou. 2021. RULF: Rust library fuzzing via API dependency graph traversal. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 581–592
2021
-
[11]
Jinho Jung, Stephen Tong, Hong Hu, Jungwon Lim, Yonghwi Jin, and Taesoo Kim. 2021. Winnie: Fuzzing windows applications with harness synthesis and fast cloning. In Proceedings of the 2021 Network and Distributed System Security Symposium (NDSS 2021). Conference’17, July 2017, ...
2021
-
[12]
Sungmin Kang, Gabin An, and Shin Yoo. 2023. A preliminary evaluation of llm-based fault localization. arXiv preprint arXiv:2308.05487 (2023)
2023 arXiv
-
[13]
Sungmin Kang, Bei Chen, Shin Yoo, and Jian-Guang Lou. 2023. Explainable automated debugging via large language model-driven scientific debugging. arXiv preprint arXiv:2304.02195 (2023)
2023 arXiv
-
[14]
Seyedehzahra Khoshmanesh and Robyn R Lutz. 2018. The role of similarity in detecting feature interaction in software product lines. In 2018 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) . IEEE, 286– 292
2018
-
[15]
Jia Li, Yunfei Zhao, Yongmin Li, Ge Li, and Zhi Jin. 2024. AceCoder: An Effective Prompting Technique Specialized in Code Generation. ACM Transactions on Software Engineering and Methodology (2024)
2024
-
[16]
Yuekang Li, Yinxing Xue, Hongxu Chen, Xiuheng Wu, Cen Zhang, Xiaofei Xie, Haijun Wang, and Yang Liu. 2019. Cerebro: context-aware adaptive fuzzing for effective vulnerability detection. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conferen...
2019
-
[17]
Adam Lipowski and Dorota Lipowska. 2012. Roulette-wheel selection via sto- chastic acceptance. Physica A: Statistical Mechanics and its Applications 391, 6 (2012), 2193–2196
2012
-
[18]
Yunlong Lyu, Yuxuan Xie, Peng Chen, and Hao Chen. 2024. Prompt Fuzzing for Fuzz Driver Generation. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS 2024, Salt Lake City, UT, USA, October 14-18, 2024, Bo Luo, Xiaojing Liao, Jun Xu...
2024
-
[19]
Ruijie Meng, Martin Mirchev, Marcel Böhme, and Abhik Roychoudhury. 2024. Large language model guided protocol fuzzing. In Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS)
2024
-
[20]
Guowang Miao, Jens Zander, Ki Won Sung, and Slimane Ben Slimane. 2016. Fundamentals of mobile data networks . Cambridge University Press
2016
-
[21]
OSS-Fuzz. 2023. Continuous fuzzing for open source software. https://github. com/google/oss-fuzz
2023
-
[22]
Sudipta Paria, Aritra Dasgupta, and Swarup Bhunia. 2023. Divas: An llm-based end-to-end framework for soc security analysis and policy-based protection. arXiv preprint arXiv:2308.06932 (2023)
2023 arXiv
-
[23]
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering (2023)
2023
-
[24]
Snappy. 2023. A fast compressor/decompressor. https://github.com/google/ snappy
2023
-
[25]
Swi-prolog. 2023. Predicate ->/2. https://www.swi-prolog.org/pldoc/doc_for? object=(-%3E)/2
2023
-
[26]
Swi-prolog. 2023. Predicate not. https://www.swi-prolog.org/show-tag?tag=not
2023
-
[27]
SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. 2024. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313 (2024)
2024 arXiv
-
[28]
Vasudev Vikram, Caroline Lemieux, and Rohan Padhye. 2023. Can large language models write good property-based tests? arXiv preprint arXiv:2307.04346 (2023)
2023 arXiv
-
[29]
Jan Wielemaker, Tom Schrijvers, Markus Triska, and Torbjörn Lager. 2010. SWI- Prolog. CoRR abs/1011.5332 (2010). arXiv:1011.5332 http://arxiv.org/abs/1011. 5332
2010 arXiv
-
[30]
Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024. Fuzz4all: Universal fuzzing with large language models. In Pro- ceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–13
2024
-
[31]
Zhuokui Xie, Yinghao Chen, Chen Zhi, Shuiguang Deng, and Jianwei Yin. 2023. ChatUniTest: a ChatGPT-based automated unit test generation tool.arXiv preprint arXiv:2305.04764 (2023)
2023 arXiv
-
[32]
Hanxiang Xu, Wei Ma, Ting Zhou, Yanjie Zhao, Kai Chen, Qiang Hu, Yang Liu, and Haoyu Wang. 2024. A Code Knowledge Graph-Enhanced System for LLM-Based Fuzz Driver Generation. arXiv preprint arXiv:2411.11532 (2024)
2024 arXiv
-
[33]
Cen Zhang, Yuekang Li, Hao Zhou, Xiaohan Zhang, Yaowen Zheng, Xian Zhan, Xiaofei Xie, Xiapu Luo, Xinghua Li, Yang Liu, and Sheikh Mahbub Habib. 2023. Automata-guided control-flow-sensitive fuzz driver generation. In Proceedings of the 32nd USENIX Conference on Security Symposi...
2023
-
[34]
2021.{APICraft}: Fuzz driver generation for closed-source{SDK} libraries
Cen Zhang, Xingwei Lin, Yuekang Li, Yinxing Xue, Jundong Xie, Hongxu Chen, Xinlei Ying, Jiashui Wang, and Yang Liu. 2021.{APICraft}: Fuzz driver generation for closed-source{SDK} libraries. In 30th USENIX Security Symposium (USENIX Security 21). 2811–2828
2021
-
[35]
Cen Zhang, Yaowen Zheng, Mingqiang Bai, Yeting Li, Wei Ma, Xiaofei Xie, Yuekang Li, Limin Sun, and Yang Liu. 2024. How Effective Are They? Exploring Large Language Model Based Fuzz Driver Generation. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Te...
2024
-
[36]
Hongxiang Zhang, Yuyang Rong, Yifeng He, and Hao Chen. 2024. LLAMA- FUZZ: Large Language Model Enhanced Greybox Fuzzing. arXiv preprint arXiv:2406.07714 (2024)
2024
-
[37]
error: no matching function for call to '([^']*)'
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv prep...
2023 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.