{"id":"c023467c-d84a-42fc-a9c6-aff082792869","arxiv_id":"2507.18289","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Scheduzz combines LLM-extracted API usage constraints with a dual scheduling loop to automatically generate and run fuzz drivers, beating existing tools in coverage and finding 33 bugs in real libraries.","lead":"Scheduzz is a new tool that uses a large language model to generate fuzz drivers for C/C++ libraries, adding rules about how APIs should be used together and scheduling which drivers to run. In tests on 33 libraries it reported higher code coverage than several existing fuzz-driver generators and found 33 bugs, 3 with CVEs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline comparison is not a controlled head-to-head: resource budgets are self-contradictory in §4, no repeated trials are reported, and Utopia/PromptFuzz numbers may come from prior runs; the 16/21 win and 1.62×/1.50×/1.89× ratios are therefore not yet established.","rationale":"The most load-bearing assertion is the quantitative outperformance claim. My concern is not about the internal constraint solver (the reader's weakest assumption) but about the external validity of the comparison that backs the headline numbers. The reader's constraint-accuracy concern is real but partially mitigated: the paper samples 228 constraints, finds 31 false positives, and Table 4 shows removing implicit constraints lowers average coverage from 56.61% to 38.59%, indicating the constraints are net useful even with error. The comparison validity problem, however, has no internal control: a fuzzing comparison is only meaningful when baselines are rerun in the same environment with matched budgets and multiple trials. The text's own budget statements are inconsistent, and the Utopia table note is contradictory, making it impossible to rule out that the 16/21 result comes from uneven settings. A single run of a stochastic pipeline (GPT-3.5-turbo at temperature 1 plus randomized fuzzing) cannot separate signal from noise. I therefore recommend keeping the conditional verdict: the paper should be accepted only if the authors provide the artifact and reproduce the comparison under controlled, repeated conditions. I disagree with the reader's identification of LLM constraint accuracy as the single weakest assumption; it is not the most load-bearing because the design already includes empirical evidence that constraints help overall. The lack of a controlled baseline comparison is the bottleneck.","tokens_in":26094,"tokens_out":13560,"duration_ms":142163,"concrete_test":"Re-run the four baselines and Scheduzz on the shared 21/12 benchmark libraries on the same machine, with identical 24-hour, 16-core budgets, same compiler/sanitizer flags, and the same region/branch coverage instrumentation; repeat each configuration at least 5 times with different random seeds and report the median and IQR of per-library coverage. If the 16/21 win rate and the 1.62×/1.50×/1.89× aggregate ratios do not reproduce outside the noise, the headline comparative claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 (Evaluation Setup) says 'Utopia were run for 100 per-core hours on 112-core CPUs, we limit Scheduzz to 20 per-core hours on 5 cores', then states Scheduzz's campaign is '16 cores × 24 hours per library'—a different budget. This makes the resource comparison ambiguous. The paper does not state whether Utopia, PromptFuzz, CKGFuzzer, and OSS-Fuzz were rerun in the same environment or whether Table 2/3 values were taken from the original publications. The note under Table 2 ('Since Utopia does not publicly disclose its total region coverage, we leave it empty') is contradicted by the per-library Utopia coverage values shown, suggesting table construction may mix sources. All coverage curves in Figures 7-9 are single traces from a pipeline with GPT-3.5-turbo at temperature 1 and randomized fuzzing; no seeds, repetitions, or error bars are given. If baseline numbers came from different hardware, compiler, sanitizer, or fuzzing duration, the 'higher coverage in 16/21 libraries' and the aggregate ratios (1.62×, 1.50×, 1.89×) are not measurements of a difference attributable to Scheduzz. This is the load-bearing support for the central claim that the constraint/scheduling design is a practical improvement over state-of-the-art.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Scheduzz proposes an LLM-based automatic library fuzzing technique that extracts explicit type constraints via AST analysis and implicit usage constraints (imply/conflict) via LLM prompts, solves them with Prolog to obtain rational API groups, and then applies two asynchronous schedulers (group scheduler and driver scheduler) to prioritize API combinations and fuzz drivers during a fuzzing campaign. The paper evaluates Scheduzz on 21 Utopia-benchmark libraries and 12 PromptFuzz-benchmark libraries, comparing region/branch coverage against Utopia, CKGFuzzer, PromptFuzz, and OSS-Fuzz. It reports higher coverage than Utopia on 16 out of 21 libraries, overall branch-coverage improvements of 1.62x, 1.50x, and 1.89x over CKGFuzzer, PromptFuzz, and OSS-Fuzz, and 33 confirmed previously unknown bugs, 3 of which received CVEs. A component ablation shows that implicit constraints, group scheduling, and driver scheduling each contribute to coverage.","tokens_in":26398,"tokens_out":4376,"duration_ms":46348,"significance":"If the empirical results survive scrutiny, Scheduzz would be a practical advance: it formalizes API rationality as solver constraints, introduces a dual-scheduling view of driver generation and execution, and demonstrates the approach on a large set of real libraries with four baselines and a component ablation. The paper also reports concrete bug findings, including three CVEs, which is a strong external indicator. The strengths are the breadth of the benchmark, the inclusion of an ablation for each major component, and the new formulation of driver generation as an online optimization problem. However, the current evaluation has several load-bearing weaknesses that prevent the central comparative claims from being accepted as stated: inconsistent resource budgets, unstated provenance of baseline numbers, lack of repeated trials, and only partial validation of the LLM-extracted constraints. These issues are addressable by additional experiments and analysis, so the contribution is not fundamentally flawed, but the evidence base needs substantial strengthening.","major_comments":[{"comment":"The resource budgets for the comparison are internally contradictory. The text first states: 'while Utopia were run for 100 per-core hours on 112-core CPUs, we limit Scheduzz to 20 per-core hours on 5 cores,' which is 100 core-hours total, but then says 'we configure Scheduzz's fuzzing campaign to 16 cores × 24 hours per library,' which is 384 core-hours per library. These are different budgets by a factor of nearly four, and the latter is not 'stricter' than Utopia's budget. Since the paper's core claim is that Scheduzz reduces computational overhead while achieving higher coverage, the exact per-library core-hour budget for Scheduzz and for every baseline must be stated unambiguously, and the comparison should be made under equalized compute where possible.","section":"Section 4, Evaluation Setup"},{"comment":"The manuscript does not state whether Utopia, CKGFuzzer, PromptFuzz, and OSS-Fuzz were rerun in the same environment (hardware, compiler, sanitizer, fuzzing duration) or whether the numbers were taken from prior publications. The note under Table 2 — 'Since Utopia does not publicly disclose its total region coverage, we leave it empty' — is puzzling because per-library Utopia region-coverage values are shown in the same table, suggesting mixed data sources. Coverage differences are only meaningful if all tools are measured under identical conditions; otherwise the reported 16/21 win and the aggregate ratios 1.62x, 1.50x, and 1.89x are not attributable to Scheduzz's design. The paper should either rerun all baselines in a controlled environment or transparently report the exact source and conditions for each baseline number.","section":"Section 4, Tables 2 and 3"},{"comment":"All coverage curves and reported coverage numbers appear to be single-run measurements. Fuzzing is randomized and the LLM is queried with temperature 1, so any single trace can be atypical; without repeated trials, error bars, or significance testing, the claims that Scheduzz 'outperforms Utopia on 16 out of 21 libraries' and achieves '1.62x, 1.50x, and 1.89x higher coverage' are not statistically established. At minimum, the paper should report multiple independent runs per configuration (e.g., 5 runs) and provide a statistical comparison such as a paired test or a confidence interval for the per-library differences.","section":"Section 4, Figures 7–9 and Tables 2–3"},{"comment":"The implicit constraints, which are the key mechanism for ensuring 'rational' API groups, are only validated on a sample: 228 of 362 extracted constraints are inspected, and 31 are classified as false positives, with 15 of those involving non-function entities. All constraints, including the unvalidated ones, are used by the solver. Section 6 acknowledges the hallucination risk and the lack of saturation checks, but the central evaluation does not quantify how sensitive the coverage and bug results are to constraint errors. A concrete test would be to validate all extracted constraints, or to ablate the constraint set by removing low-confidence or sampled-false constraints and re-measuring coverage; otherwise the 'rationality' contribution remains a correctness risk rather than an established property of the approach.","section":"Section 4.1 and Section 6"}],"minor_comments":[{"comment":"There is a typo in 'Thechallenge 1' — it should be 'The challenge 1'.","section":"Section 1, first paragraph"},{"comment":"The prompt text contains 'contraint' instead of 'constraint' in two places; this is a presentation issue but should be fixed for clarity.","section":"Figure 4 prompt"},{"comment":"The formal notation for implication and conflict constraints is hard to parse; the arrow symbols are typeset in a way that obscures the intended meaning, and the quantifier structure of the rationality definition should be rewritten in standard notation or explained in prose.","section":"Section 2, Definition 2.3"},{"comment":"The note about Utopia's total region coverage being unavailable is contradicted by the per-library Utopia coverage values shown in the same table; please clarify which values come from public reports and which were measured or computed by the authors.","section":"Table 2 note"},{"comment":"The sentence 'Scheduzz outperforms it in 17 out of 21 libraries' refers to driver counts, while the later sentence reports 16/21 for region coverage; the two statements are easy to confuse and should be explicitly distinguished.","section":"Section 4.2, Utopia comparison"},{"comment":"The ablation table shows that the 'without GS' condition generates more drivers than the full system in many rows, which is plausible, but the paper should explain why the full system produces fewer drivers while achieving higher coverage; a sentence on the interaction between scheduling and driver count would help.","section":"Section 4.3, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising core idea and a broad evaluation, but the current empirical support for the headline claims is weakened by the budget contradiction, unclear baseline provenance, and lack of repeated trials. I believe these are fixable with additional experiments and careful reporting, so I recommend major revision rather than rejection. Please also ask the authors to state artifact availability and release scripts for reproducible baseline runs, since the comparison fairness hinges on that."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Scheduzz is a serious systems paper: the combination of type-based explicit constraints, LLM-extracted imply/conflict rules, a Prolog solver, and two feedback-driven schedulers is a real new design, and the evaluation is substantial. The authors also did the right thing in isolating components—the ablation shows implicit constraints and both schedulers each matter, with implicit constraints making the biggest difference. The bug case studies are concrete, and the limitation section is honest about hallucination and type-constraint false positives.\n\nThe main problem is the evaluation's comparability. Section 4 first says Scheduzz is limited to 20 per-core hours on 5 cores, then says the campaign is 16 cores × 24 hours per library. Those are very different budgets, and the 'stricter resource constraints' claim wobbles. The paper also never states whether Utopia, PromptFuzz, CKGFuzzer, and OSS-Fuzz were rerun in the same environment or lifted from prior publications, and the coverage traces are single runs of a pipeline using GPT-3.5-turbo at temperature 1 with no seeds or repetitions. The 16/21 win and the 1.62×/1.50×/1.89× ratios are point estimates from an unknown distribution; a skeptical reader cannot yet treat them as established.\n\nOne note on the stress test: the Table 2 remark about Utopia's total region coverage is about the aggregate row, not the per-library values, so that specific contradiction is a misread. The per-library values and the empty total are consistent with the note.\n\nThe constraint validation is a softer spot than the scheduling claims: only 228 constraints were sampled, with 31 false positives, and all constraints go into the solver. The ablation partially reassures, but the paper doesn't quantify how much the FPs distort group rationality.\n\nBottom line: this is a paper worth taking seriously. The design is coherent, the work is extensive, and the stated limitations are genuine. What's missing is a release artifact, a consistent resource budget, and repeated runs. A good referee should ask for those, not reject the idea. I'd send it to peer review and expect a conditional accept after the evaluation is tightened.","headline":"A genuinely new combination of LLM constraints and dual scheduling with a substantial but not yet airtight evaluation; deserves peer review with a required artifact and repeated runs.","tokens_in":26960,"tokens_out":3775,"would_cite":true,"duration_ms":35856,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Constraint-based API grouping beats manual and AI fuzz drivers","keywords":["fuzz driver generation","library fuzzing","LLM-based fuzzing","API constraints","Prolog constraint solving","scheduling","coverage-guided fuzzing","C/C++ libraries"],"falsifier":"Label all 362 implicit constraints by reading each library's documentation and maintainer-approved usage; if the false-positive rate among the unsampled constraints is much higher than the reported 31 of 228, or if removing all implicit constraints does not lower coverage and bug count, the rationality filter is not doing the claimed work.","tokens_in":25891,"feed_emoji":"🐛","tokens_out":5811,"duration_ms":57777,"temperature":0.7,"pith_summary":"Scheduzz claims that fuzz drivers for C/C++ libraries can be generated automatically by formalizing how library APIs are meant to be used, rather than merely collecting type-compatible API sequences. It extracts two kinds of constraints — explicit type dependencies from source code and implicit \"must call\" and \"must not call\" relationships inferred by a large language model — and solves them in Prolog to obtain rational API groups. A dual scheduler then treats driver generation and execution as one online optimization problem, selecting which API groups to turn into drivers and which drivers to keep running. If the claim holds, library fuzzing no longer depends on handcrafted drivers: the same pipeline outperforms existing tools in coverage and found 33 previously unknown bugs, three assigned CVEs.","feed_headline":"Constraint-based API grouping beats manual and AI fuzz drivers","feed_subtitle":"Automated driver generation lifts branch coverage to 53.5% and uncovers 33 bugs, 3 with CVEs.","key_machinery":"The load-bearing mechanism is the constraint pair plus dual scheduler. Explicit constraints are extracted by parsing the library's source with an AST visitor, recording each function's parameter and return types. Implicit constraints are extracted by prompting a large language model to read header files and return JSON-encoded imply and conflict relations. Both are encoded as Prolog facts; the solver's rules eliminate any API group that has an orphan function (no type dependency on another group member) or that violates an imply or conflict relation. On top sits the dual scheduler: the Group Scheduler ranks API groups by four objectives using nondominated sorting, and the Driver Scheduler scores each compiled driver by coverage over execution time and remaining energy, suspending saturated drivers. Together they turn driver generation and execution into one online optimization loop.","core_discovery":"The central claim is that rationality of API combinations, not just validity, can be captured by two constraint types and automated. Explicit constraints record shared parameter and return types so that each API in a group depends on another; implicit constraints, inferred from headers by a large language model, encode imply and conflict relations, such as \"GetUncompressedLength must precede RawUncompress.\" A Prolog solver filters all API permutations to those satisfying both constraint types, yielding rational groups. The paper then shows that scheduling both group-to-driver generation and driver execution — using coverage feedback, group similarity, group length, group entropy, and driver saturation — substantially improves fuzzing: higher region coverage than a unit-test-based tool on 16 of 21 libraries, branch coverage of 53.54% versus 33.12%, 35.69%, and 28.22% for two LLM-based tools and a handcrafted fuzzing project, and 33 new bugs with 3 CVEs.","pith_inferences":["A natural extension is to let the LLM also propose structured input layouts when the driver needs them; the paper's three underperforming libraries suggest such a component would close the gap.","The scheduling metrics could be tested in isolation: a randomized trial with different seeds or a multi-armed-bandit selector would show whether the specific Pareto ranking is essential.","If the implicit-constraint false-positive rate (31 of 228 sampled) is representative, roughly one in eight API groups is labeled rational when it is not; measuring how coverage and bug reports change as the constraint set is cleaned could quantify the ceiling of the approach."],"forward_implications":["If Scheduzz's results hold, library fuzzing can start from source code alone and still beat handcrafted drivers, removing a major human bottleneck.","Rationality filtering via implicit constraints reduces stillborn and false-positive-prone drivers, so compute time goes to useful campaigns rather than crash-heavy irrational ones.","The dual scheduling formulation makes fuzzing a library an online optimization problem; future work can apply bandit or online-learning algorithms to the same loop.","Coverage gains on well-tested libraries translate directly into vulnerability discovery: 33 new bugs, 3 with CVEs, in under 24-hour campaigns."],"supporting_citations":[{"why":"Supplies the unit-test-based baseline whose 21 libraries and region coverage are the comparison set for Scheduzz's claim of 16/21 wins.","marker":"[9]"},{"why":"Baseline LLM-based fuzz driver generator; its 12 libraries and branch-coverage numbers anchor the 1.50x coverage comparison.","marker":"[18]"},{"why":"Second LLM-based baseline; the 1.62x branch-coverage improvement is measured against it.","marker":"[32]"},{"why":"Handcrafted OSS-Fuzz drivers provide the human-written baseline that Scheduzz claims to outperform with less effort.","marker":"[21]"},{"why":"SWI-Prolog is the constraint solver that filters valid and rational API groups from the explicit and implicit constraints.","marker":"[29]"},{"why":"CEREBRO's nondominated sorting algorithm implements the multi-objective ranking in the Group Scheduler.","marker":"[16]"}],"fun_headline_variants":["Dual-scheduled fuzz drivers: 33 bugs, 3 CVEs","Constraint-based fuzz drivers beat handcrafted and AI in 33 libraries","LLM-driven constraints: 1.62x coverage gain","1.89x coverage: dual-scheduled fuzz drivers","33 new bugs, 3 CVEs: constraint-based fuzz drivers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline rests on the trust that the LLM's extracted \"must call\" and \"must not call\" constraints are mostly correct, since only 228 of 362 were manually checked and a faulty constraint makes many supposedly rational API groups irrational.","fun_headline_variants_meta":{"raw":{"variants":["Dual-scheduled fuzz drivers: 33 bugs, 3 CVEs","Constraint-based fuzz drivers beat handcrafted and AI in 33 libraries","LLM-driven constraints: 1.62x coverage gain","1.89x coverage: dual-scheduled fuzz drivers","33 new bugs, 3 CVEs: constraint-based fuzz drivers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001585,"raw_usage":{"total_tokens":6384,"prompt_tokens":1070,"completion_tokens":5314,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":686,"completion_tokens_details":{"reasoning_tokens":5219}},"tokens_in":686,"tokens_out":5314,"duration_ms":39425,"temperature":1.0,"reasoning_tokens":5219,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:15:31.003068+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Label all 362 implicit constraints by reading each library's documentation and maintainer-approved usage; if the false-positive rate among the unsampled constraints is much higher than the reported 31 of 228, or if removing all implicit constraints does not lower coverage and bug count, the rationality filter is not doing the claimed work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the unit-test-based baseline whose 21 libraries and region coverage are the comparison set for Scheduzz's claim of 16/21 wins."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Handcrafted OSS-Fuzz drivers provide the human-written baseline that Scheduzz claims to outperform with less effort."}],"review_version":2}