{"id":"3348a50a-7fce-4ee7-8f59-29e090ea632c","arxiv_id":"2507.10845","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A bandit-based scheduler that picks which of ten fuzzers to run next on one core improves coverage and mutation scores over individual fuzzers and over autofz's greedy scheduler.","lead":"BandFuzz combines ten existing fuzzers into one scheduler that uses a Thompson-sampling bandit to decide, on a single CPU core, which fuzzer to run at each time step. It reports higher code coverage than the individual fuzzers and than the autofz collaborative framework, and it won the mutation-scoring track of SBFT 2024.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline outperformance over AFL++ is within FuzzBench's own 1% noise threshold: BandFuzz 95.26 vs AFL++ 94.46 (Δ=0.80), which Table 1 itself treats as random; the central claim needs significance testing.","rationale":"The reader's weakest_assumption targets the reward metric in Algorithm 1; I agree that is a genuine weakness, but the more load-bearing failure is that the headline experiment does not establish a significant improvement over AFL++ even under the paper's own criterion. A biased reward would invalidate the mechanism claim, yet the empirical outperformance claim could in principle survive a biased reward; the significance issue directly attacks the primary claim. The paper's own Table 1 note sets a 1% random-effect threshold, and the observed BandFuzz-vs-AFL++ gap is 0.80 points, below that threshold. No paired significance test or confidence interval is provided for the main FuzzBench comparison, and the FTS/autofz comparison similarly lacks inferential statistics despite large per-target variances. The SBFT 2024 mutation-score result is meaningful independent evidence, but it uses a different metric and a contest setting, so it does not rescue the FuzzBench coverage claim. I therefore recommend keeping the reader's CONDITIONAL verdict, with conditions that the authors report per-target significance and effect sizes for Tables 1-2 and release the implementation for independent verification.","tokens_in":26267,"tokens_out":7837,"duration_ms":91066,"concrete_test":"Recompute the FuzzBench comparison from the 10 trials behind Table 1: for each of the 21 targets, form paired per-target median branch-coverage values for BandFuzz and AFL++ across the 10 runs, then apply a Wilcoxon signed-rank test and a bootstrap 95% CI on the mean score difference. If p > 0.05 or the CI includes values below FuzzBench's 1% random-effect threshold, the abstract's \"outperforms widely used individual fuzzers\" claim is unsupported and the verdict should require re-analysis before acceptance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that BandFuzz \"outperforms widely used individual fuzzers\" (Abstract, §1) is supported on FuzzBench by an average score of 95.26 vs 94.46 for AFL++ (Table 1). The paper's own Table 1 note states that \"any difference less than 1% is considered as a random effect and thus disregarded.\" The observed gap is 0.80 points—below that threshold—so by the authors' criterion the headline edge over the strongest individual fuzzer is within noise and should be disregarded. No per-target significance test, paired test, or confidence interval is reported for Experiment I, so the claim that BandFuzz outperforms individual fuzzers is not statistically established by the central experiment. The FTS/autofz comparison (Table 2) also lacks significance tests; several rows have very large standard deviations (e.g., sqlite-2016-11-14: 1981.0±112.8; proj4-2017-08-14 autofz-8: 1015.1±595.9), so the reported 16.0% average enhancement may be driven by a few targets. The reward-bias issue identified by the reader (Algorithm 1, Lines 4-6) is real and compounds this: c = t - t_p depends on the round in which the predecessor block was first discovered, which is set by the bandit's own past selections; with seed synchronization, a fuzzer selected after a long gap can trivially traverse pending branches and receive large rewards, so the reward does not cleanly measure fuzzer quality or branch difficulty. Therefore neither the empirical superiority nor the claimed \"globally optimal\" learning mechanism is currently established. Conditional acceptance with mandatory significance testing and code release is the right bar.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BandFuzz, a collaborative fuzzing framework that selects one of ten individual fuzzers to run on a single CPU core at each round, using a Thompson-sampling multi-armed bandit to allocate resources. A global seed pool synchronizes discoveries among fuzzers, and a novel reward signal—based on the round interval between a branch's discovery and its predecessor block's first discovery—is used to update each fuzzer's Beta distribution. The paper evaluates BandFuzz on 21 FuzzBench targets and 25 Fuzzer Test Suite targets with 10 runs of 24 hours each, compares it to individual fuzzers and autofz, performs ablations for seed synchronization, reward design, and bandit allocation, tests hyperparameter sensitivity, and reports a first-place finish in the SBFT 2024 fuzzing competition with an average mutation score of 98.","tokens_in":26661,"tokens_out":5834,"duration_ms":67336,"significance":"If the empirical claims are statistically sound, BandFuzz would be a valuable contribution: it is a collaborative fuzzer that requires no additional CPU resources over a single fuzzer, reports strong results on public benchmarks, and has a competition victory as external validation. The paper's experimental scale is substantial (approximately 7.5 CPU-years of computation), it uses public benchmarks (FuzzBench, FTS), each experiment is repeated 10 times, and it includes a systematic ablation study and hyperparameter sensitivity analysis. The proposed coverage-interval reward is a creative departure from simple coverage or uniqueness metrics. However, the headline FuzzBench advantage over AFL++ is within the 1% threshold that the paper's own benchmark note says should be treated as a random effect, and no significance tests are provided for the central comparisons. In addition, the reward definition in Algorithm 1 is entangled with the bandit's own scheduling history, which threatens the validity of the learning signal and the claimed global optimality. These issues are load-bearing and require revision before the central claims can be accepted.","major_comments":[{"comment":"The paper's own benchmark note states that any difference less than 1% is considered a random effect and disregarded. The average FuzzBench score for BandFuzz is 95.26 versus 94.46 for AFL++, a difference of 0.80 percentage points, which is below that threshold. No paired significance test, confidence interval, or effect size is reported for the per-target scores. The claim that BandFuzz 'outperforms widely used individual fuzzers' is therefore not statistically established by the main FuzzBench experiment. Please report a paired test (e.g., Wilcoxon signed-rank on per-target scores) or a bootstrap confidence interval on the average score, and explicitly address how the result stands relative to the 1% random-effect threshold.","section":"§5.3, Table 1"},{"comment":"The reported 16.0% average coverage enhancement of BandFuzz-8 over autofz-8 is not accompanied by any significance testing. Several rows show very large standard deviations (e.g., proj4-2017-08-14: autofz-8 1015.1±595.9; sqlite-2016-11-14: BandFuzz-8 1981.0±112.8), so the average enhancement may be driven by a few targets. Please provide a paired test across the 25 targets, per-target confidence intervals, and clarify whether the 16.0% is the mean of per-target relative improvements or a mean of absolute coverage differences. Without this, the superiority of BandFuzz's collaboration strategy over the greedy autofz baseline is not statistically supported.","section":"§5.3, Table 2"},{"comment":"The reward for a new branch is computed as c = t − t_p, where t_p is the round in which the predecessor basic block was first discovered. This t_p is determined by which fuzzer happened to discover that predecessor and by the bandit's own scheduling; when rounds are skipped or seeds are synchronized from the global pool, a fuzzer can traverse pending branches and receive a large c without having worked harder than a fuzzer that follows immediately after a discovery. The reward is therefore confounded with the scheduler's history, which corrupts the bandit's updates and undermines the claim that the algorithm learns 'globally optimal' strategies. Please either replace the reward with a scheduler-independent measure (e.g., actual CPU time or number of fuzzer cycles since the predecessor was discovered) or provide an analysis demonstrating that the bias does not affect the learning outcome, along with a sensitivity test using an alternative reward definition.","section":"§3.2, Algorithm 1 (lines 4-6)"},{"comment":"The paper repeatedly claims that BandFuzz 'enables discovery of globally optimal collaborative strategies', but no theoretical guarantee is provided. The customized Thompson sampling uses a Bernoulli discretization of a continuous reward, periodic parameter resets, and an auto-cycle mechanism; none of these is covered by standard Thompson sampling regret bounds. The term 'globally optimal' is not supported by any theorem or convergence result. Please either supply a formal optimality or regret analysis under a clearly stated non-stationary reward model, or soften the claim to describe the method as an effective heuristic strategy with empirical support.","section":"Abstract, §1, §3.3, §9"}],"minor_comments":[{"comment":"The arithmetic in the worked reward example is inconsistent: the text first states that branch b3 has coverage interval c = IZ − IX, but the final reward sum uses (IZ − IY) for that branch. The total should be 2(IZ − IX) if both b3 and b8 use IZ − IX, or the text should justify why b3 uses IZ − IY. Please correct the example.","section":"§3.2, worked example"},{"comment":"Equation (1) updates β with subscript Fk: 'β_Fk ← β_Fk + 1 − r_t'. This should be β_Ft, consistent with Algorithm 2 line 16. The current subscript makes the update ambiguous.","section":"§3.3, Equation (1)"},{"comment":"Both captions say 'Comparison among BAND FUZZ and two variants', but Figure 4 shows four curves (BandFuzz, random, no-sync, naive-reward) and Figure 5 appears to show three. Please adjust the captions to match the number of configurations displayed.","section":"Figures 4 and 5 captions"},{"comment":"The text states that the second-ranked fuzzer is TuneFuzz [66], but Table 3 lists FishFuzz and reference [66] is FishFuzz. Please align the name used in the text with the table and reference.","section":"§6, Table 3"},{"comment":"The text says 'we run each method 10 times and report the mean', but FuzzBench scores are based on median branch coverage relative to the maximum per target. Please clarify which per-target statistic is used (median over runs) and how the average score is computed.","section":"§5.2, Experiment I"},{"comment":"The Nemenyi post-hoc test is mentioned, but no p-values or critical difference values are reported in the text; only a CD diagram is referenced. Please report the actual statistical results or specify where they can be found, so that 'marginal changes' is supported by the data.","section":"§5.3, Experiment V"},{"comment":"The paper does not state an artifact availability plan or provide a link to source code and benchmark configuration. For a systems paper with such large experiments, releasing the code and exact build scripts would substantially aid reproducibility and is recommended.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The external SBFT 2024 competition result is a strong point in favor of the work, but the FuzzBench threshold issue is particularly concerning because the paper itself reproduces the note that <1% differences are random effects and then claims superiority over AFL++ on a 0.80-point gap. The reward-bias problem in Algorithm 1 is not merely a theoretical quibble; it affects the interpretation of every reported learning outcome. I would ask the authors to conduct a proper statistical reanalysis for Experiments I and II and to rerun at least one ablation or a subset of targets with an alternative reward definition. Also, the discrepancy between 'TuneFuzz' and 'FishFuzz' in the competition section, together with the caption errors, suggests the manuscript needs a careful proofreading pass."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a genuine engineering contribution, and the core idea is new to collaborative fuzzing — treating fuzzer selection as a bandit problem with Thompson sampling and a coverage-interval reward, all on a single core. That framing, plus the seed-sync and auto-cycle machinery, is worth the community's attention. But the abstract's claim that BandFuzz \"outperforms widely used individual fuzzers\" is not actually established by the FuzzBench experiment. The gap over AFL++ is 95.26 vs 94.46, and the paper's own Table 1 says any difference under 1% is random. The stress-test note is right on this: the headline edge is inside the noise threshold, and no significance test or confidence interval is reported.\n\nWhat the paper does well: the experimental campaign is large — 24h runs, 10 repetitions, 21 FuzzBench targets plus 25 FTS targets, about 7.5 CPU years. The ablation shows each of the three components (sync, reward, bandit allocation) contributes, and the 16% improvement over autofz-8 on FTS is substantial, even without formal significance testing. The SBFT 2024 mutation track win is external evidence and should not be discounted just because the paper analyzes its own result.\n\nSoft spots, in proportion. The FTS comparison lacks paired tests, and a few rows have very large standard deviations (sqlite, proj4), so the 16% average could be pulled by outliers — though 19-of-25 wins is a reasonable counterweight. The reward-feedback concern is real: the coverage interval c = t - t_p depends on when the predecessor block was first discovered, which the bandit itself influences. With seed synchronization, a fuzzer selected after a long gap can trivially traverse pending branches and collect inflated rewards. So the \"globally optimal\" language in the paper is not earned; the mechanism is a plausible heuristic, not a proven optimizer. The paper does not analyze this contamination. No code is released, which matters for a system paper whose value is in the engineering.\n\nI'd send this to peer review. The bandit-at-the-scheduler level idea is novel, the autofz comparison is meaningful, and the competition result gives real-world anchor. But it needs mandatory significance testing on both FuzzBench and FTS, an analysis of the reward bias or a change to the metric, and code release. Without those, the central claims stay unverified. A serious referee should demand those rather than desk-reject.","headline":"A solid system paper with a real bandit-based scheduling idea; the headline outperformance over AFL++ is inside its own noise threshold, but the autofz comparison and ablation make it worth a serious referee.","tokens_in":27220,"tokens_out":1892,"would_cite":true,"duration_ms":24540,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Thompson-sampling bandit that treats each fuzzer as an arm lets one CPU core match or beat a team of fuzzers.","keywords":["collaborative fuzzing","multi-armed bandits","Thompson sampling","resource allocation","fuzzer evaluation","coverage interval","seed synchronization","mutation score"],"falsifier":"Run a controlled comparison in which the same target, same fuzzers, and same seed pool are scheduled by BandFuzz but the coverage-interval reward in Algorithm 1 is computed with a scheduler-independent round counter (for example, actual execution time or cycles spent since the predecessor was found) instead of the bandit's round number; if total branch coverage or the chosen fuzzer sequence changes materially, the reward is not measuring branch difficulty alone and the global-optimality claim is scheduling-dependent. A simpler check: count rounds where a selected fuzzer produces no new branches — those skipped rounds inflate $c$ for any later discovery — and ask whether an easy branch discovered after a long drought earns the same inflated reward as a genuinely hard branch.","tokens_in":26077,"feed_emoji":"🧪","tokens_out":6244,"duration_ms":64929,"temperature":0.7,"pith_summary":"The paper is trying to establish that collaborative fuzzing — running several fuzzers as a team — can beat both individual fuzzers and prior ensembles without consuming extra CPU cores. The mechanism is a multi-armed bandit: each fuzzer is an arm, and a Thompson-sampling rule decides, round by round, which fuzzer gets to run on the single core. The reward that drives learning is a new difficulty-aware coverage metric, the coverage interval $c = t - t_p$ for each newly covered branch. The paper argues that this long-horizon, exploration-aware allocation is globally better than the greedy allocation used by autofz, and reports that BandFuzz beats ten individual fuzzers on FuzzBench, autofz-8 by 16.0% average branch coverage on FTS, and all competitors in the SBFT 2024 mutation-score ranking. A sympathetic reader would care because fuzzing progress has stagnated on standard benchmarks, and this is a concrete proposal for getting ensemble robustness from a single core.","feed_headline":"BandFuzz beats greedy ensembles with a learning scheduler on one core","feed_subtitle":"A Thompson-sampling bandit hands time to the fuzzer that earns it, lifting coverage 16% over autofz.","key_machinery":"The load-bearing object is the coverage interval $c = t - t_p$: the number of fuzzing rounds between the discovery of a branch's predecessor basic block and the discovery of the branch itself, used as a proxy for branch difficulty and as the per-branch reward. Around it, the machinery is a customized Thompson-sampling bandit in which each fuzzer is an arm with a $\\mathrm{Beta}(\\alpha_F, \\beta_F)$ weight distribution; each round the algorithm samples a success probability per arm, runs the fuzzer with the highest sample, converts the normalized coverage-interval reward into a Bernoulli observation, updates that arm's Beta parameters, and periodically resets all parameters so the bandit can track non-stationary reward distributions. The auto-cycle mechanism and real-time global seed synchronization keep the reward comparisons fair by giving each selected fuzzer roughly equal wall-clock time and by preventing fuzzers from re-exploring branches already in the global pool.","core_discovery":"On its own terms, the paper's claim is that a single-core collaborative fuzzer can learn, online, which fuzzer deserves the next slice of time, and that this learned schedule outperforms both fixed single-fuzzer runs and the greedy exploration-exploitation schedule of autofz. BandFuzz integrates ten fuzzers, maintains a global seed pool synchronized in real time with the selected fuzzer, and evaluates the selected fuzzer not by raw coverage but by the summed coverage intervals $c = t - t_p$ of the new branches its seeds cover, where $t_p$ is the round in which the branch's predecessor basic block was first discovered. These rewards are normalized, discretized into Bernoulli samples, and fed to Thompson sampling with periodic parameter resets; an auto-cycle mechanism converts a fixed time budget into a fair number of fuzzing cycles per fuzzer. The reported evidence: highest median and average scores among all integrated fuzzers on 21 FuzzBench targets; 16.0% higher average branch coverage than autofz-8 over 25 FTS targets and 4.8% over full autofz; and an average mutation score of 98, first place, at SBFT 2024.","pith_inferences":["If the coverage-interval reward is confounded by the scheduler — because $t_p$ was set by whichever fuzzer discovered the predecessor and skipped rounds inflate $c$ even for easy branches — then the bandit's updates may partly reward scheduling artifacts rather than fuzzer skill; a testable extension would recompute rewards under a scheduler-independent round counter and compare coverage.","The same reward signal could be lifted out of the collaborative setting and used inside a single fuzzer to prioritize mutation effort toward inputs that unlock branches whose predecessors have been known for many rounds.","Adversarial bandit algorithms designed for non-stationary rewards, such as the EXP3 variant the paper names as future work, are a natural next comparison since the reset mechanism is a heuristic for non-stationarity rather than a formal treatment.","If branch difficulty is the real quantity being measured, the coverage interval could be replaced by or combined with structural difficulty measures such as basic-block centrality or condition complexity; the paper lists these as future seed-quality metrics."],"forward_implications":["Collaborative fuzzing no longer needs parallel cores: the same single CPU that would run one fuzzer can run a team, so ensemble robustness is available on resource-constrained systems.","Greedy resource allocation is not enough: the reported 16.0% average coverage gain over autofz-8 is attributed to the bandit's ability to favor fuzzers whose payoff is delayed.","The framework can be extended to new fuzzers with only minor modifications to their underlying harnesses, since the bandit treats each fuzzer as an opaque arm.","A fuzzer that crashes or stalls does not derail the campaign: management mechanisms skip or restart it, and the bandit shifts weight to working fuzzers.","Difficulty-aware evaluation can be reused independent of the bandit, as a real-time seed-quality or fuzzer-quality score."],"supporting_citations":[{"why":"The greedy collaborative framework that BandFuzz must beat; supplies the baseline for the 16.0% average coverage comparison.","marker":"[32]"},{"why":"The original Thompson sampling algorithm that the customized bandit adapts.","marker":"[57]"},{"why":"Supplies the reward-discretization step that lets Thompson sampling accept continuous rewards; without it the customized bandit cannot update.","marker":"[61]"},{"why":"FuzzBench benchmark and its average-score metric used for the individual-fuzzer comparison.","marker":"[33]"},{"why":"Fuzzer Test Suite benchmark used for the autofz comparison.","marker":"[34]"},{"why":"SBFT 2024 competition whose mutation-analysis results supply the first-place evidence for real-world bug detection.","marker":"[68]"},{"why":"Defines mutation score as the evaluation metric for the competition; load-bearing for the reported average mutation score of 98.","marker":"[69]"},{"why":"Representative ensemble fuzzer that runs fuzzers in parallel; its extra-resource cost is the problem BandFuzz is designed to remove.","marker":"[31]"}],"fun_headline_variants":["Bandit-based fuzzing beat greedy autofz on one core","ML scheduler allocates fuzzer time, wins SBFT 2024","Collaborative fuzzing with bandit rewards, no extra resources","Thompson sampling fuzzer schedule outperforms autofz"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the number of rounds between the first discovery of a branch's predecessor block and the discovery of the branch itself measures how difficult and how valuable that branch is, independently of the bandit's own scheduling decisions.","fun_headline_variants_meta":{"raw":{"variants":["Bandit-based fuzzing beat greedy autofz on one core","ML scheduler allocates fuzzer time, wins SBFT 2024","Collaborative fuzzing with bandit rewards, no extra resources","Thompson sampling fuzzer schedule outperforms autofz"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1460,"prompt_tokens":1088,"completion_tokens":372,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":704,"completion_tokens_details":{"reasoning_tokens":296}},"tokens_in":704,"tokens_out":372,"duration_ms":4773,"temperature":1.0,"reasoning_tokens":296,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:24:03.003028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled comparison in which the same target, same fuzzers, and same seed pool are scheduled by BandFuzz but the coverage-interval reward in Algorithm 1 is computed with a scheduler-independent round counter (for example, actual execution time or cycles spent since the predecessor was found) instead of the bandit's round number; if total branch coverage or the chosen fuzzer sequence changes materially, the reward is not measuring branch difficulty alone and the global-optimality claim is scheduling-dependent. A simpler check: count rounds where a selected fuzzer produces no new branches — those skipped rounds inflate $c$ for any later discovery — and ask whether an easy branch discovered after a long drought earns the same inflated reward as a genuinely hard branch.","supporting_citations":[{"cited_title":"autofz: Automated fuzzer composition at runtime,","cited_arxiv_id":null,"evidence_quote":"The greedy collaborative framework that BandFuzz must beat; supplies the baseline for the 16.0% average coverage comparison."},{"cited_title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples,","cited_arxiv_id":null,"evidence_quote":"The original Thompson sampling algorithm that the customized bandit adapts."},{"cited_title":"Analysis of thompson sampling for the multi-armed bandit problem,","cited_arxiv_id":null,"evidence_quote":"Supplies the reward-discretization step that lets Thompson sampling accept continuous rewards; without it the customized bandit cannot update."},{"cited_title":"Fuzzbench: an open fuzzer benchmarking platform and service,","cited_arxiv_id":null,"evidence_quote":"FuzzBench benchmark and its average-score metric used for the individual-fuzzer comparison."},{"cited_title":"Fuzzer test suite (fts),","cited_arxiv_id":null,"evidence_quote":"Fuzzer Test Suite benchmark used for the autofz comparison."},{"cited_title":"Sbft 2024 fuzzing competition,","cited_arxiv_id":null,"evidence_quote":"SBFT 2024 competition whose mutation-analysis results supply the first-place evidence for real-world bug detection."},{"cited_title":"Systematic assessment of fuzzers using mutation anal- ysis,","cited_arxiv_id":null,"evidence_quote":"Defines mutation score as the evaluation metric for the competition; load-bearing for the reported average mutation score of 98."},{"cited_title":"Enfuzz: Ensemble fuzzing with seed synchronization among diverse fuzzers,","cited_arxiv_id":null,"evidence_quote":"Representative ensemble fuzzer that runs fuzzers in parallel; its extra-resource cost is the problem BandFuzz is designed to remove."}],"review_version":1}