{"id":"bdc5dc8e-268f-46b6-b719-5c2c147bddf0","arxiv_id":"2507.22442","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Legion dynamically reallocates CPU cores among a set of fuzzers using a multi-armed-bandit scheduler and a five-metric seed evaluation, improving coverage and bug detection over existing ensemble fuzzing.","lead":"Legion is an ensemble fuzzing framework that reallocates CPU cores among several fuzzers each round based on how many useful new test inputs each one produced. The authors report that Legion covers more code and finds more bugs than existing ensemble fuzzing tools, including 20 real-world vulnerabilities, five of them previously unknown.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 2 as printed cannot schedule the six cores used in evaluation: line 17 empties the candidate set after one assignment, q is not an average reward, and infinite initial exploration makes softmax undefined.","rationale":"The paper's central claim is that Legion's dynamic resource scheduling (UCB-style) and multidimensional seed evaluation produce the reported gains over EnFuzz and Autofz. That claim rests on Algorithm 2 being a correct specification of the scheduler used in the 12-hour, 6-core experiments. The pseudocode fails this test in several independent ways. Most decisively, line 17 removes every fuzzer from Q during the first call to `assign`, so the loop in `schedule-resources` over six resource units cannot proceed past the first unit; the second call receives an empty Q and soft-max over an empty set is undefined. The text explicitly says a base fuzzer may run on multiple resource units simultaneously, so the printed algorithm cannot implement the evaluation setup. Second, q=γ/(γ+t) is not an average reward: γ accumulates unbounded counts of edges/paths/crashes while t is a per-round pull count, so the ratio is not a mean and has no stated statistical justification. Third, u_init=+∞ makes the soft-max probability exp(q+∞)/Σexp(...) undefined; the 'remove from candidate list' mechanism only papers over this for the first assignment. These are not disagreements with consensus; they are internal inconsistencies. A faithful re-implementation of Algorithm 2 would either crash or allocate exactly one core per fuzzer per round, contradicting the resource usage reported in Figure 7 (e.g., AFL++ receives 26% of six cores ≈ 1.56 cores in the first two hours). Therefore, the reported empirical results cannot be attributed to the described method until the artifact is released and the pseudocode corrected. The reader's CONDITIONAL verdict already requires exactly this; our stress-test strengthens the reason but does not change the requested outcome.","tokens_in":26418,"tokens_out":14983,"duration_ms":161271,"concrete_test":"Run the published Algorithm 2 verbatim on one fuzzer-test-suite project (e.g., libxml2) with 6 cores for one 600-second round and observe the scheduler. If it attempts a second `assign` call with an empty Q, or if no fuzzer ever receives more than one core, the pseudocode does not describe the evaluated system. Alternatively, if the authors release the artifact, instrument `schedule-resources` to log Q, q, u, and per-resource-unit assignments for the first round and compare against Algorithm 2; any assignment giving a fuzzer a second core in a round directly contradicts line 17 as printed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is the UCB-style resource scheduler, but Algorithm 2 is internally inconsistent and cannot be the scheduler that produced the reported results. In `assign`, line 17 (`Q←Q/{⟨f,γ,t⟩}`) executes inside the loop over Q, so after the first call to `assign` the candidate set is empty; `schedule-resources` calls `assign` once per resource unit (Lines 3–5), and with six cores the second call receives an empty Q, making soft-max undefined. The text explicitly says a base fuzzer can run on multiple resource units simultaneously, but the pseudocode assigns each fuzzer at most once per round. Additionally, q=γ/(γ+t) is not an average reward: γ accumulates counts of newly covered edges/paths/crashes while t is the previous round's pull count, so the ratio has no coherent interpretation as a mean. The initial u_init=+∞ makes exp(q+u) infinite, so the soft-max selection in the priming round is undefined. Because the claimed improvement is attributed to this scheduler, the paper does not specify the implemented algorithm; unless the artifact is released and the pseudocode corrected, the reported gains cannot be attributed to the described method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents Legion, an ensemble fuzzing framework that dynamically schedules CPU resources among base fuzzers using a UCB-style multi-armed bandit formulation and evaluates seeds with a multidimensional reward combining edge/path coverage, crash counts, deep-edge coverage, and rare-edge coverage. The framework splits a campaign into rounds, schedules fuzzers onto resource units, synchronizes seeds through a global pool, and adaptively weights the reward components by their standard deviations across fuzzers. The evaluation compares Legion variants with EnFuzz and Autofz on 24 fuzzer-test-suite subjects and with six modern base fuzzers on nine real-world projects, reporting average improvements of 10.63%/5.62% in branches, 6.44%/3.74% in paths, and 11.54%/5.45% in unique crashes over EnFuzz/Autofz, as well as 20 bugs with five previously unknown.","tokens_in":26576,"tokens_out":9368,"duration_ms":109907,"significance":"If the reported results hold, Legion is a useful engineering contribution to ensemble fuzzing: it replaces Autofz's per-round preparation phase with a low-overhead scheduler, adds fine-grained seed metrics, and demonstrates consistent gains over strong baselines across many subjects. The evaluation is extensive and careful in several respects: all experiments are repeated ten times, Mann-Whitney U tests are reported for branch and path coverage, ablation variants (NS and Cov) isolate the scheduling and evaluation components, and the real-world bug findings include developer confirmation. The main weakness is that the central scheduling algorithm is described by pseudocode that is internally inconsistent, so the paper currently does not pin down the method that produced the results.","major_comments":[{"comment":"As printed, `schedule-resources` cannot allocate six cores to six fuzzers. `assign` removes the selected tuple from `Q` at Line 17, and `schedule-resources` calls `assign` once per resource unit (Lines 3–5); after the first call the candidate set is empty (or shorter than `R`), so later soft-max selections are undefined. If `Q` is instead passed by value, Line 17 has no effect on the caller and the accompanying statement that removing from the candidate list ensures each fuzzer is assigned only once is false; the text in §3.3 also says a base fuzzer may run on multiple resource units simultaneously. In addition, `N` is incremented only in the local scope of `assign` and never returned, so the exploration term cannot use an accumulated pull count. The pseudocode must be rewritten with explicit selection-with-replacement or state-passing semantics, and the actual implementation should be released to confirm that it implements the corrected algorithm.","section":"§3.2, Algorithm 2 (Lines 3–19)"},{"comment":"q=γ/(γ+t) is labeled 'average reward' but γ is the accumulated reward over past rounds and t is the previous round pull count, so the ratio is not a mean reward; if a normalized score is intended, the text should say so. Furthermore, a fuzzer that was not scheduled in a round has t=0 and is then treated as never-deployed at Line 11, resetting its evaluation to q_init=1, u_init=∞ despite any accumulated γ; this discards historical information and contradicts the 'historical reward' claim. Finally, when more than one fuzzer has t=0 (which happens in the initial priming round), every such fuzzer has exp(q+u)=exp(∞), so the soft-max in Line 18 is undefined; the statement that the priming round selects all base fuzzers does not follow from the formula.","section":"§3.2, Algorithm 2 Lines 11–16 and 22–29"},{"comment":"The reward γ_new used for UCB updates is computed by evaluating the entire final local pool S_f, but all fuzzers are initialized each round from the same global pool and receive synchronized seeds during the round. Coverage newly discovered in S_f can therefore originate from seeds produced by another fuzzer rather than by f itself. This makes the rewards misattribute performance and the scheduler may steer on noise. The paper should describe how seed provenance is tracked (e.g., marking newly generated seeds, isolating per-fuzzer mutations) or provide an empirical analysis of the sensitivity of scheduling decisions to this attribution.","section":"§3.1 and §3.3"}],"minor_comments":[{"comment":"There are several typos and small errors: 'Micorsoft' in §2.1, 'forseed synchronization' in §3.1, 'fuzzersz' in Related Work, and the function name 'chedule-resources' in Algorithm 2 should be 'schedule-resources'.","section":"Various"},{"comment":"The caption says 'Python example' but the code is written in C syntax with `void foo(char *A, ...)`; please correct the caption or the code.","section":"Listing 1"},{"comment":"The definition of 'less frequently covered edges' as edges whose coverage count is less than 1/2 μ is ambiguous because μ is defined as the average coverage count of all covered edges; clarify whether μ is recomputed per round and how edges with zero coverage are treated.","section":"§3.3"},{"comment":"No artifact or repository link is provided. Given the central role of the scheduler and the inconsistencies in Algorithm 2, an artifact would be needed to verify that the implementation matches the described method.","section":"§4.2 and Evaluation"},{"comment":"The overhead measure for Autofz counts preparation-phase CPU hours that include running fuzzers, while Legion's overhead counts CPU hours not used to run base fuzzers; the two definitions are not directly comparable and should be stated more explicitly.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The empirical effort is substantial and the high-level idea is plausible, but the scheduler pseudocode is so internally inconsistent that the paper does not currently specify the method that was evaluated. I would not reject because the issues appear fixable in revision: the authors can rewrite Algorithm 2, clarify the reward formula and priming behavior, and release the artifact. If the implementation is released and matches a corrected algorithm, the reported gains would be verifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. The empirical side is real: 24 fuzzer-test-suite subjects, ten independent runs, Mann-Whitney U tests, and a real-world bug campaign reporting 20 bugs with five previously unknown and developer confirmations. That is a serious evaluation for an engineering paper. The second thing is that the central Algorithm 2 does not describe a working scheduler. In 'assign', the loop removes every fuzzer from the candidate set 'Q' as it processes it, so after the first call to 'assign' the set is empty. 'schedule-resources' calls 'assign' once per resource unit; with six cores, the second call gets an empty set and soft-max is undefined. The text even says a base fuzzer can run on multiple resource units, but the pseudocode prevents that. Also, q = gamma/(gamma+t) is not an average reward, and u_init = +infinity makes the soft-max formally undefined unless you treat it as a limiting case. As printed, this cannot be the scheduler that produced the reported numbers.\n\nWhat is new and worth credit: replacing Autofz's preparation phase with a UCB-style scheduler that relies on previous-round results, and a five-metric seed evaluation that goes beyond unique path coverage. The motivation is clear, and the ablation variants NS and Cov give evidence that both components matter. The reader's circularity concern is a non-issue: the theta weights are recomputed from the same data, but that is a design choice, not curve fitting.\n\nThe soft spots are serious but localized. The algorithm description must be corrected and the artifact released before the claims can be accepted. Without the artifact, the bug reports and the scheduler's actual behavior are unverifiable. The reward model may also misattribute coverage when local pools inherit global seeds, but that is a secondary concern.\n\nWho gets value? Researchers in ensemble fuzzing and automated resource scheduling. The paper deserves a serious referee because the idea is plausible and the empirical scope is substantial, but the referee should send it back for major revision: fix the pseudocode, release the artifact, and clarify the reward model. As it stands, I would not cite it as a reliable method.","headline":"Real empirical work, but Algorithm 2 as printed cannot schedule more than one core; without a corrected algorithm and artifact, the reported gains are unverifiable.","tokens_in":27203,"tokens_out":8183,"would_cite":false,"duration_ms":79067,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that Legion, an ensemble fuzzing framework that reallocates CPU cores among base fuzzers each round via a multi-armed bandit scheduler and five-metric seed evaluation, outperforms existing ensemble fuzzing in coverage…","keywords":["ensemble fuzzing","dynamic resource scheduling","multi-armed bandit","upper confidence bound","seed evaluation","coverage-guided fuzzing","vulnerability detection","fuzzer-test-suite"],"falsifier":"Track the provenance of every seed that increases coverage during a round: if a large share of a fuzzer's rewarded seeds were inherited from the global pool rather than produced by its own mutations, the reward signal misattributes credit and the scheduler is steering on noise rather than on fuzzer skill.","tokens_in":26107,"feed_emoji":"🐛","tokens_out":8056,"duration_ms":81283,"temperature":0.7,"pith_summary":"Legion is an ensemble fuzzing framework that runs several base fuzzers in parallel and, at each round, reallocates CPU cores among them based on each fuzzer's measured recent contribution. The paper claims this dynamic scheduling, modeled as a multi-armed bandit and driven by a five-metric seed evaluation, avoids the fixed allocation of EnFuzz and the profiling overhead of Autofz while finding more code and more bugs. On the fuzzer-test-suite, Legionα reports 10.63% and 5.62% more covered branches, 6.44% and 3.74% more executed paths, and 11.54% and 5.45% more unique crashes than EnFuzz and Autofzα. On nine current open-source projects, Legionβ reports 20 vulnerabilities, five previously unknown and three classified as CVEs, all five new bugs confirmed by developers. The paper's aim is to show that online, feedback-driven core assignment makes a fixed pool of fuzzers more productive than static or phase-based ensemble methods.","feed_headline":"Ensemble fuzzing with dynamic cores finds 20 bugs, 5 new","feed_subtitle":"Legion reallocates CPU cores among fuzzers each round and scores seeds on five metrics, beating fixed ensembles on coverage and crashes.","key_machinery":"The load-bearing mechanism is the round-based scheduling loop in Algorithm 1, with the UCB-style assignment in Algorithm 2 and the multidimensional seed reward of Section 3.3. Each fuzzer is an arm; assigning one CPU core for one round is one pull; the reward is the weighted sum of the five seed contributions; and the selection score combines that reward with an exploration term so that poorly explored fuzzers are tried again while proven producers get more cores. A fine-tuning step runs every half-round, stops fuzzers that stop producing beneficial seeds, and hands their cores to the current best fuzzer.","core_discovery":"The central claim is that an ensemble fuzzer can be steered online without a separate profiling phase by treating every base fuzzer as an arm of a multi-armed bandit and using its previous round's local seed pool as the reward. Legion scores each fuzzer on five metrics - new edges, new paths, crashes, deep edges, and rarely covered edges - weights the metrics by their standard deviation across fuzzers each round, adds a UCB exploration term, and assigns cores by soft-max selection. The reported result is that this scheduler makes Legionα consistently beat EnFuzz and Autofzα on branches, paths, and crashes across the fuzzer-test-suite, and makes Legionβ find 20 vulnerabilities in nine real-world projects, including every bug that the strongest single base fuzzer found and five bugs no tool had reported before.","pith_inferences":["A direct test of the reward model: log the provenance of every new-coverage seed in a round; if most come from seeds inherited from the global pool rather than mutations by the assigned fuzzer, the scheduler is crediting the wrong fuzzer and may be steering on noise.","As printed, Algorithm 2 defines $q = \\frac{\\gamma}{\\gamma+t}$, which is not an average reward, so a literal implementation of the pseudo-code would differ from the scheduler the evaluation describes; the empirical claims depend on the implemented reward update, not the formula.","The five-metric reward could serve as a general corpus-minimization criterion for parallel fuzzing, ranking seeds by long-term promise instead of immediate new coverage.","If the scheduling signal is truly meaningful, running Legion on a pool of identical fuzzers should produce near-uniform core allocation; non-uniform allocation there would indicate that the UCB exploration term, not fuzzer skill, dominates early decisions."],"forward_implications":["If the reported numbers hold, a 12-hour six-core Legionα campaign covers 409,502 branches on the fuzzer-test-suite, compared with 370,138 for EnFuzz and 387,730 for Autofzα.","Legionβ's 20 bugs in nine projects include all 15 found by AFL++, so adding dynamic scheduling on top of a state-of-the-art fuzzer still yields a 33% improvement in bug finding.","The ablation variants attribute part of the gain to each mechanism: the random-scheduling variant NS loses 13.52% of branches and 23.40% of crashes, while the coverage-only variant Cov loses 5.92% of branches and 11.54% of crashes relative to Legionα.","Legion triggers all 58 unique crashes and all crashes found by EnFuzz (52) and Autofzα (55), with no crash lost to the scheduling decisions.","Because Legion's overhead stays stable while Autofz's profiling overhead grows past 10% in later rounds, the advantage should widen on longer campaigns."],"supporting_citations":[{"why":"Defines the baseline ensemble fuzzing approach with fixed resource allocation that Legion must beat.","marker":"[12]"},{"why":"Defines the state-of-the-art round-based ensemble fuzzer with a Preparation Phase that Legion compares against and aims to improve on.","marker":"[21]"},{"why":"Supplies the fuzzer-test-suite benchmark used for the head-to-head coverage and crash comparisons.","marker":"[23]"},{"why":"Provides the upper-confidence-bound algorithm whose exploration-exploitation form Legion adapts for scheduling fuzzers.","marker":"[8]"},{"why":"Frames the resource scheduling problem as a multi-armed bandit, justifying the arm-pull analogy for core assignment.","marker":"[59]"},{"why":"Serves as the strongest single base fuzzer in Legionβ and the best single-fuzzer baseline in the real-world bug detection study.","marker":"[20]"}],"fun_headline_variants":["Legion uses bandit-based scheduling to find 20 real-world bugs","Ensemble fuzzer scores seeds five ways, uncovers 5 new bugs","Dynamic core allocation helps fuzzing ensemble net 20 bugs","Legion fuzzing framework discovers 5 novel flaws via smart scheduling","Multi-armed bandit resource sharing boosts fuzzing to 20 bugs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scheduler assumes that the seeds found in a fuzzer's local pool at the end of a round measure that fuzzer's own contribution, even though every fuzzer starts the round from the same shared global seed pool and returns its seeds to it.","fun_headline_variants_meta":{"raw":{"variants":["Legion uses bandit-based scheduling to find 20 real-world bugs","Ensemble fuzzer scores seeds five ways, uncovers 5 new bugs","Dynamic core allocation helps fuzzing ensemble net 20 bugs","Legion fuzzing framework discovers 5 novel flaws via smart scheduling","Multi-armed bandit resource sharing boosts fuzzing to 20 bugs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000462,"raw_usage":{"total_tokens":2295,"prompt_tokens":917,"completion_tokens":1378,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":1281}},"tokens_in":533,"tokens_out":1378,"duration_ms":13399,"temperature":1.0,"reasoning_tokens":1281,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:39:45.210961+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Track the provenance of every seed that increases coverage during a round: if a large share of a fuzzer's rewarded seeds were inherited from the global pool rather than produced by its own mutations, the reward signal misattributes credit and the scheduler is steering on noise rather than on fuzzer skill.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the baseline ensemble fuzzing approach with fixed resource allocation that Legion must beat."},{"cited_title":"autofz: Automated Fuzzer Composition at Runtime","cited_arxiv_id":"2302.12879","evidence_quote":"Defines the state-of-the-art round-based ensemble fuzzer with a Preparation Phase that Legion compares against and aims to improve on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the fuzzer-test-suite benchmark used for the head-to-head coverage and crash comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the upper-confidence-bound algorithm whose exploration-exploitation form Legion adapts for scheduling fuzzers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the strongest single base fuzzer in Legionβ and the best single-fuzzer baseline in the real-world bug detection study."}],"review_version":1}