{"id":"65027c7d-76bb-483e-9f1a-05ef15172e24","arxiv_id":"2508.14300","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"MultiFuzz combines retrieval-augmented generation and multiple LLM agents within the ChatAFL protocol fuzzer, reporting marginal and statistically unsupported gains in branch coverage and state exploration for RTSP.","lead":"A new fuzzing system, MultiFuzz, uses a team of AI agents that read protocol documentation to generate test packets for network protocols, tested against the RTSP streaming protocol. It reports small coverage gains over existing fuzzers, but the experiments are only three runs and show inconsistent results, so the claimed advantage is not established.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim unsupported: model selection on the same benchmark, no ablation, and n=3 runs make the <3% gains over ChatAFL indistinguishable from tuning or noise.","rationale":"I agree with the reader's weakest-assumption analysis. The paper's central empirical claim requires that the observed differences be caused by the RAG/multi-agent architecture, but the evaluation does not rule out model choice, benchmark-specific tuning, or random variance. The explicit statement that model combinations were iteratively searched until the most effective model was found for each subtask is a direct admission of selection on the same benchmark. With only three runs per configuration and no significance testing, the 0.9-2.4% advantages over ChatAFL are not statistically distinguishable from noise—especially because one run showed ChatAFL outperforming MultiFuzz on branch coverage. The lack of any ablation means even the sign of the architectural effect is unverified. I also noted internal inconsistencies in the reported percentages (e.g., the text says 2.3% improvement over NSFuzz while the table's own arithmetic gives 4.7%), but the main concern is the unsupported causal attribution. Since the reader already rejected the paper and my read does not change that outcome, I recommend keeping the verdict unchanged. A revised version with pre-registered model assignments, ablations, and statistically powered multi-protocol evaluation could potentially support a conditional accept, but the current evidence does not. I found no reason to suspect fabrication; the flaws are in experimental design and statistical inference.","tokens_in":11536,"tokens_out":4708,"duration_ms":52426,"concrete_test":"Run a controlled comparison on RTSP with the same LLM backend for both systems: use the same Llama/Groq model family in ChatAFL as in MultiFuzz, fix all model assignments before the evaluation (tune on a held-out protocol/target, not on Live555), and perform at least 10 independent 24-hour sessions per configuration. Add ablations: MultiFuzz without dense retrieval, without the Seed Enrichment crew, and without the Coverage Plateau crew, holding inference budget fixed. Compare branch coverage and state transitions using bootstrapped confidence intervals or a Mann-Whitney U test. If ChatAFL with the same model matches MultiFuzz, or if any ablation matches the full system, the central attribution to RAG and multi-agent collaboration fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MultiFuzz's RAG + multi-agent architecture yields better branch coverage and state exploration than SOTA fuzzers. The evidence cannot carry this. The paper states that model assignments for each agent role were chosen by trying combinations 'iteratively until the most effective model was identified' on the same RTSP benchmark used for the reported comparison. This is selection on the test set, and no ablation isolates the dense-retrieval pipeline or any agent crew. The comparison with ChatAFL also confounds architecture with LLM: MultiFuzz uses Llama-family models via Groq-Cloud, while ChatAFL was originally built with GPT models; any gain could be model capability rather than the proposed design. The measured advantage over ChatAFL is small—branch coverage +0.9%, state transitions +2.3%, states +2.4%—and one of the three branch-coverage runs went the other way (Table I: 2910 vs 2998). With n=3 and no significance testing, the average differences are within run-to-run noise. The larger absolute improvements over AFLNet/NSFuzz are also not diagnostic, since those baselines lack LLM-guided grammar generation altogether. RQ2, which specifically asks whether multi-agent collaboration improves over single-LLM ChatAFL, is never answered by an experiment that varies only the architecture.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MultiFuzz, a dense-retrieval-based multi-agent system built on top of ChatAFL for network protocol fuzzing. It preprocesses RFC documents into propositions, groups them via agentic chunking, indexes them in a vector store, and uses three crews of LLM agents (grammar extraction, seed enrichment, coverage plateau) with a shared dense retrieval agent. The system is evaluated on RTSP/Live555 against NSFuzz, AFLNet, and ChatAFL, reporting branch coverage, number of states, and state transitions across three 24-hour runs. The paper claims that MultiFuzz significantly improves all three metrics over the baselines.","tokens_in":11820,"tokens_out":6169,"duration_ms":60902,"significance":"If the empirical claims held, the paper would make a useful contribution to LLM-guided protocol fuzzing: it combines RAG over protocol RFCs with a multi-agent decomposition of the fuzzing pipeline, and it provides concrete prompt designs and a system architecture. However, the current evidence is not sufficient to support the central claim. There are no ablations, no significance tests, only three runs on a single protocol, and the LLM model assignment was tuned on the same benchmark used for the reported comparison. The claimed 'dramatic improvements' are contradicted by the tabulated variance and by one run in which ChatAFL outperforms MultiFuzz. The contribution is therefore not yet established.","major_comments":[{"comment":"The claim of 'significant improvement' over ChatAFL is unsupported by the reported data. With n=3 runs and no significance tests, the average gains are small: branch coverage +0.9% (2940.0 vs 2912.67), state transitions +2.3% (163.33 vs 159.67), and states +2.4% (14.67 vs 14.33). In Experiment 2 of Table I, ChatAFL reaches 2998 branches while MultiFuzz reaches 2910, i.e., the baseline wins by a larger margin than the average advantage. The text in Section V.C also says MultiFuzz ranges from 2970 to 2940 branches, but the table includes 2910. At minimum, the paper needs confidence intervals or significance testing and more repetitions; the current numbers are within run-to-run noise.","section":"V.C and Table I"},{"comment":"The model assignment per agent role was chosen iteratively on the same RTSP benchmark used for the final evaluation: 'we explored different combinations of these models across the various agent groups... until the most effective model was identified for each specific subtask.' This is post-hoc selection on the test set, which inflates the reported numbers. The paper does not use a held-out protocol or a validation set, so the comparison to baselines is not a fair evaluation of the proposed system.","section":"V (Model selection)"},{"comment":"RQ2 asks whether multi-agent collaboration improves over single-LLM approaches, but the experiment does not isolate the architecture. MultiFuzz differs from ChatAFL not only in multi-agent coordination but also in the LLM provider and models (Groq-Cloud Llama-family vs ChatAFL's GPT), the retrieval-augmented context, and additional tools (CVE retrieval, packet parsing). Without an ablation that varies only the architecture (e.g., ChatAFL with the same LLM and RAG, or MultiFuzz with a single agent), the observed differences cannot be attributed to the multi-agent design.","section":"V, RQ2"},{"comment":"The general claim that MultiFuzz 'significantly improves' coverage and explores deeper states is based on a single protocol (RTSP) and a single server implementation (Live555). The paper evaluates no second protocol from ProFuzzBench, so the improvements could be specific to the characteristics of RTSP. The conclusions in Section VII overstate the generality. Either additional protocol targets or substantially qualified claims are needed.","section":"Abstract and V.A"}],"minor_comments":[{"comment":"Typo: 'it’s features' should be 'its features'.","section":"IV.H"},{"comment":"The arrow notation is ambiguous in Experiment 2: the '↓ -2.9%' in the ChatAFL column actually indicates that ChatAFL outperforms MultiFuzz, since 2998 > 2910. Use explicit win/loss indicators or state the comparison direction.","section":"Table I"},{"comment":"The text says 'All experiments were repeated multiple times' but only three runs are reported. Specify the exact number of runs and whether the means/std are over independent sessions.","section":"V.A"},{"comment":"The compatibility threshold θ is introduced but no value or sensitivity analysis is given. Since it controls chunk creation, its choice may affect retrieval quality.","section":"IV.C"},{"comment":"Figure 1 is referenced but the figure is not described; readers cannot verify the system architecture from the text alone. Also, no code or artifact availability statement is provided.","section":"Figure 1"}],"recommendation":"reject","confidential_remarks":"The paper has a plausible architecture and the writing is clear, but the experimental evaluation falls below the standard for a serious journal. The combination of test-set model selection, lack of ablations, n=3 runs, and one of three runs going against the central claim means the paper's headline result is currently indistinguishable from noise. A major rework of the evaluation with held-out protocols, architectural ablations, and statistical tests would be required before this could be considered for publication; as it stands, the central claim is unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The takeaway: the architecture is genuinely interesting, but the evaluation does not support the abstract's claim. You'd be reading this for the design ideas, not for the numbers.\n\nWhat's new: MultiFuzz decomposes the ChatAFL fuzzing pipeline into specialized agent crews (grammar extraction, seed enrichment, coverage plateau surpassing) and feeds them a shared dense-retrieval context built from RFC text that has been propositionalized and agentically chunked. That combination is not something I've seen verbatim. The authors integrated it concretely with ChatAFL and ProFuzzBench, and they report the raw numbers in full, including the run where ChatAFL beats MultiFuzz on branch coverage. That is good transparency.\n\nThe soft spots are real and load-bearing. The measured gains over ChatAFL are 0.9–2.4%, with n=3 and no significance test. One of three branch-coverage runs went the other way (2910 vs 2998), so the average difference is plausibly noise. The paper selects the LLM model per agent role 'iteratively' on the same benchmark used for the comparison; that is selection on the test set and makes the already-small effect look optimistic. There is no ablation that removes the RAG pipeline or the multi-agent decomposition, so RQ2 is never actually answered. The comparison also confounds architecture with model: MultiFuzz uses Llama variants via Groq-Cloud while ChatAFL originally used GPT, so any gain could be model capability. The larger gains over AFLNet and NSFuzz are not informative, since those baselines lack LLM-guided grammar generation entirely.\n\nNo code or data is released, and all results are on a single protocol (RTSP/Live555), so external validity is unestablished.\n\nWho is this for: researchers working on LLM-assisted protocol fuzzing might find the agentic chunking and crew decomposition worth building on, but they should treat the evaluation as a case study, not a benchmark result. A serious referee could help the authors design a statistically sound multi-protocol evaluation with ablations, fixed LLMs, and a public artifact. Right now the paper is a promising system description with an overstated evaluation.\n\nMy recommendation: do not reject on novelty or coherence—the thinking is clear and the writing is honest in the tables. But the load-bearing claim needs far stronger evidence. If you are deciding whether to engage, read it for the architecture and treat the numbers as preliminary.","headline":"The architecture is worth a look, but the evaluation is too thin to support the paper's claims—this is a promising system description, not a demonstrated result.","tokens_in":12345,"tokens_out":2287,"would_cite":false,"duration_ms":23829,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a multi-agent, retrieval-augmented fuzzer can outperform single-LLM and coverage-guided protocol fuzzers on RTSP, with higher branch coverage and deeper state exploration.","keywords":["protocol fuzzing","retrieval-augmented generation","multi-agent LLM systems","stateful protocol testing","RTSP","branch coverage","finite-state machine","LLM-guided fuzzing"],"falsifier":"Run the same 24-hour RTSP/Live555 benchmark with the retrieval and multi-agent layers disabled, keeping the same models and prompts collapsed into a single chat-style LLM call. If branch coverage and state transitions do not fall back to ChatAFL levels, the reported gain is not caused by the new machinery. A second check: rerun with models assigned randomly to agent roles; if the advantage vanishes, model selection is the cause.","tokens_in":11411,"feed_emoji":"🤖","tokens_out":6553,"duration_ms":66016,"temperature":0.7,"pith_summary":"The paper tries to show that protocol fuzzing can be made substantially more effective by giving an LLM-driven fuzzer structured access to the protocol's own specification. MultiFuzz builds on the ChatAFL pipeline and replaces its single LLM prompt with three specialized crews of agents—grammar extraction, seed enrichment, and coverage-plateau surpassing—that share a dense retrieval agent. That agent queries a vector store built from an RFC text that has been split into atomic propositions and grouped into semantic chunks. On the RTSP/Live555 target, the paper reports higher average branch coverage and more protocol states and transitions than NSFuzz, AFLNet, and ChatAFL over three 24-hour runs. A careful reader would care because stateful protocol bugs are hard to reach, and any repeatable gain in state exploration is a security-relevant result.","feed_headline":"Agentic RAG fuzzer reaches deeper RTSP states than ChatAFL and AFLNet","feed_subtitle":"MultiFuzz claims a retrieval-augmented multi-agent design beats single-LLM fuzzing on branch coverage and state transitions.","key_machinery":"The load-bearing mechanism is a retrieval-augmented multi-agent pipeline: RFC text is filtered, decomposed into atomic propositions, grouped into semantic chunks, embedded, and indexed in a dense vector store; a shared retrieval agent then supplies relevant chunks to three crews of LLM agents that generate grammars, enrich seeds, and break coverage plateaus. The retrieval context is what lets agents adhere to RTSP syntax and target states not yet covered.","core_discovery":"The central claim is that a fuzzer whose LLM components are organized as collaborating agents and grounded in retrieved protocol documentation will explore a stateful protocol's state machine more deeply than a single-LLM fuzzer or a coverage-guided fuzzer. The paper reports average branch coverage of 2940 branches for MultiFuzz versus 2912.67 for ChatAFL, 2860 for AFLNet, and 2807 for NSFuzz; 14.67 explored states versus 14.33, 10.0, and 11.7; and 163.33 state transitions versus 159.67, 84.0, and 90.33. The authors attribute the gains to the retrieval-augmented, multi-agent design, which keeps generated requests syntactically valid and semantically targeted at uncovered transitions.","pith_inferences":["The paper does not isolate which component is responsible; an ablation turning off retrieval and an ablation collapsing the three crews into one LLM call would test whether the multi-agent coordination or the document grounding matters more.","The per-role model search was performed on the same benchmark that produced the headline numbers; on a new protocol, the chosen models may not be optimal, so the reported margins may be optimistic.","RTSP is a favorable case for documentation retrieval because its RFC defines a small, explicit method set; the approach may not help as much for protocols with vague or incomplete specifications."],"forward_implications":["If the gains are causal, fuzzing a stateful protocol no longer depends on hand-written grammars: indexing the protocol's RFC gives agents the syntax and transition rules they need.","The same pipeline can be pointed at a new protocol by swapping the document store, making the approach scalable to protocols whose specifications are public but whose implementations are closed.","Coverage plateaus become addressable at runtime: an agent that reads fuzzing history and retrieves transition rules can generate a targeted packet to cross into an unexplored state.","Structured, explained outputs mean each generated request carries a record of the state it was meant to reach, which should make failures easier to triage."],"supporting_citations":[{"why":"The single-LLM protocol fuzzer that MultiFuzz extends and the primary baseline it must outperform.","marker":"[9]"},{"why":"AFLNet, the coverage-guided stateful protocol fuzzer whose architecture underlies the evaluation and appears as a baseline.","marker":"[30]"},{"why":"NSFuzz, a state-aware protocol fuzzer used as a baseline in the branch and state comparisons.","marker":"[31]"},{"why":"ProFuzzBench, the benchmark platform that defines the RTSP/Live555 target and the experimental setup.","marker":"[40]"},{"why":"Retrieval-augmented generation, the method the system adapts to make LLM output depend on retrieved protocol documentation.","marker":"[11]"},{"why":"ReAct, the reasoning-and-acting pattern that informs the chain-of-thought agent collaboration.","marker":"[12]"},{"why":"Dense retrieval, the retrieval type chosen for indexing and querying the protocol chunks.","marker":"[10]"},{"why":"National Vulnerability Database API, which supplies CVE context to the coverage-plateau crew for Live555.","marker":"[36]"}],"fun_headline_variants":["Multi-agent RAG fuzzer beats single-LLM fuzzers on RTSP state depth","Dense retrieval fuzzer surpasses ChatAFL in protocol state coverage","MultiFuzz: agents plus retrieval explore deeper RTSP than AFLNet","Retrieval-augmented multi-agent fuzzer improves RTSP branch coverage","LLM fuzzer with agentic RAG outpaces ChatAFL and AFLNet on RTSP"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the measured difference comes from the retrieval and multi-agent design, rather than from run-to-run variability or from picking the most effective LLM for each role after seeing the results.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent RAG fuzzer beats single-LLM fuzzers on RTSP state depth","Dense retrieval fuzzer surpasses ChatAFL in protocol state coverage","MultiFuzz: agents plus retrieval explore deeper RTSP than AFLNet","Retrieval-augmented multi-agent fuzzer improves RTSP branch coverage","LLM fuzzer with agentic RAG outpaces ChatAFL and AFLNet on RTSP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1348,"prompt_tokens":888,"completion_tokens":460,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":348}},"tokens_in":632,"tokens_out":460,"duration_ms":4865,"temperature":1.0,"reasoning_tokens":348,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:38:04.868234+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 24-hour RTSP/Live555 benchmark with the retrieval and multi-agent layers disabled, keeping the same models and prompts collapsed into a single chat-style LLM call. If branch coverage and state transitions do not fall back to ChatAFL levels, the reported gain is not caused by the new machinery. A second check: rerun with models assigned randomly to agent roles; if the advantage vanishes, model selection is the cause.","supporting_citations":[],"review_version":1}