{"id":"8aa79b98-b619-45d4-9f4d-b94dfa7dc814","arxiv_id":"2502.15727","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A RAG-plus-chain-of-thought agent improves BLEU, ROUGE, and WER scores for RTSP packet seed generation over Gemma-2-9B and Llama-3-8B baselines, though the evaluation does not test actual fuzzing effectiveness.","lead":"This paper tests whether giving a large language model access to the RTSP protocol specification, via retrieval-augmented generation and chain-of-thought reasoning, produces better fuzzing seed packets than the base model alone. The approach improved text-similarity scores on a small set of RTSP requests, but no fuzzing was run to show it actually finds more vulnerabilities.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing issue is that BLEU/ROUGE/WER similarity to ChatAFL logs is used as a proxy for fuzzing seed quality; without a fuzzing experiment, the claimed benefit to protocol fuzzing does not follow.","rationale":"The paper's numerical claim that RAG-ReAct agents improve BLEU, ROUGE, and WER over base LLMs is internally consistent with the tables, and the reported trends are plausible. However, the strongest claim in the abstract and conclusion goes beyond text similarity: it asserts improvement to protocol fuzzing frameworks and vulnerability discovery. That assertion depends entirely on treating BLEU/ROUGE/WER similarity to ChatAFL logs as a proxy for seed quality, and no fuzzing experiment is reported. The reader's weakest_assumption identifies exactly this gap, and I agree it is the most load-bearing concern. The paper should either supply direct fuzzing evidence or clearly restrict its claims to text-similarity metrics. Since the missing evidence is obtainable and the core numerical results are not in question, the appropriate verdict remains CONDITIONAL rather than ACCEPT or REJECT, so I do not change the reader's verdict.","tokens_in":6611,"tokens_out":6120,"duration_ms":64387,"concrete_test":"Run a controlled fuzzing experiment for each model: feed baseline-generated seeds and RAG-ReAct-generated seeds separately into the same stateful fuzzer (e.g., ChatAFL or AFLNet) against live555 for equal time and multiple trials, and record state/transition coverage, unique crashes, and packet-acceptance rate by a strict RTSP parser. Then correlate per-seed BLEU/ROUGE/WER scores with these fuzzing outcome metrics. If high-text-similarity seeds do not reliably yield higher coverage or more crashes, the proxy in Sections 5-6 is invalid and the paper's fuzzing claim must be withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 defines ground truth as logs captured from ChatAFL on live555, and Section 6 evaluates RAG-ReAct outputs against those logs with BLEU/ROUGE/WER. The central claim of improving LLM-based protocol fuzzing frameworks requires that higher n-gram and word-overlap with ChatAFL's logs implies seeds that induce better state-space exploration. That implication is not established and is questionable for two reasons. First, these metrics measure surface text resemblance, not protocol conformance: an output can score high BLEU by echoing a common RTSP template while violating required header or state-transition constraints, or score low BLEU while being a valid and useful seed. Second, the ground truth is itself generated by an LLM-guided fuzzer on the same task of enriched seed combinations, so high similarity may reflect stylistic mimicry of ChatAFL's LLM outputs rather than fuzz-relevant quality. No fuzzing run, coverage measurement, or packet-acceptance check is reported; Section 7 even lists integration into fuzzing frameworks as future work. Thus the causal chain from metric improvements to better vulnerability discovery is the weakest load-bearing assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RAG-ReAct, an agentic LLM pipeline that retrieves protocol specification chunks (RFC-2326 for RTSP) from a vector store and uses chain-of-thought reasoning to generate network packet seeds for protocol fuzzing. It compares Gemma-2-9B and Llama-3-8B in baseline and RAG-ReAct settings, evaluating the generated packets with BLEU, ROUGE, and WER against 120 ground-truth entries drawn from ChatAFL fuzzing logs on the live555 server. The paper reports average score improvements for both models and concludes that the approach improves LLM-based protocol fuzzing frameworks and helps identify hidden vulnerabilities.","tokens_in":6862,"tokens_out":5978,"duration_ms":54168,"significance":"If the reported improvements reflected fuzzing-relevant packet quality, the work would offer a lightweight, fine-tuning-free method for seed enrichment that could benefit protocol fuzzing practice. The technical construction (RAG + ReAct + COT) is reasonable, the paper compares two representative LLMs, and the tables are clearly presented. However, the evaluation measures surface text similarity to ChatAFL logs rather than protocol conformance, code coverage, or vulnerability discovery, and the ground truth shares the same underlying specification source as the agent's retrieval base. As a result, the central claim about improving protocol fuzzing frameworks is currently unsupported. The paper is a useful preliminary study but needs a downstream fuzzing evaluation or a validated proxy for seed quality.","major_comments":[{"comment":"The evaluation treats BLEU/ROUGE/WER similarity to ChatAFL logs as a proxy for seed quality. This is load-bearing for the abstract's claim of 'improving LLM-based protocol fuzzing frameworks', yet no fuzzing run, coverage measurement, packet-acceptance check, or vulnerability discovery is reported, and Section 7 lists integration into fuzzing frameworks as future work. Text similarity can be high for a packet that violates RTSP state transitions or headers, and low for a valid and useful seed. Please add a downstream fuzzing experiment or at least a validity test showing that higher scores correlate with better state-space exploration.","section":"Section 6 (Tables 1 and 2)"},{"comment":"The ground-truth packets are logs captured from ChatAFL, an LLM-guided fuzzer, running against live555, and ChatAFL's generation is based on protocol specifications including the same RFC-2326 that the RAG knowledge base retrieves from. The comparison therefore may reward mimicry of ChatAFL's output style rather than protocol compliance or fuzz-relevant quality. Independent ground truth (e.g., manually validated RTSP messages or real protocol traces) or a calibrated metric is needed to support the conclusions.","section":"Section 5 (Experimental Design)"},{"comment":"The reported 'up to' improvements are not as stated. The values 18.19%, 14.81%, and 23.45% are the means over the two models of the absolute percentage-point changes (with the WER value being a reduction, not an increase), so describing them as 'improvements of up to X% in WER' is misleading. Moreover, no error bars, confidence intervals, significance tests, or per-request sample sizes are given; Tables 1 and 2 show single numbers per condition, so the claim of 'consistent improvements' is not supported statistically. Please report the number of ground-truth queries per request type and repeat the generation with diverse sampling to quantify variance.","section":"Abstract and Section 6"},{"comment":"The paper is framed around protocol state machine inference, but the evaluation never checks whether the generated packet sequences correspond to valid FSM transitions from RFC-2326. The metrics only compare token overlap with ChatAFL logs, not whether the agent's state-machine reasoning is correct. A direct evaluation of FSM accuracy (e.g., validating state sequences against the RFC's transition table) or a reformulation of the paper's scope as seed text generation would be needed.","section":"Title and Sections 1 and 6"}],"minor_comments":[{"comment":"The phrase 'The protocol FSM describes defines how' appears to contain an error; it should read 'describes/defines how'.","section":"Section 3"},{"comment":"The abbreviation 'ARPE' should be 'APRE' (automatic protocol reverse engineering) in the sentence about integrating LLMs with existing techniques.","section":"Section 1"},{"comment":"The table headings read 'T able' instead of 'Table', and values such as '08.71%' should be formatted as '8.71%'.","section":"Tables 1 and 2"},{"comment":"The relationship between the '5000 plus entries' log dataset and the 'typically 120 entries' evaluation set is unclear; please state how the 120 queries were sampled and whether each request type is equally represented.","section":"Section 5"},{"comment":"The prompt templates are shown in a figure but not described in the text; a brief summary of the three prompt sections would improve reproducibility.","section":"Figure 3"},{"comment":"The hyperparameters chunk_size=1000, overlap=200, and retrieval top-k are given without justification or sensitivity analysis; adding an ablation would strengthen the methodology.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is borderline for a security/network venue because it does not evaluate fuzzing outcomes despite claiming to improve fuzzing frameworks. The abstract's numerical claims should be corrected before any acceptance. A revised version that includes a small fuzzing experiment or a validated proxy for packet quality, plus corrected statistics, would be considerably stronger."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper documents a plausible empirical result — a RAG-ReAct agent with chain-of-thought prompting gets higher BLEU/ROUGE/WER on RTSP packet texts than base LLMs — but it does not establish the advertised benefit to protocol fuzzing, because the metrics are text similarity to ChatAFL logs, not fuzzing effectiveness.\n\nWhat's new is the specific combination: RAG retrieval over RFC-2326, ReAct-style iteration, and COT for seed generation and placement, evaluated on RTSP. The components are all known, and the related work (ChatAFL, SeedMind) already uses LLMs for seeds, so the novelty is incremental. The paper does a few things well: the prompt design is clearly described, the tables are consistent across all request types, and improvements are large enough that the effect is unlikely to be a fluke.\n\nThe soft spots are real and load-bearing. Section 5 defines 'ground truth' as ChatAFL logs; Section 6 scores outputs with BLEU/ROUGE/WER. Those metrics measure surface string overlap, not protocol conformance or state-space coverage. An output can echo common RTSP templates and score well while violating state transitions, or score poorly while being a valid seed. The ground truth also derives from an LLM-guided fuzzer using the same RFC, so high similarity may reflect stylistic mimicry rather than seed quality. No fuzzing run, coverage measurement, or packet-acceptance check is reported; Section 7 lists integration into fuzzing frameworks as future work. So the paper's central causal chain — better similarity scores imply better vulnerability discovery — does not follow. There are also no error bars or significance tests, and the abstract's 'significant improvements' is overclaiming for that reason. These are addressable flaws: add a real fuzzing experiment with coverage or crash-based metrics, and validate that the generated seeds are accepted by the server.\n\nI agree with the reader's conditional verdict and the stress-test note. The narrow numerical claim is probably right, but the framing overreaches.\n\nWho's this for? Researchers working on LLM-assisted fuzzing seed generation. A serious referee should engage with it, because the evaluation methodology problem is worth airing and the approach has legs if the proxy is fixed. I would not cite it in my own work yet, but I'd bring it to a reading group as a case study in metric validation.","headline":"Plausible RAG-ReAct gains on text-similarity metrics for RTSP seed generation, but the leap from BLEU/ROUGE/WER to fuzzing effectiveness is unvalidated.","tokens_in":7396,"tokens_out":2351,"would_cite":false,"duration_ms":22360,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retrieval-augmented reasoning makes LLMs produce RTSP packet text closer to real fuzzer logs.","keywords":["finite state machine inference","protocol fuzzing","initial seed generation","retrieval augmented generation","chain-of-thought reasoning","ReAct agent","RTSP","LLM evaluation"],"falsifier":"Run the RAG-ReAct seeds and the baseline seeds through the same protocol fuzzer, for example feeding each set to ChatAFL on a live555 server, and compare state or edge coverage and unique crashes. If retrieval-reasoned seeds do not produce more coverage or crashes than baseline seeds, the paper's core assumption that closer BLEU, ROUGE, and WER match to fuzzer logs means better fuzz seeds is contradicted.","tokens_in":6411,"feed_emoji":"📡","tokens_out":6166,"duration_ms":59856,"temperature":0.7,"pith_summary":"This paper tests whether adding retrieval and step-by-step reasoning to an LLM makes it write better initial packet seeds for protocol fuzzing. The authors build a RAG-ReAct agent that queries a vector store built from the RTSP specification (RFC-2326) and uses chain-of-thought prompting to place RTSP requests correctly, then compare its outputs with the base Gemma-2-9B and Llama-3-8B models. Measured against packet sequences from ChatAFL fuzzing logs, the agent scores higher BLEU and ROUGE and lower word error rate across all ten RTSP request types, with the largest gains reported on complex commands like ANNOUNCE and SET PARAMETER. The reason this matters is that protocol fuzzing's state-space exploration is sensitive to initial seeds; if LLM-generated seeds follow the protocol's FSM more closely, fuzzing can explore more states and find more vulnerabilities.","feed_headline":"Retrieval-driven reasoning sharpens LLM-made protocol fuzz seeds","feed_subtitle":"RAG agents pull RFC-2326 knowledge, then reason step by step to write RTSP packets closer to real fuzzer logs.","key_machinery":"The central object is the RAG-ReAct agent: a retrieval-augmented generation loop in which cleaned RFC-2326 text is chunked into 1,000-token overlapping pieces, embedded with a text-embedding model, and stored in a vector database; at inference time the agent retrieves the top-k chunks by cosine similarity and uses chain-of-thought prompts to reason over that context before placing seeds. The load-bearing identity is the pairing of retrieval (RAG) with the ReAct reasoning-and-acting loop and COT prompting: retrieval supplies protocol-specific FSM facts, the reasoning loop decides where each packet belongs, and the structured prompt templates constrain the output format. The evaluation machinery is the comparison of those outputs against ChatAFL-captured ground-truth packets using BLEU, ROUGE, and WER.","core_discovery":"The authors claim that an LLM equipped with retrieval over the RTSP specification and chain-of-thought reasoning produces packet texts that are structurally closer to real protocol-fuzzer traffic than the same LLM without those additions. Using RFC-2326 as the only knowledge source, they build a RAG-ReAct agent on top of Gemma-2-9B and Llama-3-8B, retrieve top-k chunks with cosine similarity, and have the agent reason iteratively before emitting enriched RTSP request sequences. Against roughly 120 ground-truth entries drawn from two hours of ChatAFL fuzzing on a live555 server, the agent's outputs achieve up to 18.19% higher BLEU, 14.81% higher ROUGE, and 23.45% lower word error rate than the base models, with improvements on every RTSP request type and particularly large gains on ANNOUNCE, SET PARAMETER, and RECORD. The authors take this as evidence that retrieval-augmented, reasoning-based agents can generate protocol-compliant seeds that would guide fuzzing frameworks through a wider state-space exploration.","pith_inferences":["Beyond the paper: the authors never run the generated seeds through a fuzzer, so a direct test of whether the similarity gains translate into higher state coverage or more crashes remains open.","Beyond the paper: if the similarity proxy holds, the same pipeline should transfer to any RFC-documented stateful protocol by replacing the knowledge base; the paper only demonstrates RTSP, so testing SIP or FTP would be a natural extension.","Beyond the paper: a hybrid approach could pair the agent's FSM reasoning with a deterministic protocol parser to validate packet conformance before fuzzing, which would address the weakness that text similarity does not guarantee parseable, in-state messages."],"forward_implications":["Protocol fuzzers can receive an enriched initial seed set from a RAG-ReAct agent instead of relying on the base LLM's unaided output.","The approach avoids fine-tuning: retrieval plus prompting provides protocol-specific knowledge at inference time, making it cheaper to adapt to new protocols.","The biggest measured gains are on complex RTSP commands such as ANNOUNCE, SET PARAMETER, and RECORD, so the method is most likely to help where base models are weakest.","The evaluation recipe, comparing generated packets to fuzzer-captured logs with BLEU, ROUGE, and WER, can serve as a lightweight filter before committing to expensive fuzzing campaigns.","Both tested LLMs improve when wrapped in the RAG-ReAct agent, suggesting the mechanism is model-agnostic within this size class."],"supporting_citations":[{"why":"Supplies the ground-truth RTSP fuzzing log dataset and the ChatAFL fuzzer whose seed-enrichment workflow the agent's queries are modelled on.","marker":"[7]"},{"why":"Provides the retrieval-augmented generation paradigm that the RAG component is built on.","marker":"[10]"},{"why":"Provides the ReAct reasoning-and-acting framework that the agent uses to reason over retrieved RFC chunks.","marker":"[11]"},{"why":"SeedMind is the nearest LLM-seed-generation baseline the paper positions its contribution against.","marker":"[12]"},{"why":"ChatFuzz is the closest LLM-seed-variant method for greybox fuzzing, used as related-work comparison for seed enrichment.","marker":"[8]"},{"why":"Motivates why initial seeds matter for stateful fuzzing, the premise for the whole evaluation.","marker":"[2]"}],"fun_headline_variants":["RAG and chain-of-thought boost LLM packet seeds for fuzzing","LLM with retrieval and reasoning writes better fuzz packets","Retrieval-augmented LLM and CoT refine protocol fuzz seeds","RAG + CoT improve LLM-generated seeds for protocol fuzzing","RAG-guided reasoning writes protocol fuzz seeds closer to real traffic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire argument rests on the premise that the more closely LLM-generated packets textually match packets captured from ChatAFL fuzzing logs, the better those packets will work as fuzzing seeds; no experiment in the paper tests that link directly.","fun_headline_variants_meta":{"raw":{"variants":["RAG and chain-of-thought boost LLM packet seeds for fuzzing","LLM with retrieval and reasoning writes better fuzz packets","Retrieval-augmented LLM and CoT refine protocol fuzz seeds","RAG + CoT improve LLM-generated seeds for protocol fuzzing","RAG-guided reasoning writes protocol fuzz seeds closer to real traffic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001077,"raw_usage":{"total_tokens":4539,"prompt_tokens":1006,"completion_tokens":3533,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":3436}},"tokens_in":622,"tokens_out":3533,"duration_ms":24464,"temperature":1.0,"reasoning_tokens":3436,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T00:44:58.549070+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the RAG-ReAct seeds and the baseline seeds through the same protocol fuzzer, for example feeding each set to ChatAFL on a live555 server, and compare state or edge coverage and unique crashes. If retrieval-reasoned seeds do not produce more coverage or crashes than baseline seeds, the paper's core assumption that closer BLEU, ROUGE, and WER match to fuzzer logs means better fuzz seeds is contradicted.","supporting_citations":[{"cited_title":"In: Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS) (2024)","cited_arxiv_id":null,"evidence_quote":"Supplies the ground-truth RTSP fuzzing log dataset and the ChatAFL fuzzer whose seed-enrichment workflow the agent's queries are modelled on."},{"cited_title":"In: 2023 IEEE International Conference on Sensors, Electronics and Computer Engineering (ICSECE), pp","cited_arxiv_id":null,"evidence_quote":"Motivates why initial seeds matter for stateful fuzzing, the premise for the whole evaluation."}],"review_version":1}