Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

Retrieval Augmented Generation Based LLM Evaluation For Protocol State Machine Inference With Chain-of-Thought Reasoning

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Retrieval-augmented reasoning makes LLMs produce RTSP packet text closer to real fuzzer logs.

desk verdict Plausible RAG-ReAct gains on text-similarity metrics for RTSP seed generation, but the leap from BLEU/ROUGE/WER to fuzzing effectiveness is unvalidated. read the letter →

arxiv 2502.15727 v2 pith:EXZLBRKY submitted 2025-01-30 cs.NI cs.AIcs.CRcs.IR

Youssef Maklad , Fares Wael , Wael Elsersy , Ali Hamdi This is my paper · ORCID
classification cs.NIcs.AIcs.CRcs.IR
keywords finitestatemachineinferenceprotocolfuzzinginitialseedgenerationretrievalaugmentedchain-of-thoughtreasoningReActagentRTSPLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether adding retrieval and step-by-step reasoning to an LLM makes it write better initial packet seeds for protocol fuzzing. The authors build a RAG-ReAct agent that queries a vector store built from the RTSP specification (RFC-2326) and uses chain-of-thought prompting to place RTSP requests correctly, then compare its outputs with the base Gemma-2-9B and Llama-3-8B models. Measured against packet sequences from ChatAFL fuzzing logs, the agent scores higher BLEU and ROUGE and lower word error rate across all ten RTSP request types, with the largest gains reported on complex commands like ANNOUNCE and SET PARAMETER. The reason this matters is that protocol fuzzing's state-space exploration is sensitive to initial seeds; if LLM-generated seeds follow the protocol's FSM more closely, fuzzing can explore more states and find more vulnerabilities.

What carries the argument

The central object is the RAG-ReAct agent: a retrieval-augmented generation loop in which cleaned RFC-2326 text is chunked into 1,000-token overlapping pieces, embedded with a text-embedding model, and stored in a vector database; at inference time the agent retrieves the top-k chunks by cosine similarity and uses chain-of-thought prompts to reason over that context before placing seeds. The load-bearing identity is the pairing of retrieval (RAG) with the ReAct reasoning-and-acting loop and COT prompting: retrieval supplies protocol-specific FSM facts, the reasoning loop decides where each packet belongs, and the structured prompt templates constrain the output format. The evaluation machinery is the comparison of those outputs against ChatAFL-captured ground-truth packets using BLEU, ROUGE, and WER.

What would settle it

Run the RAG-ReAct seeds and the baseline seeds through the same protocol fuzzer, for example feeding each set to ChatAFL on a live555 server, and compare state or edge coverage and unique crashes. If retrieval-reasoned seeds do not produce more coverage or crashes than baseline seeds, the paper's core assumption that closer BLEU, ROUGE, and WER match to fuzzer logs means better fuzz seeds is contradicted.

Watch

Extended reading notes

Core claim

The authors claim that an LLM equipped with retrieval over the RTSP specification and chain-of-thought reasoning produces packet texts that are structurally closer to real protocol-fuzzer traffic than the same LLM without those additions. Using RFC-2326 as the only knowledge source, they build a RAG-ReAct agent on top of Gemma-2-9B and Llama-3-8B, retrieve top-k chunks with cosine similarity, and have the agent reason iteratively before emitting enriched RTSP request sequences. Against roughly 120 ground-truth entries drawn from two hours of ChatAFL fuzzing on a live555 server, the agent's outputs achieve up to 18.19% higher BLEU, 14.81% higher ROUGE, and 23.45% lower word error rate than the base models, with improvements on every RTSP request type and particularly large gains on ANNOUNCE, SET PARAMETER, and RECORD. The authors take this as evidence that retrieval-augmented, reasoning-based agents can generate protocol-compliant seeds that would guide fuzzing frameworks through a wider state-space exploration.

Load-bearing premise

The entire argument rests on the premise that the more closely LLM-generated packets textually match packets captured from ChatAFL fuzzing logs, the better those packets will work as fuzzing seeds; no experiment in the paper tests that link directly.

Editorial extensions

If this is right

  • Protocol fuzzers can receive an enriched initial seed set from a RAG-ReAct agent instead of relying on the base LLM's unaided output.
  • The approach avoids fine-tuning: retrieval plus prompting provides protocol-specific knowledge at inference time, making it cheaper to adapt to new protocols.
  • The biggest measured gains are on complex RTSP commands such as ANNOUNCE, SET PARAMETER, and RECORD, so the method is most likely to help where base models are weakest.
  • The evaluation recipe, comparing generated packets to fuzzer-captured logs with BLEU, ROUGE, and WER, can serve as a lightweight filter before committing to expensive fuzzing campaigns.
  • Both tested LLMs improve when wrapped in the RAG-ReAct agent, suggesting the mechanism is model-agnostic within this size class.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the authors never run the generated seeds through a fuzzer, so a direct test of whether the similarity gains translate into higher state coverage or more crashes remains open.
  • Beyond the paper: if the similarity proxy holds, the same pipeline should transfer to any RFC-documented stateful protocol by replacing the knowledge base; the paper only demonstrates RTSP, so testing SIP or FTP would be a natural extension.
  • Beyond the paper: a hybrid approach could pair the agent's FSM reasoning with a deterministic protocol parser to validate packet conformance before fuzzing, which would address the weakness that text similarity does not guarantee parseable, in-state messages.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes RAG-ReAct, an agentic LLM pipeline that retrieves protocol specification chunks (RFC-2326 for RTSP) from a vector store and uses chain-of-thought reasoning to generate network packet seeds for protocol fuzzing. It compares Gemma-2-9B and Llama-3-8B in baseline and RAG-ReAct settings, evaluating the generated packets with BLEU, ROUGE, and WER against 120 ground-truth entries drawn from ChatAFL fuzzing logs on the live555 server. The paper reports average score improvements for both models and concludes that the approach improves LLM-based protocol fuzzing frameworks and helps identify hidden vulnerabilities.

Significance. If the reported improvements reflected fuzzing-relevant packet quality, the work would offer a lightweight, fine-tuning-free method for seed enrichment that could benefit protocol fuzzing practice. The technical construction (RAG + ReAct + COT) is reasonable, the paper compares two representative LLMs, and the tables are clearly presented. However, the evaluation measures surface text similarity to ChatAFL logs rather than protocol conformance, code coverage, or vulnerability discovery, and the ground truth shares the same underlying specification source as the agent's retrieval base. As a result, the central claim about improving protocol fuzzing frameworks is currently unsupported. The paper is a useful preliminary study but needs a downstream fuzzing evaluation or a validated proxy for seed quality.

major comments (4)
  1. [Section 6 (Tables 1 and 2)] The evaluation treats BLEU/ROUGE/WER similarity to ChatAFL logs as a proxy for seed quality. This is load-bearing for the abstract's claim of 'improving LLM-based protocol fuzzing frameworks', yet no fuzzing run, coverage measurement, packet-acceptance check, or vulnerability discovery is reported, and Section 7 lists integration into fuzzing frameworks as future work. Text similarity can be high for a packet that violates RTSP state transitions or headers, and low for a valid and useful seed. Please add a downstream fuzzing experiment or at least a validity test showing that higher scores correlate with better state-space exploration.
  2. [Section 5 (Experimental Design)] The ground-truth packets are logs captured from ChatAFL, an LLM-guided fuzzer, running against live555, and ChatAFL's generation is based on protocol specifications including the same RFC-2326 that the RAG knowledge base retrieves from. The comparison therefore may reward mimicry of ChatAFL's output style rather than protocol compliance or fuzz-relevant quality. Independent ground truth (e.g., manually validated RTSP messages or real protocol traces) or a calibrated metric is needed to support the conclusions.
  3. [Abstract and Section 6] The reported 'up to' improvements are not as stated. The values 18.19%, 14.81%, and 23.45% are the means over the two models of the absolute percentage-point changes (with the WER value being a reduction, not an increase), so describing them as 'improvements of up to X% in WER' is misleading. Moreover, no error bars, confidence intervals, significance tests, or per-request sample sizes are given; Tables 1 and 2 show single numbers per condition, so the claim of 'consistent improvements' is not supported statistically. Please report the number of ground-truth queries per request type and repeat the generation with diverse sampling to quantify variance.
  4. [Title and Sections 1 and 6] The paper is framed around protocol state machine inference, but the evaluation never checks whether the generated packet sequences correspond to valid FSM transitions from RFC-2326. The metrics only compare token overlap with ChatAFL logs, not whether the agent's state-machine reasoning is correct. A direct evaluation of FSM accuracy (e.g., validating state sequences against the RFC's transition table) or a reformulation of the paper's scope as seed text generation would be needed.
minor comments (6)
  1. [Section 3] The phrase 'The protocol FSM describes defines how' appears to contain an error; it should read 'describes/defines how'.
  2. [Section 1] The abbreviation 'ARPE' should be 'APRE' (automatic protocol reverse engineering) in the sentence about integrating LLMs with existing techniques.
  3. [Tables 1 and 2] The table headings read 'T able' instead of 'Table', and values such as '08.71%' should be formatted as '8.71%'.
  4. [Section 5] The relationship between the '5000 plus entries' log dataset and the 'typically 120 entries' evaluation set is unclear; please state how the 120 queries were sampled and whether each request type is equally represented.
  5. [Figure 3] The prompt templates are shown in a figure but not described in the text; a brief summary of the three prompt sections would improve reproducibility.
  6. [Section 4.1] The hyperparameters chunk_size=1000, overlap=200, and retrieval top-k are given without justification or sensitivity analysis; adding an ablation would strengthen the methodology.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the claimed BLEU/ROUGE/WER gains are empirical comparisons, not reductions to inputs or fitted renames.

full rationale

The paper's central claim is that a RAG-ReAct agent with chain-of-thought prompting produces RTSP packet texts closer to ChatAFL fuzzing logs than baseline LLM outputs. This claim is not circular by construction: the RFC-2326 knowledge base is an input to retrieval, but the ground-truth packets from ChatAFL are not used to fit, train, or construct the RAG-ReAct outputs, and no equation defines the evaluated metric in terms of the system's own outputs. The high similarity is an empirical outcome, not an identity. The main weakness is external validity: BLEU, ROUGE, and WER measure surface-text overlap with ChatAFL logs, and the paper does not run a fuzzing experiment, so the inference from text similarity to fuzzing effectiveness is unsupported. That is a correctness or evaluation-validity concern, not a circularity. The self-citations in the reference list appear only in general background passages and are not load-bearing for the protocol-evaluation argument. Therefore no specific circular reduction can be exhibited, and the honest finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the domain assumption that RFC-2326 is a sufficient and accurate basis for RTSP packet generation, and on two evaluation assumptions: that text-similarity metrics are a valid proxy for seed quality, and that ChatAFL logs are ground truth. No free parameters are fitted to data, but two preprocessing hyperparameters are chosen by hand. No invented entities are introduced.

free parameters (2)
  • chunk_size = 1000 tokens, 200-character overlap
    Hand-chosen preprocessing hyperparameter for RFC chunking. It affects retrieval context but is not tuned or justified.
  • retrieval_top_k = not reported
    The number of retrieval chunks used per query is not specified, so it is an implicit, unreported design choice.
assumptions (3)
  • domain assumption RFC-2326 is a complete and accurate specification of the RTSP protocol FSM
    The RAG agent uses RFC-2326 as the sole knowledge base for generating seed packets. The paper provides no independent verification that the RFC captures all relevant FSM behavior, and it assumes the retrieved chunks are sufficient for correct packet generation.
  • ad hoc to paper Textual similarity to ChatAFL logs is a valid proxy for seed quality and fuzzing effectiveness
    The evaluation uses BLEU, ROUGE, and WER against fuzzer logs as the only measure of quality. This proxy is asserted in Section 5 and 6 without any fuzzing experiment or correlation analysis to justify it.
  • domain assumption ChatAFL fuzzing logs constitute ground-truth protocol packets
    The paper treats packets from a 2-hour fuzzing session as ground truth for evaluation. Fuzzer logs are generated by a tool that may itself produce mutated or non-conforming packets, so they are not an authoritative protocol reference.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Retrieval Augmented Generation Based LLM Evaluation For Protocol State Machine Inference With Chain-of-Thought Reasoning." pith.science (2026). https://pith.science/paper/EXZLBRKY

@misc{pith2026250215727,
  author       = {Pith},
  title        = {Pith review of: Retrieval Augmented Generation Based LLM Evaluation For Protocol State Machine Inference With Chain-of-Thought Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EXZLBRKY}},
  note         = {Machine review of arXiv:2502.15727}
}
read the original abstract

This paper presents a novel approach to evaluate the efficiency of a RAG-based agentic Large Language Model (LLM) architecture for network packet seed generation and enrichment. Enhanced by chain-of-thought (COT) prompting techniques, the proposed approach focuses on the improvement of the seeds' structural quality in order to guide protocol fuzzing frameworks through a wide exploration of the protocol state space. Our method leverages RAG and text embeddings to dynamically reference to the Request For Comments (RFC) documents knowledge base for answering queries regarding the protocol's Finite State Machine (FSM), then iteratively reasons through the retrieved knowledge, for output refinement and proper seed placement. We then evaluate the response structure quality of the agent's output, based on metrics as BLEU, ROUGE, and Word Error Rate (WER) by comparing the generated packets against the ground-truth packets. Our experiments demonstrate significant improvements of up to 18.19%, 14.81%, and 23.45% in BLEU, ROUGE, and WER, respectively, over baseline models. These results confirm the potential of such approach, improving LLM-based protocol fuzzing frameworks for the identification of hidden vulnerabilities.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MultiFuzz: A Dense Retrieval-based Multi-Agent System for Network Protocol Fuzzing

    cs.CR 2025-08 reject novelty 4.0 of 10

    MultiFuzz combines retrieval-augmented generation and multiple LLM agents within the ChatAFL protocol fuzzer, reporting marginal and statistically unsupported gains in branch coverage and state exploration for RTSP.

  2. Pixels to Play: A Foundation Model for 3D Gameplay

    cs.CV 2025-08 unverdicted novelty 4.0 of 10

    Pixels2Play-0.1 is a decoder-only transformer trained via behavior cloning and inverse-dynamics-imputed actions to play 3D games from pixels, with only qualitative results reported.

  3. An Agentic Flow for Finite State Machine Extraction using Prompt Chaining

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A three-stage LLM prompt-chaining system extracts FSM rulebooks from RFC documents, achieving F1 scores near 85% on FTP and RTSP.

Reference graph

Works this paper leans on

23 extracted references · 11 canonical work pages · cited by 3 Pith papers

  1. [1]

    Electronics 12(13), 2904 (2023)

    Zhang, Z., Zhang, H., Zhao, J., Yin, Y.: A survey on the development of network protocol fuzzing techniques. Electronics 12(13), 2904 (2023)

    work page 2023

  2. [2]

    In: 2023 IEEE International Conference on Sensors, Electronics and Computer Engineering (ICSECE), pp

    Hu, F., Gu, Y., Xiong, X.: Exploring the importance of initial seeds in stateful fuzzing of network services. In: 2023 IEEE International Conference on Sensors, Electronics and Computer Engineering (ICSECE), pp. 174–178 (2023). IEEE 8

    work page 2023

  3. [3]

    In: 32nd USENIX Security Symposium (USENIX Security 23), pp

    Shi, Q., Xu, X., Zhang, X.: Extracting protocol format as state machine via controlled static loop analysis. In: 32nd USENIX Security Symposium (USENIX Security 23), pp. 7019–7036 (2023)

    work page 2023

  4. [4]

    In: 31st USENIX Security Symposium (USENIX Security 22), pp

    Zhou, S., Yang, Z., Qiao, D., Liu, P., Yang, M., Wang, Z., Wu, C.: Ferry: {State- Aware} symbolic execution for exploring {State-Dependent} program paths. In: 31st USENIX Security Symposium (USENIX Security 22), pp. 4365–4382 (2022)

    work page 2022

  5. [5]

    In: Proceedings of the 22nd ACM Workshop on Hot Topics in Networks, pp

    Sharma, P., Yegneswaran, V.: Prosper: Extracting protocol specifications using large language models. In: Proceedings of the 22nd ACM Workshop on Hot Topics in Networks, pp. 41–47 (2023)

    work page 2023

  6. [6]

    arXiv preprint arXiv:2405.00393 (2024)

    Wei, H., Du, Z., Huang, H., Liu, Y., Cheng, G., Wang, L., Mao, B.: Inferring state machine from the protocol implementation via large langeuage model. arXiv preprint arXiv:2405.00393 (2024)

    arXiv 2024

  7. [7]

    In: Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS) (2024)

    Meng, R., Mirchev, M., B¨ ohme, M., Roychoudhury, A.: Large language model guided protocol fuzzing. In: Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS) (2024)

    work page 2024

  8. [8]

    arXiv preprint arXiv:2306.06782 (2023)

    Hu, J., Zhang, Q., Yin, H.: Augmenting greybox fuzzing with generative ai. arXiv preprint arXiv:2306.06782 (2023)

    arXiv 2023

Show all 23 references
  1. [9]

    ACM Computing Surveys (CSUR) 48(3), 1–26 (2015)

    Narayan, J., Shukla, S.K., Clancy, T.C.: A survey of automatic protocol reverse engineering tools. ACM Computing Surveys (CSUR) 48(3), 1–26 (2015)

    2015

  2. [10]

    Advances in Neural Information Processing Systems 33, 9459–9474 (2020)

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K¨ uttler, H., Lewis, M., Yih, W.-t., Rockt¨ aschel, T.,et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33, 9459–9474 (2020)

    2020

  3. [11]

    arXiv preprint arXiv:2210.03629 (2022)

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y.: React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022)

    2022 arXiv

  4. [12]

    arXiv preprint arXiv:2411.18143 (2024)

    Shi, W., Zhang, Y., Xing, X., Xu, J.: Harnessing large language models for seed generation in greybox fuzzing. arXiv preprint arXiv:2411.18143 (2024)

    2024 arXiv

  5. [13]

    In: 14th USENIX Workshop on Offensive Technologies (WOOT 20) (2020)

    Fioraldi, A., Maier, D., Eißfeldt, H., Heuse, M.: {AFL++}: Combining incre- mental steps of fuzzing research. In: 14th USENIX Workshop on Offensive Technologies (WOOT 20) (2020)

    2020

  6. [14]

    IEEE Access (2024)

    Black, G., Vaidyan, V., Comert, G.: Evaluating large language models for enhanced fuzzing: An analysis framework for llm-driven seed generation. IEEE Access (2024)

    2024

  7. [15]

    In: 2022 IEEE Symposium on Security and Privacy (SP), pp

    Pacheco, M.L., Hippel, M., Weintraub, B., Goldwasser, D., Nita-Rotaru, C.: 9 Automated attack synthesis by extracting finite state machines from protocol specification documents. In: 2022 IEEE Symposium on Security and Privacy (SP), pp. 51–68 (2022). IEEE

    2022

  8. [16]

    Advances in Neural Information Processing Systems (2017)

    Vaswani, A.: Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  9. [17]

    arXiv preprint arXiv:2107.03374 (2021)

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.D.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al.: Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    2021 arXiv

  10. [18]

    arXiv preprint arXiv:2412.18063 (2024)

    Abdellaif, O.H., Nader, A., Hamdi, A.: Lmrpa: Large language model-driven efficient robotic process automation for ocr. arXiv preprint arXiv:2412.18063 (2024)

    2024 arXiv

  11. [19]

    arXiv preprint arXiv:2412.17965 (2024)

    Abdellatif, O., Ayman, A., Hamdi, A.: Lmv-rpa: Large model voting-based robotic process automation. arXiv preprint arXiv:2412.17965 (2024)

    2024 arXiv

  12. [20]

    arXiv preprint arXiv:2412.11672 (2024)

    Wassim, L., Mohamed, K., Hamdi, A.: Llm-daas: Llm-driven drone-as-a-service operations from text user requests. arXiv preprint arXiv:2412.11672 (2024)

    2024 arXiv

  13. [21]

    In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp

    Hamad, O., Shaban, K., Hamdi, A.: Asem: Enhancing empathy in chatbot through attention-based sentiment and emotion modeling. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 1588–...

    2024

  14. [22]

    arXiv preprint arXiv:2412.13765 (2024)

    Hamdi, A., Mazrou, A.A., Shaltout, M.: Llm-sem: A sentiment-based stu- dent engagement metric using llms for e-learning platforms. arXiv preprint arXiv:2412.13765 (2024)

    2024 arXiv

  15. [23]

    arXiv preprint arXiv:2412.15254 (2024) 10

    Hamdi, A., Kassab, H., Bahaa, M., Mohamed, M.: Riro: Reshaping inputs, refin- ing outputs unlocking the potential of large language models in data-scarce contexts. arXiv preprint arXiv:2412.15254 (2024) 10

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.