Pith. sign in

REVIEW 3 major objections 5 minor 37 references

An Agentic Flow for Finite State Machine Extraction using Prompt Chaining

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FlowFSM claims that a three-stage prompt-chaining agent can extract protocol finite-state machines from raw RFC documents with precision above 81%, recall above 85%, and few hallucinated transitions.

desk verdict A clearly-described, honest LLM pipeline for RFC-to-FSM extraction, but the headline numbers rest on an unpublished ground truth and no baseline comparison, so the evaluation is not yet trustworthy. read the letter →

arxiv 2507.11222 v1 pith:2J347PV3 submitted 2025-07-15 cs.CL cs.AIcs.NI

Fares Wael , Youssef Maklad , Ali Hamdi , Wael Elsersy This is my paper
classification cs.CLcs.AIcs.NI
keywords finite-statemachineextractionRFCdocumentspromptchainingchain-of-thoughtLLMagentshallucinationreductionFTPRTSP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a structured chain of LLM prompts can extract a finite-state machine—a model of a protocol's states and the transitions between them—directly from the prose of an RFC document (the Internet's protocol specification series), with no hand-written protocol logic. The proposed system, FlowFSM, first parses the RFC into sections, then runs three chained stages—command extraction, state-transition analysis, and rulebook synthesis—so each prompt is built from the previous output. The authors report that on FTP and RTSP the extractions reach 83.33% and 81.82% precision and 88.24% and 85.71% recall, with F1 scores of 85.71% and 83.72%. This matters because protocol state machines are the input to verification, fuzzing, and vulnerability discovery, and prior extraction methods depend on code analysis or heavier NLP machinery.

What carries the argument

The operative mechanism is prompt chaining, formalized as $R_{i+1}=M(P_i(R_i))$, where the $i$-th prompt is constructed from the previous model response so the LLM iteratively refines its own extraction rather than attempting the whole task at once. FlowFSM's chain has three named stages: command extraction, state-transition analysis, and rulebook synthesis. The rulebook's three-chapter structure is the concrete output that functions as the FSM, and the chained format is what the paper credits for reducing false positive transitions while preserving coverage.

What would settle it

Publish the full set of true FTP and RTSP transitions before running the pipeline, then have a separate annotator or a reference implementation score the same outputs; if precision or recall falls materially under that pre-registered ground truth, the claim of high-fidelity extraction is not supported.

Watch

Extended reading notes

Core claim

The central claim is that the chained prompting procedure, not a larger model or hand-built parser, is what makes accurate FSM extraction possible: each stage outputs a structured intermediate artifact, and the next stage consumes it until a rulebook encodes the protocol's states and transitions. The rulebook organizes every command into three chapters: its purpose and effect on state, the valid commands that must precede it, and the valid commands that may follow it. The authors argue that this decomposition suppresses hallucinated transitions while keeping recall high enough for security applications where a missed transition is more harmful than an extra candidate.

Load-bearing premise

The ground-truth labels for every reported precision and recall number come from the authors' own manual reading of the RFCs, with no published transition list or second annotator; if those judgments are off, every metric loses its anchor.

Editorial extensions

If this is right

  • A protocol analyst could obtain a first-pass state machine from an RFC alone, before any source code or traffic trace is available.
  • The rulebook output is structured enough to hand to downstream verification or fuzzing tools without additional parsing.
  • The recall-over-precision tilt is the safe direction for security work: extra transitions can be filtered later, but a missed transition is invisible to any downstream analysis.
  • The small spread between FTP and RTSP results suggests the same three-stage chain may carry across protocol families with different document styles.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper, the real test of these rulebooks would be behavioral: feed them to a protocol fuzzer and see whether the extracted states improve coverage or expose deviations in real implementations.
  • The authors' manual labeling is the only ground truth; an independent evaluator with a pre-published transition list would reveal how much of the reported precision is the method's and how much reflects the annotator's expectations.
  • The same command-extraction / state-transition / rulebook chain could be applied to other semi-structured specification corpora, such as API documentation, where 'states' are implicit rather than labeled.
  • Ablating the chain—running each stage alone against the same RFCs—would separate the contribution of chaining from the raw capability of the underlying LLM.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces FlowFSM, an agent-based LLM pipeline that extracts finite-state machines from RFC documents via three chained prompts (command extraction, state-transition analysis, and rulebook synthesis) operating on a parsed RFC tree. The authors report experimental results for FTP and RTSP: FTP precision 83.33%, recall 88.24%, and F1 85.71%; RTSP precision 81.82%, recall 85.71%, and F1 83.72%. They interpret these numbers as evidence that FlowFSM 'achieves high extraction precision while minimizing hallucinated transitions' and that the method generalizes across protocols. The evaluation is based on manual validation of extracted transitions against the RFCs, but the manuscript does not define or release the ground-truth transition sets used for that validation.

Significance. If the empirical claims were fully supported, FlowFSM would be a modest but useful contribution to protocol reverse engineering, with a clearly described prompt-chaining architecture and open-sourced code. The method is anchored to external RFC documents rather than to the model's own outputs, so the central derivation is not circular, and the arithmetic in Table I is correct. However, the current lack of a defined ground truth, an annotation protocol, baselines, and repeated runs means that the quantitative contributions are not yet established. The work's significance therefore hinges on whether the evaluation can be made reproducible and comparative in revision.

major comments (3)
  1. [Section V-A, Table I] The metrics in Table I are computed against an unpublished and undefined ground truth. Section V-A defines TP, FP, and FN only in terms of 'official protocol behavior' and manual verification, but the manuscript never enumerates the set of protocol states or the valid transition set for FTP or RTSP, nor does it map extracted transitions to the RFC 959 state table or any other reference enumeration. The sample rulebook in Section IV-D shows one command but no state inventory. Consequently precision and recall are not well-defined: different annotators could enumerate different valid transitions from the same RFCs, and the reported 83.33% and 88.24% figures may measure agreement with the authors' implicit reading rather than correctness against the protocols. This is the load-bearing condition for the abstract's central claim, and it is not met in the current manuscript.
  2. [Section V-A] Manual validation is described but not operationalized. There is no inter-annotator agreement, no annotation guideline, no release of the ground-truth transition lists, and no statement of how disagreements would be resolved. Since every TP/FP/FN count in Table I depends on these manual judgments, the reported precision and recall are not reproducible. The authors should release the exact ground-truth transition sets (or an independently derivable abstraction, such as the RFC 959 state table) and report agreement measures such as Cohen's kappa for the validation step.
  3. [Section VI, RQ2] The generalization claim rests on only two protocols, FTP and RTSP, both relatively simple and structurally similar text-based protocols. No comparison is made to existing extraction systems such as PROSPER, RFCNLP, or ProtocolGPT, even though Section II describes those systems and reports their quantitative results. In addition, LLM outputs are stochastic, but the paper reports a single set of counts without repeated runs or variance. Without baselines, ablations, and multiple runs, the results do not establish that prompt chaining, rather than the underlying LLM, is responsible for the reported accuracy, and they do not support the conclusion that FlowFSM 'minimizes hallucinated transitions' beyond what a single unbaselined run can show.
minor comments (5)
  1. [Section I, contributions] The phrase 'reduced false postives' contains a typo; it should read 'false positives'.
  2. [Section VII] The sentence 'as a limitation of FlowFSM it it's high computational and runtime cost' is grammatically broken; it should be rewritten, for example as 'a limitation of FlowFSM is its high computational and runtime cost'.
  3. [Section IV-D, Figure caption] The figure is captioned 'Figure IV-D', which mixes section numbering with figure numbering; it should be numbered consistently with other figures.
  4. [Section II, reference [14]] The title of ProtocolGPT is mistyped as 'large langeuage model'; it should be 'large language model'.
  5. [Section V] The paper states that three LLMs were used but does not explain how their outputs were combined or whether Table I reports results from a single model, an ensemble, or a consensus; this should be clarified.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: FlowFSM's metrics come from manual validation against RFC documents rather than from fitting to its own outputs; the one same-author citation is motivational and not load-bearing.

full rationale

The paper's claimed derivation is not circular in the formal sense. FlowFSM's outputs are rule-book transitions extracted from RFC text, and Table I reports precision, recall, and F1 computed by manually comparing extracted transitions against RFC documents. The RFC documents are external to the method, and no fitted parameter is later renamed as a prediction: the only formulas are the standard metric definitions, and no equation in Section V equates a reported quantity to an input of the prompt-chaining pipeline. The main caveat is that the ground-truth transition lists are not enumerated in the paper, so the manual annotations are not independently auditable; this threatens reproducibility and construct validity, but it is an evaluation-quality problem rather than a circular derivation. The only same-author citation is reference [36], used in Section III-B to support the motivational claim that retrieval-augmented generation plus chain-of-thought improves LLM accuracy for network packet generation. That reference is not used to justify the architecture choice as forced, nor does any result in Table I depend on it; it is a minor self-citation that is not load-bearing. No self-definitional reduction, fitted-input-as-prediction, imported uniqueness theorem, or renamed known result is present. Therefore the appropriate circularity verdict is 'no significant circularity.'

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three domain assumptions about evaluation validity, protocol representativeness, and LLM stability. There are no free numeric parameters fitted to data and no new postulated entities.

assumptions (3)
  • domain assumption The authors' manual validation of extracted transitions against RFC documents is an accurate and objective ground truth.
    The paper reports TP/FP/FN counts judged by the authors, with no inter-annotator agreement, no released labels, and no external reference FSM.
  • domain assumption The two protocols, FTP and RTSP, are representative enough to support the generalization claim (RQ2).
    Only two RFCs are tested, both from the IETF style; no statistical justification is given for coverage of the space of protocol specifications.
  • domain assumption LLM outputs in the pipeline are stable enough that a single run per protocol is meaningful.
    No repeated runs or variance statistics are reported, so the effect of stochastic model responses is unmeasured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Agentic Flow for Finite State Machine Extraction using Prompt Chaining." pith.science (2026). https://pith.science/paper/2J347PV3

@misc{pith2026250711222,
  author       = {Pith},
  title        = {Pith review of: An Agentic Flow for Finite State Machine Extraction using Prompt Chaining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2J347PV3}},
  note         = {Machine review of arXiv:2507.11222}
}
read the original abstract

Finite-State Machines (FSMs) are critical for modeling the operational logic of network protocols, enabling verification, analysis, and vulnerability discovery. However, existing FSM extraction techniques face limitations such as scalability, incomplete coverage, and ambiguity in natural language specifications. In this paper, we propose FlowFSM, a novel agentic framework that leverages Large Language Models (LLMs) combined with prompt chaining and chain-of-thought reasoning to extract accurate FSMs from raw RFC documents. FlowFSM systematically processes protocol specifications, identifies state transitions, and constructs structured rule-books by chaining agent outputs. Experimental evaluation across FTP and RTSP protocols demonstrates that FlowFSM achieves high extraction precision while minimizing hallucinated transitions, showing promising results. Our findings highlight the potential of agent-based LLM systems in the advancement of protocol analysis and FSM inference for cybersecurity and reverse engineering applications.

Figures

Figures reproduced from arXiv: 2507.11222 by the authors.

Figure 1
Figure 1. FTP protocol’s finite-state machine as defined in RFC-959. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. RFC Documents Processing Pipeline A. RFC Document Processing Parsing RFC documents is crucial for extracting protocol specifications and FSM structures. RFCs are first preprocessed to remove page headers, footers, and formatting artifacts, producing continuous raw text T ∈ S, where S denotes the set of all string sequences. We define an RFC document as a hierarchical tree: T = (N, E) where N is the set of nodes (sec… view at source ↗
Figure 4
Figure 4. Three-stage prompt chaining template for iterative rulebook update [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 12 canonical work pages

  1. [36]

    Retrieval augmented generation based llm evaluation for protocol state machine inference with chain-of-thought reasoning,

    Y . Maklad, F. Wael, W. Elsersy, and A. Hamdi, “Retrieval augmented generation based llm evaluation for protocol state machine inference with chain-of-thought reasoning,” arXiv preprint arXiv:2502.15727 , 2025

    arXiv 2025

  2. [1]

    Sutton, A

    M. Sutton, A. Greene, and P. Amini, Fuzzing: brute force vulnerability discovery. Pearson Education, 2007

    2007

  3. [2]

    Verified models and reference implementations for the tls 1.3 standard candidate,

    K. Bhargavan, B. Blanchet, and N. Kobeissi, “Verified models and reference implementations for the tls 1.3 standard candidate,” in 2017 IEEE Symposium on Security and Privacy (SP) . IEEE, 2017, pp. 483– 502

    work page 2017

  4. [3]

    A survey on the development of network protocol fuzzing techniques,

    Z. Zhang, H. Zhang, J. Zhao, and Y . Yin, “A survey on the development of network protocol fuzzing techniques,” Electronics, vol. 12, no. 13, p. 2904, 2023

    2023

  5. [4]

    Large language models based fuzzing techniques: A survey,

    L. Huang, P. Zhao, H. Chen, and L. Ma, “Large language models based fuzzing techniques: A survey,” arXiv preprint arXiv:2402.00350 , 2024

    arXiv 2024

  6. [5]

    A survey of automatic protocol reverse engineering tools,

    J. Narayan, S. K. Shukla, and T. C. Clancy, “A survey of automatic protocol reverse engineering tools,” ACM Computing Surveys (CSUR) , vol. 48, no. 3, pp. 1–26, 2015

    2015

  7. [6]

    About RFCs — ietf.org,

    “About RFCs — ietf.org,” https://www.ietf.org/process/rfcs/, [Accessed 28-04-2025]

    work page 2025

  8. [7]

    Extracting protocol format as state machine via controlled static loop analysis,

    Q. Shi, X. Xu, and X. Zhang, “Extracting protocol format as state machine via controlled static loop analysis,” in 32nd USENIX Security Symposium (USENIX Security 23) , 2023, pp. 7019–7036

    work page 2023

Show all 37 references
  1. [8]

    Ferry: {State-Aware} symbolic execution for exploring {State- Dependent} program paths,

    S. Zhou, Z. Yang, D. Qiao, P. Liu, M. Yang, Z. Wang, and C. Wu, “Ferry: {State-Aware} symbolic execution for exploring {State- Dependent} program paths,” in 31st USENIX Security Symposium (USENIX Security 22) , 2022, pp. 4365–4382

    2022

  2. [9]

    State selection algorithms and their impact on the performance of stateful network protocol fuzzing,

    D. Liu, V .-T. Pham, G. Ernst, T. Murray, and B. I. Rubinstein, “State selection algorithms and their impact on the performance of stateful network protocol fuzzing,” in 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2022, p...

    2022

  3. [10]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017

    2017

  4. [11]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  5. [12]

    Language models can solve computer tasks,

    G. Kim, P. Baldi, and S. McAleer, “Language models can solve computer tasks,” Advances in Neural Information Processing Systems , vol. 36, pp. 39 648–39 677, 2023

    2023

  6. [13]

    Generative ai and large language models for cyber security: All insights you need,

    M. A. Ferrag, F. Alwahedi, A. Battah, B. Cherif, A. Mechri, and N. Tihanyi, “Generative ai and large language models for cyber security: All insights you need,” Available at SSRN 4853709 , 2024

    2024

  7. [14]

    Inferring state machine from the protocol implementation via large langeuage model,

    H. Wei, Z. Du, H. Huang, Y . Liu, G. Cheng, L. Wang, and B. Mao, “Inferring state machine from the protocol implementation via large langeuage model,” arXiv preprint arXiv:2405.00393 , 2024

    2024 arXiv

  8. [15]

    Prosper: Extracting protocol specifica- tions using large language models,

    P. Sharma and V . Yegneswaran, “Prosper: Extracting protocol specifica- tions using large language models,” in Proceedings of the 22nd ACM Workshop on Hot Topics in Networks , 2023, pp. 41–47

    2023

  9. [16]

    Large language model guided protocol fuzzing,

    R. Meng, M. Mirchev, M. B ¨ohme, and A. Roychoudhury, “Large language model guided protocol fuzzing,” in Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS) , 2024

    2024

  10. [17]

    Fsm: A finite state machine based zero-shot prompting paradigm for multi-hop question answering,

    X. Wang, J. He, Y . Wang, X. Meng, K. Pan, Z. Sui et al. , “Fsm: A finite state machine based zero-shot prompting paradigm for multi-hop question answering,” arXiv preprint arXiv:2407.02964 , 2024

    2024 arXiv

  11. [18]

    Can large language models help developers with robotic finite state machine modification?

    X. R. Gan, Y . R. Song, N. Walker, and M. Cakmak, “Can large language models help developers with robotic finite state machine modification?” arXiv preprint arXiv:2412.05625 , 2024

    2024 arXiv

  12. [19]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  13. [20]

    Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts,

    T. Wu, M. Terry, and C. J. Cai, “Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts,” in Proceedings of the 2022 CHI conference on human factors in computing systems, 2022, pp. 1–22

    2022

  14. [21]

    Aflnet: a greybox fuzzer for network protocols,

    V .-T. Pham, M. B¨ohme, and A. Roychoudhury, “Aflnet: a greybox fuzzer for network protocols,” in 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST) . IEEE, 2020, pp. 460–465

    2020

  15. [22]

    Automated attack synthesis by extracting finite state machines from protocol specification documents,

    M. L. Pacheco, M. von Hippel, B. Weintraub, D. Goldwasser, and C. Nita-Rotaru, “Automated attack synthesis by extracting finite state machines from protocol specification documents,” in 2022 IEEE Sym- posium on Security and Privacy (SP) . IEEE, 2022, pp. 51–68

    2022

  16. [23]

    Hermes: unlocking security analysis of cellular network protocols by synthesizing finite state machines from natural language specifications,

    A. Al Ishtiaq, S. S. S. Das, S. M. M. Rashid, A. Ranjbar, K. Tu, T. Wu, Z. Song, W. Wang, M. Akon, R. Zhang et al. , “Hermes: unlocking security analysis of cellular network protocols by synthesizing finite state machines from natural language specifications,” in 33rd USENIX S...

    2024

  17. [24]

    Netplier: Probabilistic network protocol reverse engineering from message traces

    Y . Ye, Z. Zhang, F. Wang, X. Zhang, and D. Xu, “Netplier: Probabilistic network protocol reverse engineering from message traces.” in NDSS, 2021

    2021

  18. [25]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948 , 2025

    2025 arXiv

  19. [26]

    Chatphishdetector: Detecting phishing sites using large language models,

    T. Koide, H. Nakano, and D. Chiba, “Chatphishdetector: Detecting phishing sites using large language models,” IEEE Access, 2024

    2024

  20. [27]

    Pentestagent: Incorporating llm agents to automated penetration testing,

    X. Shen, L. Wang, Z. Li, Y . Chen, W. Zhao, D. Sun, J. Wang, and W. Ruan, “Pentestagent: Incorporating llm agents to automated penetration testing,” arXiv preprint arXiv:2411.05185 , 2024

    2024 arXiv

  21. [28]

    Erpa: Efficient rpa model integrating ocr and llms for intelligent document processing,

    O. H. Abdellaif, A. N. Hassan, and A. Hamdi, “Erpa: Efficient rpa model integrating ocr and llms for intelligent document processing,” in 2024 International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC). IEEE, 2024, pp. 295–300

    2024

  22. [29]

    Lmrpa: Large lan- guage model-driven efficient robotic process automation for ocr,

    O. H. Abdellaif, A. Nader, and A. Hamdi, “Lmrpa: Large lan- guage model-driven efficient robotic process automation for ocr,” arXiv preprint arXiv:2412.18063, 2024

    2024 arXiv

  23. [30]

    Lmv-rpa: Large model voting- based robotic process automation,

    O. Abdellatif, A. Ayman, and A. Hamdi, “Lmv-rpa: Large model voting- based robotic process automation,” arXiv preprint arXiv:2412.17965 , 2024

    2024 arXiv

  24. [31]

    Llm-daas: Llm-driven drone-as-a-service operations from text user requests,

    L. Wassim, K. Mohamed, and A. Hamdi, “Llm-daas: Llm-driven drone-as-a-service operations from text user requests,” arXiv preprint arXiv:2412.11672, 2024

    2024 arXiv

  25. [32]

    Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,

    C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 919–931

    2023

  26. [33]

    Augmenting greybox fuzzing with generative ai,

    J. Hu, Q. Zhang, and H. Yin, “Augmenting greybox fuzzing with generative ai,” arXiv preprint arXiv:2306.06782 , 2023

    2023 arXiv

  27. [34]

    Harnessing large lan- guage models for seed generation in greybox fuzzing,

    W. Shi, Y . Zhang, X. Xing, and J. Xu, “Harnessing large lan- guage models for seed generation in greybox fuzzing,” arXiv preprint arXiv:2411.18143, 2024

    2024 arXiv

  28. [35]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel et al. , “Retrieval- augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems , vol. 33, pp. 9459–9474, 2020

    2020

  29. [37]

    “CrewAI,” https://www.crewai.com/, [Accessed 27-04-2025]

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.