{"id":"71da7d13-0ce2-435b-b4ea-9e730137cf65","arxiv_id":"2507.11222","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A three-stage LLM prompt-chaining system extracts FSM rulebooks from RFC documents, achieving F1 scores near 85% on FTP and RTSP.","lead":"FlowFSM is a multi-step LLM pipeline that turns RFC protocol documents into structured rulebooks describing states and transitions. It reports high precision and recall on FTP and RTSP, but without comparing to existing FSM extraction tools.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Precision/recall are computed against an unpublished, undefined ground truth: no state set or transition definition is given, so Table I may measure agreement with the authors' implicit reading rather than protocol correctness.","rationale":"The reader's weakest assumption correctly identifies the missing ground truth and manual validation as a core issue. My stress-test sharpens this: the problem is not only that the ground truth is unpublished, but that the paper never formally defines the target FSM abstraction. The rulebook structure in Section IV-D is command-centric — valid preceding and subsequent commands — rather than a state machine with named states and transitions. Without a state set, 'state transition' has no precise meaning, so precision and recall cannot be objectively computed. Even a released ground-truth list would be contestable unless it is derived from an explicit, justified abstraction (e.g., RFC 959's own FSM).\n\nThe paper has independent strengths worth crediting: the source code is released, the prompt-chaining pipeline is clearly described, and the three-stage decomposition is a plausible way to constrain LLM outputs. The reported numbers are internally consistent (precision and recall are correctly derived from the stated TP/FP/FN counts). However, the central claim of 'high precision while minimizing hallucinated transitions' is only as strong as the evaluation's construct validity. The missing baselines and error bars compound the problem but are secondary; the primary issue is that the measured quantity is not well-defined.\n\nBecause this concern is addressable — by releasing a precise ground truth with a defined state model and independent annotation — I do not move the verdict to reject. The conditional verdict remains appropriate, pending the requested artifacts and analysis. My agreement with the reader is partial because the reader's formulation emphasizes missing data and bias, while I additionally question whether the data could ever be objective without a formal definition of the target FSM.","tokens_in":8149,"tokens_out":5371,"duration_ms":67354,"concrete_test":"Publish the exact state set and ground-truth transition lists for FTP and RTSP, with per-transition citations to RFC sections, and have two independent annotators reconstruct those lists from the RFCs; compute inter-annotator agreement (Cohen's kappa) and recompute Table I using only transitions endorsed by both. If kappa < 0.8 or precision/recall change by more than 3 percentage points, the reported metrics are not stable. Alternatively, run FlowFSM on RFC 959's normative FSM table and require exact state/transition match.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that FlowFSM 'achieves high extraction precision while minimizing hallucinated transitions' rests entirely on Table I. Section V-A defines TP/FP/FN but does not define what counts as a state or a state transition, and Section IV-D's rulebook artifact records valid preceding/subsequent commands (Figure IV-D) without named states. RFC 959 contains a normative FSM table, but the paper never maps its extracted transitions to that table or to any other enumerated reference set. Since the ground-truth transition lists are neither released nor independently derivable from a stated abstraction, precision and recall are not well-defined: different annotators could enumerate different valid transitions from the same RFCs. The reported 83.33%/88.24% figures therefore may quantify agreement with the authors' implicit interpretation rather than correctness against the protocol. This is the load-bearing condition for the paper's main claim, and it is not met.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FlowFSM, an agent-based LLM pipeline that extracts finite-state machines from RFC documents via three chained prompts (command extraction, state-transition analysis, and rulebook synthesis) operating on a parsed RFC tree. The authors report experimental results for FTP and RTSP: FTP precision 83.33%, recall 88.24%, and F1 85.71%; RTSP precision 81.82%, recall 85.71%, and F1 83.72%. They interpret these numbers as evidence that FlowFSM 'achieves high extraction precision while minimizing hallucinated transitions' and that the method generalizes across protocols. The evaluation is based on manual validation of extracted transitions against the RFCs, but the manuscript does not define or release the ground-truth transition sets used for that validation.","tokens_in":8309,"tokens_out":3712,"duration_ms":45473,"significance":"If the empirical claims were fully supported, FlowFSM would be a modest but useful contribution to protocol reverse engineering, with a clearly described prompt-chaining architecture and open-sourced code. The method is anchored to external RFC documents rather than to the model's own outputs, so the central derivation is not circular, and the arithmetic in Table I is correct. However, the current lack of a defined ground truth, an annotation protocol, baselines, and repeated runs means that the quantitative contributions are not yet established. The work's significance therefore hinges on whether the evaluation can be made reproducible and comparative in revision.","major_comments":[{"comment":"The metrics in Table I are computed against an unpublished and undefined ground truth. Section V-A defines TP, FP, and FN only in terms of 'official protocol behavior' and manual verification, but the manuscript never enumerates the set of protocol states or the valid transition set for FTP or RTSP, nor does it map extracted transitions to the RFC 959 state table or any other reference enumeration. The sample rulebook in Section IV-D shows one command but no state inventory. Consequently precision and recall are not well-defined: different annotators could enumerate different valid transitions from the same RFCs, and the reported 83.33% and 88.24% figures may measure agreement with the authors' implicit reading rather than correctness against the protocols. This is the load-bearing condition for the abstract's central claim, and it is not met in the current manuscript.","section":"Section V-A, Table I"},{"comment":"Manual validation is described but not operationalized. There is no inter-annotator agreement, no annotation guideline, no release of the ground-truth transition lists, and no statement of how disagreements would be resolved. Since every TP/FP/FN count in Table I depends on these manual judgments, the reported precision and recall are not reproducible. The authors should release the exact ground-truth transition sets (or an independently derivable abstraction, such as the RFC 959 state table) and report agreement measures such as Cohen's kappa for the validation step.","section":"Section V-A"},{"comment":"The generalization claim rests on only two protocols, FTP and RTSP, both relatively simple and structurally similar text-based protocols. No comparison is made to existing extraction systems such as PROSPER, RFCNLP, or ProtocolGPT, even though Section II describes those systems and reports their quantitative results. In addition, LLM outputs are stochastic, but the paper reports a single set of counts without repeated runs or variance. Without baselines, ablations, and multiple runs, the results do not establish that prompt chaining, rather than the underlying LLM, is responsible for the reported accuracy, and they do not support the conclusion that FlowFSM 'minimizes hallucinated transitions' beyond what a single unbaselined run can show.","section":"Section VI, RQ2"}],"minor_comments":[{"comment":"The phrase 'reduced false postives' contains a typo; it should read 'false positives'.","section":"Section I, contributions"},{"comment":"The sentence 'as a limitation of FlowFSM it it's high computational and runtime cost' is grammatically broken; it should be rewritten, for example as 'a limitation of FlowFSM is its high computational and runtime cost'.","section":"Section VII"},{"comment":"The figure is captioned 'Figure IV-D', which mixes section numbering with figure numbering; it should be numbered consistently with other figures.","section":"Section IV-D, Figure caption"},{"comment":"The title of ProtocolGPT is mistyped as 'large langeuage model'; it should be 'large language model'.","section":"Section II, reference [14]"},{"comment":"The paper states that three LLMs were used but does not explain how their outputs were combined or whether Table I reports results from a single model, an ensemble, or a consensus; this should be clarified.","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":"The evaluation is the main obstacle: without a released ground truth, inter-annotator agreement, baselines, and repeated runs, the empirical claims cannot be assessed. The manuscript reads like an extended workshop paper; for a journal-level venue it needs a substantially strengthened evaluation. The self-citation [36] is used to support RAG plus chain-of-thought but is not essential to the proposed method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on LLM-based protocol analysis, but read it with the evaluation caveat in mind. The paper describes a concrete three-stage prompt-chaining pipeline (command extraction, state transition analysis, rulebook synthesis) for pulling FSMs out of RFCs, and it ships code and a sample rulebook. The architecture is easy to follow, and the idea of structuring the output as valid-preceding/subsequent command pairs is a sensible, modest extension of earlier work like PROSPER and RFCNLP. It is not a big leap, but it is a real new combination and the two-protocol evaluation (FTP, RTSP) is new.\n\nThe soft spot is exactly where the reader and the stress-test note point: the ground truth for Table I is not defined or released. Section V defines true positives as transitions \"manually verified to be correct according to the official protocol behavior,\" but it never states what counts as a state or a transition, and it never enumerates the set of valid transitions against which recall is computed. RFC 959 has a normative FSM table; the paper never maps its output to that table or to any other explicit reference set. Without that, precision and recall measure agreement with the authors' implicit reading, not correctness against the protocol. That is load-bearing, and it is not addressed. There is also no baseline (even a single-shot prompt would help), no repeated runs despite known LLM randomness, and only two protocols for the generalization claim. The word \"significantly\" in the contributions is an overclaim when no comparison is made.\n\nWhere the paper deserves credit: the method is described carefully, the code link is real, and the limitation about computational cost is stated honestly in the conclusion. The citation pattern is fine; the self-citation [36] is related prior work and is not used deceptively.\n\nMy bottom line: the core idea is plausible and the writing is clear, but the evaluation currently cannot support the headline numbers. This is a conditional-accept paper, not a reject or a strong accept. It deserves a serious referee, but the authors should be pushed to release their ground-truth transition lists, add a baseline comparison, run repeated trials, and evaluate at least one more protocol. If they do that, the paper would be a useful incremental contribution.","headline":"A clearly-described, honest LLM pipeline for RFC-to-FSM extraction, but the headline numbers rest on an unpublished ground truth and no baseline comparison, so the evaluation is not yet trustworthy.","tokens_in":730,"tokens_out":1349,"would_cite":false,"duration_ms":29532,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FlowFSM claims that a three-stage prompt-chaining agent can extract protocol finite-state machines from raw RFC documents with precision above 81%, recall above 85%, and few hallucinated transitions.","keywords":["finite-state machine extraction","RFC documents","prompt chaining","chain-of-thought","LLM agents","hallucination reduction","FTP","RTSP"],"falsifier":"Publish the full set of true FTP and RTSP transitions before running the pipeline, then have a separate annotator or a reference implementation score the same outputs; if precision or recall falls materially under that pre-registered ground truth, the claim of high-fidelity extraction is not supported.","tokens_in":7968,"feed_emoji":"🤖","tokens_out":7830,"duration_ms":90507,"temperature":0.7,"pith_summary":"The paper sets out to show that a structured chain of LLM prompts can extract a finite-state machine—a model of a protocol's states and the transitions between them—directly from the prose of an RFC document (the Internet's protocol specification series), with no hand-written protocol logic. The proposed system, FlowFSM, first parses the RFC into sections, then runs three chained stages—command extraction, state-transition analysis, and rulebook synthesis—so each prompt is built from the previous output. The authors report that on FTP and RTSP the extractions reach 83.33% and 81.82% precision and 88.24% and 85.71% recall, with F1 scores of 85.71% and 83.72%. This matters because protocol state machines are the input to verification, fuzzing, and vulnerability discovery, and prior extraction methods depend on code analysis or heavier NLP machinery.","feed_headline":"Prompt chaining pulls protocol state machines out of RFCs","feed_subtitle":"Three chained LLM steps hit 83% precision on FTP and 82% on RTSP, with few invented transitions.","key_machinery":"The operative mechanism is prompt chaining, formalized as $R_{i+1}=M(P_i(R_i))$, where the $i$-th prompt is constructed from the previous model response so the LLM iteratively refines its own extraction rather than attempting the whole task at once. FlowFSM's chain has three named stages: command extraction, state-transition analysis, and rulebook synthesis. The rulebook's three-chapter structure is the concrete output that functions as the FSM, and the chained format is what the paper credits for reducing false positive transitions while preserving coverage.","core_discovery":"The central claim is that the chained prompting procedure, not a larger model or hand-built parser, is what makes accurate FSM extraction possible: each stage outputs a structured intermediate artifact, and the next stage consumes it until a rulebook encodes the protocol's states and transitions. The rulebook organizes every command into three chapters: its purpose and effect on state, the valid commands that must precede it, and the valid commands that may follow it. The authors argue that this decomposition suppresses hallucinated transitions while keeping recall high enough for security applications where a missed transition is more harmful than an extra candidate.","pith_inferences":["Going beyond the paper, the real test of these rulebooks would be behavioral: feed them to a protocol fuzzer and see whether the extracted states improve coverage or expose deviations in real implementations.","The authors' manual labeling is the only ground truth; an independent evaluator with a pre-published transition list would reveal how much of the reported precision is the method's and how much reflects the annotator's expectations.","The same command-extraction / state-transition / rulebook chain could be applied to other semi-structured specification corpora, such as API documentation, where 'states' are implicit rather than labeled.","Ablating the chain—running each stage alone against the same RFCs—would separate the contribution of chaining from the raw capability of the underlying LLM."],"forward_implications":["A protocol analyst could obtain a first-pass state machine from an RFC alone, before any source code or traffic trace is available.","The rulebook output is structured enough to hand to downstream verification or fuzzing tools without additional parsing.","The recall-over-precision tilt is the safe direction for security work: extra transitions can be filtered later, but a missed transition is invisible to any downstream analysis.","The small spread between FTP and RTSP results suggests the same three-stage chain may carry across protocol families with different document styles."],"supporting_citations":[{"why":"Prior LLM pipeline that extracts FSMs from RFC documents; FlowFSM's design is positioned against it.","marker":"[15]"},{"why":"Prior system extracting FSMs from protocol implementations; it anchors the comparison between specification- and implementation-derived machines.","marker":"[14]"},{"why":"Prior hybrid NLP method for FSM extraction from RFCs; used to motivate the need for better handling of natural-language specifications.","marker":"[22]"},{"why":"Prior neural-symbolic FSM synthesis from specifications; another baseline for specification-to-FSM extraction.","marker":"[23]"},{"why":"Source of the chain-of-thought / reasoning prompting technique that FlowFSM adopts.","marker":"[19]"},{"why":"Source of the prompt-chaining principle: decomposing a task into modular steps whose outputs feed the next prompt.","marker":"[20]"}],"fun_headline_variants":["Chained prompt agents extract protocol FSMs from RFCs","FlowFSM: prompt chaining for precise FSM extraction","Agentic LLM flow builds state machines from RFC text","Chained prompts decode RFCs into protocol state machines","LLM agent chain extracts FSMs from RFCs with few hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground-truth labels for every reported precision and recall number come from the authors' own manual reading of the RFCs, with no published transition list or second annotator; if those judgments are off, every metric loses its anchor.","fun_headline_variants_meta":{"raw":{"variants":["Chained prompt agents extract protocol FSMs from RFCs","FlowFSM: prompt chaining for precise FSM extraction","Agentic LLM flow builds state machines from RFC text","Chained prompts decode RFCs into protocol state machines","LLM agent chain extracts FSMs from RFCs with few hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000532,"raw_usage":{"total_tokens":2493,"prompt_tokens":808,"completion_tokens":1685,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":424,"completion_tokens_details":{"reasoning_tokens":1602}},"tokens_in":424,"tokens_out":1685,"duration_ms":15563,"temperature":1.0,"reasoning_tokens":1602,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:12:22.213267+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Publish the full set of true FTP and RTSP transitions before running the pipeline, then have a separate annotator or a reference implementation score the same outputs; if precision or recall falls materially under that pre-registered ground truth, the claim of high-fidelity extraction is not supported.","supporting_citations":[{"cited_title":"Prosper: Extracting protocol specifica- tions using large language models,","cited_arxiv_id":null,"evidence_quote":"Prior LLM pipeline that extracts FSMs from RFC documents; FlowFSM's design is positioned against it."},{"cited_title":"Unleashing the Power of LLM to Infer State Machine from the Protocol Implementation","cited_arxiv_id":"2405.00393","evidence_quote":"Prior system extracting FSMs from protocol implementations; it anchors the comparison between specification- and implementation-derived machines."},{"cited_title":"Automated attack synthesis by extracting finite state machines from protocol specification documents,","cited_arxiv_id":null,"evidence_quote":"Prior hybrid NLP method for FSM extraction from RFCs; used to motivate the need for better handling of natural-language specifications."},{"cited_title":"Hermes: unlocking security analysis of cellular network protocols by synthesizing finite state machines from natural language specifications,","cited_arxiv_id":null,"evidence_quote":"Prior neural-symbolic FSM synthesis from specifications; another baseline for specification-to-FSM extraction."},{"cited_title":"Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts,","cited_arxiv_id":null,"evidence_quote":"Source of the prompt-chaining principle: decomposing a task into modular steps whose outputs feed the next prompt."}],"review_version":1}