{"id":"5a731ee5-2f79-402b-acfe-9b4f9ac57eef","arxiv_id":"2608.09526","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RangeFactory automatically builds 1,148 validated multi-hop cyber ranges from 239 single-vulnerability environments, then shows frontier LLM agents lose 24.5 to 47.0 percent of runs after the first compromise.","lead":"This paper presents a system that automatically stitches isolated software vulnerabilities into multi-step attack chains inside realistic fake networks, and it validates each chain by actually running the attack. The authors then test four AI agents on these chains and find that even after breaking into the first machine, 24.5 to 47.0 percent of runs fail later in the chain.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The closed six-type ontology and Eq. 5 gate pruning before validation, so missing capabilities silently remove potentially constructible chains; this recall/coverage assumption is currently untested.","rationale":"The reader's weakest assumption correctly identifies the closed six-type capability ontology and the fixed closure rule (Eq. 5) as the load-bearing abstraction. My reading confirms this: the Composer prunes proposals before the end-to-end validator, so any missing capability type or missed probe removes a chain from the candidate set permanently. This is a recall/coverage concern rather than a soundness concern; the accepted 1,148 ranges are backed by execution witnesses, an independent audit, and a conservative chain-grouped statistical treatment. The paper deserves credit for those checks, and the constructive evidence supporting every retained range is genuine. The weakness is that no experiment measures what fraction of valid chains the abstraction excludes, and the 8% false-rejection estimate only probes Executor capability, not ontology completeness. This does not overturn the conditional verdict: the framework may still be a scalable and useful construction method, but the headline numbers should be read as conditional on an unverified completeness assumption. Public artifacts and a recall-oriented validation test would be needed to move from CONDITIONAL to ACCEPT. I therefore leave the reader's verdict unchanged.","tokens_in":19123,"tokens_out":4899,"duration_ms":51916,"concrete_test":"Extract 100 proposals rejected at a slot because the accumulated prefix state did not satisfy an Atom's required capability, excluding rejections due to runtime or static checks. Independently annotate, using only the packaged PoCs and CVE descriptions, whether the preceding compromise actually confers the missing capability in practice. For every confirmed case, add the corresponding verified grant to the Atom (or extend the ontology with a new capability type and probe adapter) and rerun composition and end-to-end validation on those chains only, with the same templates and protocol. If more than a small fraction (e.g., >5%) of the confirmed-capability chains deploy and pass end-to-end validation, the closed ontology demonstrably prunes valid chains and the 1,148-instance count is a biased underestimate; if none validate, the ontology is adequate for the sampled source set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The accepted ranges are supported by constructive witnesses, so the soundness of the 1,148 instances is not in question. The load-bearing weakness is the recall side of the funnel: Stage 2 Composer prunes 21,949 of 23,789 proposals before deployment, using only the six capability types in Table 8 and the fixed closure rule (Eq. 5). Any capability an exploit actually confers that is not in that ontology, or any probe the Explorer fails to run, is invisible to the satisfaction relation in Eq. 1, so the chain is discarded before the end-to-end validator ever sees it. The paper's 62.4% validation rate and the 1,148-instance benchmark are therefore conditional on the hand-authored ontology and probe set being complete for the sampled CVEs. Nothing in Section 4 measures this: the audit in Table 19 checks accepted ranges and Atom annotations, not whether rejected proposals would have validated if the missing capability were represented. The false-rejection estimate in Section 5 (8% recovered by Kimi-K3) is a lower bound that tests Executor strength, not ontology completeness. Thus, if the ontology is incomplete, the central scalability claim understates the number of constructible ranges and may systematically exclude entire classes of multi-hop attacks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RangeFactory is a multi-agent framework that automatically converts isolated, single-vulnerability CVE environments into multi-hop cyber ranges. The pipeline has three stages: Atomization, in which an Exploiter attacks each CVE environment and an Explorer probes post-exploitation capabilities to produce a CVE Capability Atom with a fixed capability ontology; Orchestration, in which a Generator creates enterprise-style network templates and a Composer binds compatible Atoms into candidate attack chains using capability matching and static runtime checks; and Validation, in which an Executor attempts the complete end-to-end attack and a private verifier checks ordered objectives, with a Diagnoser routing failures to bounded retry or persistent incompatibility records. The paper reports 239 Atoms, 1,840 candidates, and 1,148 validated range instances spanning 287 ordered attack chains (62.4% validation rate), plus a corpus of 5,541 outcome-annotated trajectories. It also evaluates four attack agents and reports a sustained-compromise gap: among runs that compromise the entry vulnerability, 24.5–47.0% fail to complete the remaining path. The manuscript includes a detailed formal model of capability and runtime dependencies, construction ablations, an independent audit of 150 accepted ranges, conservative chain-group uncertainty bounds, and explicit statements of limitations.","tokens_in":19341,"tokens_out":5148,"duration_ms":51867,"significance":"If the construction claims hold, RangeFactory is a useful infrastructure contribution: it removes per-range expert binding, deployment, and success annotation, prunes 92.3% of proposals before deployment, and produces accepted ranges with constructive execution witnesses. The paper is unusually transparent about the conditional nature of its validation protocol, and it ships typed algorithms, a formal compatibility gate, a documented automation boundary, and conservative statistical bounds that do not overstate within-task variance. The main scientific value is in demonstrating that execution-derived dependency extraction, template-guided composition, and end-to-end validation can together yield a large corpus of validated multi-hop ranges. The principal weakness is that the recall side of the construction funnel — the completeness of the hand-authored capability ontology and probe set — is not measured, so the reported yield is conditional on an untested abstraction. Overall, the central construction claim is defensible but needs one additional experiment to substantiate its scalability generalization.","major_comments":[{"comment":"The pre-deployment prune in Stage 2 is gated entirely by the closed six-type capability ontology, the fixed closure rule of Eq. (5), and the Explorer's probe results, yet the manuscript reports no measurement of the recall of this abstraction. The Composer rejects 21,949 of 23,789 proposals (92.3%) before end-to-end validation, and the Section 5 false-rejection check (8.0% recovered by Kimi-K3) only measures Executor strength on candidates that already passed the capability gate. If a real vulnerability confers a capability outside the six types, or if the Explorer's probes fail to confirm a capability the exploit actually grants, the corresponding chains never reach the validator, so the reported 62.4% validation rate and 1,148-instance benchmark are conditional on an untested completeness assumption. I recommend a rejected-proposal audit (for example, manually inspecting a sample of the 21,949 rejections to estimate how many would have been accepted under an expanded ontology) or a probe-completeness experiment in which known ground-truth capabilities are checked against Explorer output. The accepted-set soundness is not in question; the recall side of the funnel is.","section":"§3.2, §B.3, Eq. (5), Table 2"},{"comment":"The sustained-compromise gap is a headline empirical claim, but the conditional continuation metric of Eq. (14) is reported as a point estimate with no uncertainty interval. The paper's own paired chain-group bound in Table 18 has a 95% half-width of 16.0 percentage points (because the chain-level variable lies in [-1,1] for 287 groups), while the reported depth drops from hop-1 to end-to-end success are 18.1, 21.6, 21.6, and 20.8 points for the four models (73.8% to 55.7%, 62.8% to 41.2%, 57.8% to 36.2%, and 44.3% to 23.5%, respectively). These differences are comparable to or larger than the paired bound, so the statement that 24.5–47.0% of foothold-establishing runs fail on later hops is not strongly supported by the reported statistics. Either repeated rollouts per instance or a confidence interval for the conditional failure proportions is needed before this can be presented as a robust finding.","section":"§4.2, Eq. (14), Table 18"}],"minor_comments":[{"comment":"The flow from the Diagnoser to the Composer is not explicitly labeled in Figure 1; adding a label such as 'incompatibility records' would make the closed feedback loop easier to follow.","section":"Figure 1"},{"comment":"The arithmetic relating 460 candidate groups to 287 validated groups could be clarified: 460 groups × 4 contexts = 1,840 candidates and 287 groups × 4 contexts = 1,148 instances, but the 173 groups with no validated context are not discussed. Reporting how many groups had partial validation across the four contexts would help readers interpret chain-level attrition.","section":"§4.1, Eq. (10)"},{"comment":"The temperature difference for Kimi-K3 is stated as the endpoint minimum, but this constraint is easy to miss in the main-text evaluation; moving the note from the appendix to the main-text experimental setup would prevent readers from misinterpreting cross-model comparisons.","section":"Table 17"},{"comment":"The conservative Hoeffding bound is well suited to the matched-context design, but the sentence 'the adjacent model gaps are 14.5, 5.0, and 12.7 points' would be clearer if it explicitly stated that these gaps are not all beyond the 16.0-point paired bound, which the following sentence does say.","section":"§F.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a security or systems venue and is unusually careful in disclosing the conditional nature of its validation protocol. The main risk for the review process is that the recall-side completeness issue could be seen as disqualifying; I do not think it is, because the accepted set is supported by constructive witnesses and the authors already frame rejection as 'unrecovered, not invalid.' A rejected-proposal audit would substantially strengthen the scalability claim. I would also ask the editor to ensure the future-dated references (2026) are appropriate for this arXiv posting and that the reported release controls are verified if artifacts are to be made available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core result stands: they turn isolated CVE environments into validated multi-hop ranges without per-range expert annotation, by extracting dependencies from actual attacks and then running end-to-end validation. The 1,148-instance benchmark and the 24.5–47.0% sustained-compromise gap are measured outcomes, and the paper handles statistics honestly—chain-grouped Hoeffding bounds, explicit single-rollout caveat, and a conservative interpretation of the benchmark protocol.\n\nWhat's genuinely good: the construction funnel is typed and auditable, the 150-range audit (97.3% behavior preservation) gives precision evidence, and the ablation shows capability matching is what does the work. They also control for the validator-overlap concern—DeepSeek-V4-Pro builds and validates yet ranks below Kimi-K3 and GLM-5.2. The trajectory corpus is a useful side product.\n\nThe soft spot is the one the stress-test flags. Stage 2 prunes 92.3% of proposals before any validation, using a closed six-type capability ontology and a fixed closure rule (Eq. 5). If an exploit grants a capability outside that ontology, or if the Explorer's probes miss one, valid chains get discarded before the end-to-end validator sees them. The audit checks accepted ranges, not rejected proposals, so recall is untested. The 8% false-rejection estimate with Kimi-K3 measures Executor strength, not ontology completeness. This is a real limitation, but I would not call it fatal: the paper claims scalable construction, not exhaustive coverage, and every accepted range has a constructive witness. Still, the 62.4% validation rate is conditional on the ontology being adequate for the sampled CVEs, and no experiment shows how many chains are lost to missing capability types.\n\nSecond soft spot: no code or data released. The aggregate claims cannot be independently checked, and the benchmark is protocol-relative. That is a reproduction problem, not necessarily a correctness one.\n\nThis paper should get a serious referee. If I were editing, I'd send it out and ask for two things: release artifacts or at least a substantial sample, and add a recall probe—deliberately extend the ontology or run broader capability extraction on rejected proposals to estimate how many would validate. Who benefits: anyone building cyber ranges from CVE pools and anyone evaluating whether LLM agents can sustain multi-hop attacks. I'd cite it if I worked in that area.\n\nRecommendation: accept for peer review, with the recall question on the table.","headline":"RangeFactory is a serious systems contribution—automated multi-hop range construction with real validation—but its recall depends on an untested hand-authored capability ontology; still deserves refereeing.","tokens_in":19892,"tokens_out":2504,"would_cite":true,"duration_ms":23694,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RangeFactory claims it can turn isolated vulnerability environments into end-to-end validated multi-hop attack ranges, reporting 1,148 validated instances from 239 single-exploit atoms.","keywords":["multi-hop cyber range","CVE Capability Atom","dependency resolution","end-to-end validation","LLM attack agents","sustained compromise","attack chain composition","cyber range orchestration"],"falsifier":"Construct a known-good multi-hop chain whose connecting step depends on a capability outside the six-type ontology, for example a CVE granting only the ability to write one specific configuration file that a later exploit consumes, and run it through RangeFactory. If the pipeline prunes the chain as incompatible even though a human operator can complete it, the ontology is incomplete; running the Explorer's probes against all atoms and auditing whether every real post-exploit capability appears as a verified grant would bound the miss rate.","tokens_in":18922,"feed_emoji":"🕸️","tokens_out":7720,"duration_ms":66113,"temperature":0.7,"pith_summary":"RangeFactory claims that isolated single-vulnerability environments can be automatically assembled into end-to-end validated multi-hop cyber ranges without per-range expert annotation of how the vulnerabilities connect. The method first attacks each vulnerability environment to extract, from real successful exploits, what the exploit requires and what verified capability it grants; it then composes those capabilities along enterprise-style network templates while pruning incompatible bindings; and finally it deploys each composed candidate and executes the complete attack, retaining only ranges with a verifier-confirmed path. The paper reports 239 CVE Capability Atoms and 20 network configurations producing 1,840 candidate ranges, of which 1,148 passed end-to-end validation and span 287 distinct ordered attack chains, a 62.4% validation rate. On the resulting benchmark, between 24.5% and 47.0% of runs that compromise the entry vulnerability still fail to complete the remaining multi-hop path, a sustained-compromise gap that motivates the infrastructure.","feed_headline":"Auto-pipeline turns single CVEs into 1,148 validated attack chains","feed_subtitle":"Multi-hop ranges built without per-chain expert design reveal a steep drop-off after the first foothold.","key_machinery":"The load-bearing object is the CVE Capability Atom, a package that binds a reproducible vulnerability environment to four interfaces: exploit_access, the structured preconditions under which the exploit was exercised; capability_grants, the probe-verified post-compromise capabilities drawn from a fixed six-type ontology of command execution, file read and write, network vantage, credential discovery, and authentication; exploit_guide, the successful procedure and observed runtime requirements; and poc_materials, the artifacts it references. The Composer consumes these atoms as interfaces, matching each atom's requirements against the accumulated capability state of the chain prefix and using one closure rule, that verified command execution implies file read and network vantage under the same principal, to derive additional grants without free-form agent inference. End-to-end validation then exercises the actual guides inside the composed network, making the deployed attack itself the final check on runtime dependencies that only appear after joint deployment.","core_discovery":"The central claim is that multi-hop cyber-range construction reduces to dependency resolution, and that a three-stage pipeline can resolve it at scale. Atomization converts each successfully exploited vulnerability environment into a CVE Capability Atom that records the exploit's preconditions, verified post-exploitation capability grants, the working attack procedure, and required materials. Orchestration binds these atoms into candidate attack chains by matching grants to requirements along template-defined slots and closes capabilities with a fixed rule, rejecting 92.3% of proposals before deployment. Validation then deploys each remaining candidate and runs the full chain end to end under hidden ordered objectives, with a Diagnoser distinguishing attack-execution failures from chain-composition failures and feeding evidence back into composition. Only candidates with a witnessed, verifier-confirmed path are accepted, yielding 1,148 validated range instances across 287 distinct ordered attack chains.","pith_inferences":["A testable extension is to grow the capability ontology: adding types for partial or artifact-specific write access, protocol tunneling, or orchestrated multi-host effects would let the Composer keep chains that its current closed ontology prunes, and the paper itself states that the ontology is closed, marking this as the main scalability boundary.","The same dependency-resolution pipeline could be applied beyond CVE exploits to misconfigurations, credential-reuse chains, or multi-host service interactions, provided each step can be packaged with verified grants and a reusable procedure.","Because the reported experiments cover only linear three-slot chains, branching or merging attack graphs remain unvalidated; if the atom interface extends to general acyclic dependency graphs, the framework could move from benchmark construction toward automated scenario generation for red-team practice."],"forward_implications":["Existing collections of single-vulnerability environments become a renewable source of multi-hop ranges: each new atom can be bound into many templates, so benchmark size can track the supply of verified exploits rather than hand-authored scenarios.","RangeBench's 1,148 validated instances let evaluations measure sustained multi-hop compromise on matched one-, two-, and three-hop objectives within the same rollout, separating entry-exploit difficulty from follow-through.","The reported 24.5–47.0% drop-off after the entry compromise implies that agent evaluations stopping at the first foothold overstate practical attack capability, since attack depth and network scale both erode success.","The 5,541 outcome-annotated trajectories, pairing guide-assisted demonstrations with autonomous runs on identical ranges and verifier-labeled milestone turns, provide execution-grounded data for studying where agents lose the thread and for future training.","The 8.0% false-rejection estimate under an alternative validator means reported pass@1 numbers are conditional on the constructing validation protocol, so benchmark scores carry a validator-dependence caveat."],"supporting_citations":[{"why":"supplies the single-vulnerability task packages that RangeFactory screens as its input environment pool.","marker":"Luo et al. 2026"},{"why":"provides the deployable vulnerability environments whose reproduction materials feed Atomization.","marker":"Vulhub 2026"},{"why":"represents the executable single-vulnerability benchmark line that RangeFactory extends into multi-hop ranges.","marker":"Zhu et al. 2025"},{"why":"defines the small fixed-collection multi-host benchmarks whose manual construction bottleneck RangeFactory targets.","marker":"Liu et al. 2026"},{"why":"prior automated scenario generation that still requires expert-specified composition semantics, the limitation RangeFactory removes.","marker":"Skandylas and Asplund 2026"},{"why":"motivates the need for multi-step cyber attack scenarios, framing the problem RangeFactory's ranges are built to serve.","marker":"Folkerts et al. 2026"},{"why":"shows automatic reproduction of CVE entries into executable tasks, one route to building the atom library.","marker":"Ullah et al. 2026"}],"fun_headline_variants":["Dependency resolution turns CVEs into 1,148 validated attack chains","Multi-hop ranges at scale: 287 chains from single-host exploits","Foothold isn't enough: 24.5-47% fail after first compromise","RangeFactory auto-validates 1,148 attack chains from CVE atoms","Scaling cyber ranges: dependency resolution builds 1,148 chains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system's completeness hinges on its closed six-type capability ontology and fixed closure rule: if a real exploit grants or requires a capability outside that list, the Composer will silently discard the chain before end-to-end validation can ever test it.","fun_headline_variants_meta":{"raw":{"variants":["Dependency resolution turns CVEs into 1,148 validated attack chains","Multi-hop ranges at scale: 287 chains from single-host exploits","Foothold isn't enough: 24.5-47% fail after first compromise","RangeFactory auto-validates 1,148 attack chains from CVE atoms","Scaling cyber ranges: dependency resolution builds 1,148 chains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1602,"prompt_tokens":960,"completion_tokens":642,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":542}},"tokens_in":576,"tokens_out":642,"duration_ms":5819,"temperature":1.0,"reasoning_tokens":542,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:37:31.258207+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a known-good multi-hop chain whose connecting step depends on a capability outside the six-type ontology, for example a CVE granting only the ability to write one specific configuration file that a later exploit consumes, and run it through RangeFactory. If the pipeline prunes the chain as incompatible even though a human operator can complete it, the ontology is incomplete; running the Explorer's probes against all atoms and auditing whether every real post-exploit capability appears as a verified grant would bound the miss rate.","supporting_citations":[],"review_version":1}