Pith. sign in

REVIEW 2 major objections 4 minor 12 references

RangeFactory: Scalable Construction of Multi-Hop Cyber Ranges

T0 review · 2 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RangeFactory claims it can turn isolated vulnerability environments into end-to-end validated multi-hop attack ranges, reporting 1,148 validated instances from 239 single-exploit atoms.

desk verdict RangeFactory is a serious systems contribution—automated multi-hop range construction with real validation—but its recall depends on an untested hand-authored capability ontology; still deserves refereeing. read the letter →

arxiv 2608.09526 v1 pith:NDP6TTR6 submitted 2026-08-10 cs.CR cs.AI

Hanlin Jiang , Puyi Wang , Jiandong Jin , Shaofei Li , Zhan Shen , Pengli Wang , Ziming Wang , Yifeng Cai
show 5 more authors
Ning Jia Yuxin Ren Peng Jiang Yao Guo Ding Li
This is my paper · ORCID
classification cs.CRcs.AI
keywords multi-hopcyberrangeCVECapabilityAtomdependencyresolutionend-to-endvalidationLLMattackagentssustainedcompromisechaincompositionorchestration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RangeFactory claims that isolated single-vulnerability environments can be automatically assembled into end-to-end validated multi-hop cyber ranges without per-range expert annotation of how the vulnerabilities connect. The method first attacks each vulnerability environment to extract, from real successful exploits, what the exploit requires and what verified capability it grants; it then composes those capabilities along enterprise-style network templates while pruning incompatible bindings; and finally it deploys each composed candidate and executes the complete attack, retaining only ranges with a verifier-confirmed path. The paper reports 239 CVE Capability Atoms and 20 network configurations producing 1,840 candidate ranges, of which 1,148 passed end-to-end validation and span 287 distinct ordered attack chains, a 62.4% validation rate. On the resulting benchmark, between 24.5% and 47.0% of runs that compromise the entry vulnerability still fail to complete the remaining multi-hop path, a sustained-compromise gap that motivates the infrastructure.

What carries the argument

The load-bearing object is the CVE Capability Atom, a package that binds a reproducible vulnerability environment to four interfaces: exploit_access, the structured preconditions under which the exploit was exercised; capability_grants, the probe-verified post-compromise capabilities drawn from a fixed six-type ontology of command execution, file read and write, network vantage, credential discovery, and authentication; exploit_guide, the successful procedure and observed runtime requirements; and poc_materials, the artifacts it references. The Composer consumes these atoms as interfaces, matching each atom's requirements against the accumulated capability state of the chain prefix and using one closure rule, that verified command execution implies file read and network vantage under the same principal, to derive additional grants without free-form agent inference. End-to-end validation then exercises the actual guides inside the composed network, making the deployed attack itself the final check on runtime dependencies that only appear after joint deployment.

What would settle it

Construct a known-good multi-hop chain whose connecting step depends on a capability outside the six-type ontology, for example a CVE granting only the ability to write one specific configuration file that a later exploit consumes, and run it through RangeFactory. If the pipeline prunes the chain as incompatible even though a human operator can complete it, the ontology is incomplete; running the Explorer's probes against all atoms and auditing whether every real post-exploit capability appears as a verified grant would bound the miss rate.

Watch

Extended reading notes

Core claim

The central claim is that multi-hop cyber-range construction reduces to dependency resolution, and that a three-stage pipeline can resolve it at scale. Atomization converts each successfully exploited vulnerability environment into a CVE Capability Atom that records the exploit's preconditions, verified post-exploitation capability grants, the working attack procedure, and required materials. Orchestration binds these atoms into candidate attack chains by matching grants to requirements along template-defined slots and closes capabilities with a fixed rule, rejecting 92.3% of proposals before deployment. Validation then deploys each remaining candidate and runs the full chain end to end under hidden ordered objectives, with a Diagnoser distinguishing attack-execution failures from chain-composition failures and feeding evidence back into composition. Only candidates with a witnessed, verifier-confirmed path are accepted, yielding 1,148 validated range instances across 287 distinct ordered attack chains.

Load-bearing premise

The system's completeness hinges on its closed six-type capability ontology and fixed closure rule: if a real exploit grants or requires a capability outside that list, the Composer will silently discard the chain before end-to-end validation can ever test it.

Editorial extensions

If this is right

  • Existing collections of single-vulnerability environments become a renewable source of multi-hop ranges: each new atom can be bound into many templates, so benchmark size can track the supply of verified exploits rather than hand-authored scenarios.
  • RangeBench's 1,148 validated instances let evaluations measure sustained multi-hop compromise on matched one-, two-, and three-hop objectives within the same rollout, separating entry-exploit difficulty from follow-through.
  • The reported 24.5–47.0% drop-off after the entry compromise implies that agent evaluations stopping at the first foothold overstate practical attack capability, since attack depth and network scale both erode success.
  • The 5,541 outcome-annotated trajectories, pairing guide-assisted demonstrations with autonomous runs on identical ranges and verifier-labeled milestone turns, provide execution-grounded data for studying where agents lose the thread and for future training.
  • The 8.0% false-rejection estimate under an alternative validator means reported pass@1 numbers are conditional on the constructing validation protocol, so benchmark scores carry a validator-dependence caveat.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to grow the capability ontology: adding types for partial or artifact-specific write access, protocol tunneling, or orchestrated multi-host effects would let the Composer keep chains that its current closed ontology prunes, and the paper itself states that the ontology is closed, marking this as the main scalability boundary.
  • The same dependency-resolution pipeline could be applied beyond CVE exploits to misconfigurations, credential-reuse chains, or multi-host service interactions, provided each step can be packaged with verified grants and a reusable procedure.
  • Because the reported experiments cover only linear three-slot chains, branching or merging attack graphs remain unvalidated; if the atom interface extends to general acyclic dependency graphs, the framework could move from benchmark construction toward automated scenario generation for red-team practice.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. RangeFactory is a multi-agent framework that automatically converts isolated, single-vulnerability CVE environments into multi-hop cyber ranges. The pipeline has three stages: Atomization, in which an Exploiter attacks each CVE environment and an Explorer probes post-exploitation capabilities to produce a CVE Capability Atom with a fixed capability ontology; Orchestration, in which a Generator creates enterprise-style network templates and a Composer binds compatible Atoms into candidate attack chains using capability matching and static runtime checks; and Validation, in which an Executor attempts the complete end-to-end attack and a private verifier checks ordered objectives, with a Diagnoser routing failures to bounded retry or persistent incompatibility records. The paper reports 239 Atoms, 1,840 candidates, and 1,148 validated range instances spanning 287 ordered attack chains (62.4% validation rate), plus a corpus of 5,541 outcome-annotated trajectories. It also evaluates four attack agents and reports a sustained-compromise gap: among runs that compromise the entry vulnerability, 24.5–47.0% fail to complete the remaining path. The manuscript includes a detailed formal model of capability and runtime dependencies, construction ablations, an independent audit of 150 accepted ranges, conservative chain-group uncertainty bounds, and explicit statements of limitations.

Significance. If the construction claims hold, RangeFactory is a useful infrastructure contribution: it removes per-range expert binding, deployment, and success annotation, prunes 92.3% of proposals before deployment, and produces accepted ranges with constructive execution witnesses. The paper is unusually transparent about the conditional nature of its validation protocol, and it ships typed algorithms, a formal compatibility gate, a documented automation boundary, and conservative statistical bounds that do not overstate within-task variance. The main scientific value is in demonstrating that execution-derived dependency extraction, template-guided composition, and end-to-end validation can together yield a large corpus of validated multi-hop ranges. The principal weakness is that the recall side of the construction funnel — the completeness of the hand-authored capability ontology and probe set — is not measured, so the reported yield is conditional on an untested abstraction. Overall, the central construction claim is defensible but needs one additional experiment to substantiate its scalability generalization.

major comments (2)
  1. [§3.2, §B.3, Eq. (5), Table 2] The pre-deployment prune in Stage 2 is gated entirely by the closed six-type capability ontology, the fixed closure rule of Eq. (5), and the Explorer's probe results, yet the manuscript reports no measurement of the recall of this abstraction. The Composer rejects 21,949 of 23,789 proposals (92.3%) before end-to-end validation, and the Section 5 false-rejection check (8.0% recovered by Kimi-K3) only measures Executor strength on candidates that already passed the capability gate. If a real vulnerability confers a capability outside the six types, or if the Explorer's probes fail to confirm a capability the exploit actually grants, the corresponding chains never reach the validator, so the reported 62.4% validation rate and 1,148-instance benchmark are conditional on an untested completeness assumption. I recommend a rejected-proposal audit (for example, manually inspecting a sample of the 21,949 rejections to estimate how many would have been accepted under an expanded ontology) or a probe-completeness experiment in which known ground-truth capabilities are checked against Explorer output. The accepted-set soundness is not in question; the recall side of the funnel is.
  2. [§4.2, Eq. (14), Table 18] The sustained-compromise gap is a headline empirical claim, but the conditional continuation metric of Eq. (14) is reported as a point estimate with no uncertainty interval. The paper's own paired chain-group bound in Table 18 has a 95% half-width of 16.0 percentage points (because the chain-level variable lies in [-1,1] for 287 groups), while the reported depth drops from hop-1 to end-to-end success are 18.1, 21.6, 21.6, and 20.8 points for the four models (73.8% to 55.7%, 62.8% to 41.2%, 57.8% to 36.2%, and 44.3% to 23.5%, respectively). These differences are comparable to or larger than the paired bound, so the statement that 24.5–47.0% of foothold-establishing runs fail on later hops is not strongly supported by the reported statistics. Either repeated rollouts per instance or a confidence interval for the conditional failure proportions is needed before this can be presented as a robust finding.
minor comments (4)
  1. [Figure 1] The flow from the Diagnoser to the Composer is not explicitly labeled in Figure 1; adding a label such as 'incompatibility records' would make the closed feedback loop easier to follow.
  2. [§4.1, Eq. (10)] The arithmetic relating 460 candidate groups to 287 validated groups could be clarified: 460 groups × 4 contexts = 1,840 candidates and 287 groups × 4 contexts = 1,148 instances, but the 173 groups with no validated context are not discussed. Reporting how many groups had partial validation across the four contexts would help readers interpret chain-level attrition.
  3. [Table 17] The temperature difference for Kimi-K3 is stated as the endpoint minimum, but this constraint is easy to miss in the main-text evaluation; moving the note from the appendix to the main-text experimental setup would prevent readers from misinterpreting cross-model comparisons.
  4. [§F.4] The conservative Hoeffding bound is well suited to the matched-context design, but the sentence 'the adjacent model gaps are 14.5, 5.0, and 12.7 points' would be clearer if it explicitly stated that these gaps are not all beyond the 16.0-point paired bound, which the following sentence does say.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark rests on constructive execution witnesses, not on inputs fitted to the reported numbers.

full rationale

RangeFactory's derivation chain is not circular. The dependency model (Eqs. 1–5, Tables 7–8) is a hand-authored abstraction; Eq. 5 is an explicit closure axiom, not an equation fitted to the reported validation counts. The pipeline separates construction from validation: the Composer prunes 21,949 of 23,789 proposals before deployment (Section 4.1), and Validation accepts only candidates satisfying Eq. 9, which requires EnvironmentReady, GraphValid, ChainConnectivity, GuidedWitness, and PrivateObjectives. The 62.4% validation rate and the 1,148-instance benchmark are measured outcomes, not predictions derived from inputs. The same-model overlap (DeepSeek-V4-Pro constructs and is evaluated) is disclosed and controlled: Section 5 reports that DeepSeek-V4-Pro ranks below Kimi-K3 and GLM-5.2, and Kimi-K3 revalidation of 150 rejections recovers only 8.0%, so construction-role advantage does not force the evaluation results. The closed capability ontology (Table 8) and Eq. 5 are a stated completeness assumption affecting recall, but the paper explicitly says rejected candidates are not labeled invalid (Section 5) and that the benchmark is a pass@1 value conditional on a protocol-validated task distribution (Section F.5); that is a modeling limitation, not a circular reduction. No load-bearing step is justified solely by a self-citation or by redefining a fitted parameter as a prediction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

No numerical model is fitted; the reported success rates and agent scores are measured outcomes. The central abstraction is the CVE Capability Atom with a hand-authored six-type ontology and the closure rule in Eq. (5); these are domain assumptions rather than fitted parameters. The protocol's acceptance oracle is an additional domain assumption. Statistical inference uses standard Hoeffding bounds with chain groups as independent units.

assumptions (4)
  • domain assumption The closed six-type capability ontology (execute_command, read_file, write_file, network_vantage, read_credential, authenticate) is sufficient to express all cross-vulnerability dependencies that matter for composing the chosen CVE environments.
    Adopted in Section 3.2 and Table 8. If a real exploit grants a capability not in the ontology, the Composer can misclassify the chain and prune it before validation.
  • domain assumption Closure rule (Eq. 5): verified command execution on a host implies read_file and network_vantage for the same principal within allowed files and networks, and does not imply credentials or authentication.
    Appendix B.3, Eq. (5). This rule drives capability propagation; an incorrect rule changes which candidates are composed.
  • domain assumption The guided Executor using DeepSeek-V4-Pro with Atom artifacts, 300 turns, and one hour is an adequate witness oracle for declaring a range validated.
    Used in Section 4.1 and Algorithm 2. Acceptance is constructive evidence under a fixed protocol; the paper acknowledges 8.0% false rejection on revalidation.
  • standard math The 287 attack-chain groups are independent units for the Hoeffding-based statistical bounds.
    Appendix F.4 applies Hoeffding's inequality with G=287 chain groups. The independence assumption is reasonable but not empirically verified.
invented entities (1)
  • CVE Capability Atom independent evidence
    purpose: A reusable package that binds a deployable vulnerability environment to normalized exploit preconditions, verified post-exploitation capability grants, an exploit guide, and PoC materials, enabling dependency-aware composition.
    It is a formal data object (Appendix B.2, Eq. 3). Its claims are testable because each grant must be backed by a probe with evidence_level=verified, so the construct has an independent falsifiable handle through actual execution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RangeFactory: Scalable Construction of Multi-Hop Cyber Ranges." pith.science (2026). https://pith.science/paper/NDP6TTR6

@misc{pith2026260809526,
  author       = {Pith},
  title        = {Pith review of: RangeFactory: Scalable Construction of Multi-Hop Cyber Ranges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NDP6TTR6}},
  note         = {Machine review of arXiv:2608.09526}
}
read the original abstract

Real-world cyberattacks often require sustained progress across multiple hosts and network segments, making multi-hop cyber ranges essential infrastructure for studying and improving LLM agents' ability to sustain complete attack chains. Prior work has scaled isolated vulnerability tasks and constructed multi-host scenarios from manually specified vulnerability semantics. However, they are still unable to automatically orchestrate the growing supply of vulnerability environments into end-to-end validated multi-hop ranges. To this end, we present RangeFactory, an automated cyber-range orchestration framework that constructs multi-hop cyber ranges at scale from isolated vulnerability environments. RangeFactory formulates range construction as dependency resolution: it extracts dependency information from agents' actual attacks against real vulnerabilities, resolves known dependencies through template-guided orchestration, and uses end-to-end attack execution to validate runtime dependencies that emerge after composition. Using RangeFactory, we construct RangeBench with 1,148 validated range instances spanning 287 distinct attack chains and evaluate frontier attack agents across attack depth, network scale, and task information. Among runs that compromise the entry vulnerability, 24.5-47.0% still fail to complete the remaining attack path, revealing a substantial sustained-compromise gap between establishing an initial foothold and completing a multi-hop attack. RangeFactory further produces a corpus of 5,541 outcome-annotated multi-hop attack trajectories, providing execution data for attack-process analysis and future agent training.

Figures

Figures reproduced from arXiv: 2608.09526 by the authors.

Figure 1
Figure 1. Overview of RangeFactory’s multi-agent workflow for constructing validated multi-hop cyber ranges. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. RangeBench factor analysis across four attack agents. (a) Success at successive attack depths under L2 information. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 5 canonical work pages

  1. [1]

    Automatingthe Generation of Cyber Range Virtual Scenarios with VSDL

    Costa,G.;Russo,E.;andArmando,A.2022. Automatingthe Generation of Cyber Range Virtual Scenarios with VSDL. Journal of Wireless Mobile Networks, Ubiquitous Comput- ing, and Dependable Applications, 13(4): 61–80. Deng, G.; Liu, Y.; Mayoral-Vilches, V.; Liu, P.; Li, Y.; Xu, Y.; Zhang, T.; Liu, Y.; Pinzger, M.; and Rass, S

    work page 2022

  2. [4]

    AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges

    AgentCyberRange: Bench- marking Frontier AI Systems in Realistic Cyber Ranges. arXiv:2606.14295. Liu, Z.; Huang, L.; Zhang, J.; Liu, D.; Tian, Y.; and Shao, J

    work page Pith review arXiv

  3. [6]

    arXiv:2602.03012

    CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability. arXiv:2602.03012. Lupinacci,M.;Blefari,F.;Romeo,F.;Pironti,F.A.;andFur- faro, A

    arXiv

  4. [8]

    Ullah, S.; Balasubramanian, P.; Guo, W.; Burnett, A.; Pearce, H.; Kruegel, C.; Vigna, G.; and Stringhini, G

    Automated Genera- tionofCybersecurityExerciseScenarios.arXiv:2604.01079. Ullah, S.; Balasubramanian, P.; Guo, W.; Burnett, A.; Pearce, H.; Kruegel, C.; Vigna, G.; and Stringhini, G

    work page arXiv

  5. [9]

    arXiv:2509.01835

    From CVE Entries to Verifiable Exploits: An Au- tomated Multi-Agent Framework for Reproducing CVEs. arXiv:2509.01835. Vulhub

    arXiv

  6. [10]

    arXiv:2506.02548

    CyberGym: Evaluating AI Agents’ Real-World Cybersecurity Capabilities at Scale. arXiv:2506.02548. Zhang,A.K.;Perry,N.;Dulepet,R.;Ji,J.;Menders,C.;Lin, J.;Jones,E.;Hussein,G.;Liu,S.;Jasper,D.;Peetathawatchai, P.; Glenn, A.; Sivashankar, V.; Zamoshchin, D.; Glikbarg, L.; Askaryar, D.; Yang, H.; Zhang, A.; Alluri, R.; Tran, N.; Sangpisit, R.; Oseleononmen, K...

    arXiv

  7. [11]

    In Yue, Y.; Garg, A.; Peng, N.; Sha, F.; and Yu, R., eds.,Inter- national Conference on Learning Representations, volume 2025, 25094–25243

    Cybench: A Framework for Evaluating Cy- bersecurity Capabilities and Risks of Language Models. In Yue, Y.; Garg, A.; Peng, N.; Sha, F.; and Yu, R., eds.,Inter- national Conference on Learning Representations, volume 2025, 25094–25243. Zhu, Y.; Kellermann, A.; Bowman, D.; Li, P.; Gupta, A.; Danda,A.;Fang,R.;Jensen,C.;Ihli,E.;Benn,J.;Geronimo, J.; Dhir, A.;...

    work page 2025

  8. [12]

    CVE-Bench:ABenchmarkforAIAgents’AbilitytoExploit Real-WorldWebApplicationVulnerabilities.InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, 79850–79867. PMLR. Zhuo, T. Y.; Wang, D.; Ding, H.; Kumar, V.; and Wang, Z. 2025a. Cyber-Zero: Training Cybersecurity Agents without Runtime....

    arXiv

Show all 12 references
  1. [2016]

    InProceedings of the 7th Symposium on Information and Communication Technology

    Cyris: A cyber range instantiation system for facilitating se- curity training. InProceedings of the 7th Symposium on Information and Communication Technology. Rizos,G.S.;Kopalidis,N.;Mengidis,N.;Lalas,A.;andVo- tis,K.2025. FromConcepttoDeployment:AnAIAssistant for Generating ...

    2025

  2. [2024]

    arXiv:2404.08144

    LLM Agents can Autonomously Exploit One-day Vulnerabilities. arXiv:2404.08144. Folkerts, L.; Payne, W.; Inman, S.; Giavridis, P.; Skinner, J.; Deverett, S.; Aung, J.; Zorer, E.; Schmatz, M.; Ghanem, M.; Wilkinson, J.; Steer, A.; Hong, V.; and Wang, J

    arXiv

  3. [2025]

    arXiv:2510.11688

    PACEbench: A Framework for Evaluating Practical AI Cyber-Exploitation Capabilities. arXiv:2510.11688. Luo, X.; Zhang, J.; Zhou, S.; Huang, J.; Xiao, C.; Zhu, Q.; Ma, Z.; Yue, X.; Yue, Y.; Zeng, W.; and Che, W

  4. [2026]

    arXiv:2603.11214

    Measuring AI Agents’ Progress on Multi-Step Cyber Attack Scenarios. arXiv:2603.11214. Liu, F.; Dai, J.; Fan, Y.; Mai, W.; Li, Z.; Chen, B.; Zhang, J.; Lou, Z.; Xiang, B.; Zhang, Q.; Pan, X.; Hong, G.; Zhang, Y.; and Yang, M

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.