REVIEW 4 major objections 6 minor 25 references
Signals and Symptoms: ICS Attack Dataset From Railway Cyber Range
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper publicly releases two railway-specific ICS attack datasets, generated by executing end-to-end cyberattacks on a miniature railway cyber range and capturing disk, memory, network, and log evidence.
desk verdict A novel railway-specific ICS attack dataset release, with the validity of the emulated environment left unproven; the artifact is the contribution, and the paper deserves a rigorous referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing artifact is the railway cyber range, a miniature digital-twin testbed arranged as four layered networks—corporate IT, supervisory control, production with programmable logic controllers and remote terminal units, and a physical-world emulator that simulates train movement, signalling, and power supply. The argument runs through the attack-scenario design method, which translates cyber-threat-intelligence reports and historical advanced-persistent-threat patterns into executable chronologies with specific hosts, network segments, and benign background activity. This machinery is what lets the authors capture evidence across memory, disk, network, and logs that is meant to approximate real-world railway ICS attack artifacts without risking live infrastructure.
What would settle it
Mount the released AS1 disk image and AS2 memory images in standard forensic tools and check whether the documented indicators are actually present—the spyTrojan and its installer names, the FCI module and its Modbus coil commands, the S7Comm-injected RTU values, and the C2 communications shown in the case study. If any of these claimed indicators cannot be recovered from the evidence, the datasets do not support the paper's reconstruction of the attack chain.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a deliberately assembled railway cyber range can produce publicly releasable, multi-evidence ICS attack datasets whose indicators are concrete enough to reconstruct the full attack chain. The authors design two end-to-end scenarios: AS1 follows an advanced-persistent-threat-style intrusion using a spy trojan and a false-control-injection module that ultimately disables collision avoidance and crashes a train; AS2 adapts a power-grid attack of the kind seen in 2015, using a web-application vulnerability, a spy trojan, and a false-data-injection script to trigger a railway power outage. The datasets capture the resulting indicators of compromise in memory, network traffic, disk, and logs, and a case study on AS2 shows that the artifacts line up with the documented attack steps, from the malicious image upload to the command-and-control communication to the power-grid failure.
Load-bearing premise
The entire value of the datasets rests on the miniature railway emulator being a faithful enough stand-in for real railway IT and OT operations that the simulated attacks leave behind evidence that resembles what a real intrusion would produce.
Editorial extensions
If this is right
- Forensic analysts can practice memory and disk forensics, network capture analysis, and log review on railway-specific ICS attacks without access to real rail infrastructure.
- Detection tools can be tested against the documented indicators, including the Modbus coil tampering in AS1 and the S7Comm false-data injection in AS2, providing reproducible benchmarks.
- The two end-to-end scenarios demonstrate how IT-side initial access (phishing, web vulnerability) translates into OT-side physical consequences, giving researchers concrete chains to study the IT-OT boundary.
- Because the datasets include multiple evidence types from the same attack, they enable cross-evidence correlation studies, such as matching memory-resident processes to network flows.
- The public release lets other teams reproduce the authors' case-study findings and extend them, making follow-on research comparable.
Reading between the lines
- A natural next step, not pursued here, would be to enrich the releases with the missing evidence types—host-level logs for AS1 and disk images for AS2—so that log-based and file-based detection can be evaluated on the same incidents.
- The same CTI-to-scenario-to-evidence pipeline could be transferred to other critical-infrastructure domains (water, power, manufacturing), producing comparable public corpora; the paper focuses on railways but nothing in the method is railway-specific.
- If the released datasets gain wide use, the indicator counts reported in Table 3 could serve as a coarse difficulty gauge for detection tools, though a formal difficulty metric would need to be defined.
- Whether the emulator's operational logic matches real railways closely enough to make the indicators representative remains an open empirical question; a direct validation against real railway OT traffic would settle it.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents two ICS attack datasets for a railway cyber range. The environment is a miniature railway IT/OT system with corporate, SCADA, production, and physical-world networks, and two attack scenarios are designed from CTI reports (Dragonfly/Allanite for AS1, APT44/Sandworm BlackEnergy for AS2) and executed on this cyber range. Evidence is collected as disk images, memory images, network captures, and system logs, and the datasets are publicly released via Zenodo. The central claim is that these datasets support future research in railway ICS security and DFIR. The main evaluation is a case study of AS2 showing that indicators from the attack (base64-encoded web shell upload, ZoomMeetingInstaller.exe process in memory, C2 exfiltration traffic) can be recovered from the collected evidence, plus indicator counts per evidence type in Table 3.
Significance. If the datasets are released and usable, the contribution is a public railway-specific ICS forensics corpus containing memory, disk, network, and log evidence from two end-to-end attacks with physical consequences (train crash, power outage). Existing public ICS datasets are mostly water/energy/general SCADA, and railway-specific ones are rare, so this is a useful gap-filling resource. Strengths include: public release with a DOI, multi-type evidence in standard forensically sound formats (E01, pcap), scenario design grounded in CTI reports, and a Cyber Range design described in enough detail to be suggestive for replication. The paper's approach of using a real cyber exercise as the testbed is a reasonable way to obtain realistic multi-day attack traces. However, the usefulness claim is conditional on fidelity to real railway OT and on the absence of experimental artifacts, neither of which is demonstrated. The case study only shows internal consistency of the artifacts with the planted attack steps, and the indicator counting methodology is unspecified, so the evidential value of the evaluation is limited.
major comments (4)
- [Section 4.3 and Table 3] The evaluation is self-confirming: the AS2 case study recovers IoCs that the authors themselves planted (base64 web shell in Listing 1.1, ZoomMeetingInstaller.exe in Listing 1.2, C2 URL in Figure 7). This demonstrates internal consistency between scenario and evidence, but does not demonstrate external validity or usefulness for detection research. The paper cites [12,18] to argue that realistic design matters to avoid experimental artifacts, yet no artifact analysis is performed on the released data (e.g., checking whether CUE scheduling periodicities introduce machine-detectable patterns, or whether the absence of real railway interlocking/ATP traffic creates unrealistic coverage). I recommend either adding an artifact analysis or a baseline detection experiment, or explicitly reframing the contribution as a consistency-checked artifact release whose realism must be validated by downstream users.
- [Table 3 and Section 4.2] The 'Indicator Count' rows in Table 3 are reported without any definition of what constitutes an indicator, how the count was obtained, or which indicators are included. The counts (e.g., 14 network indicators for AS1, 8 for AS2) are not tied to the case study nor to an indicator list provided with the repository. Without a counting methodology or a ground-truth annotation file, these numbers are not verifiable or useful to downstream users, and may overstate the evidence content. Please provide the full indicator list, the enumeration method, and ideally a mapping from each indicator to the attack step in Table 1 or Table 2.
- [Section 2.2 and Section 3.2] The fidelity of the cyber range to real railway IT/OT is asserted, not demonstrated. Section 2.2 states the platform 'reflects the fundamental operational logic' and Section 3.2 states the testbed 'mirrors the physical railway infrastructure,' but no comparison to real railway deployments, no fidelity metrics, and no validation of the train/PLC/RTU emulation against actual railway interlocking or ATP behavior is provided. Since the datasets' research value depends on being a valid approximation of real railway attack evidence, the manuscript should either provide such validation or clearly state this is an emulation whose realism is not established and which users must assess for their own purposes.
- [Section 4.2] For AS1, the paper states that a disk image, two memory images, and one network dump were collected, and Table 3 marks system logs as unavailable (✗). However, Section 4.2 says 'we capture the network traffic from all interfaces firewall and router' without stating whether firewall or router logs were captured. Clarify what 'system-level logs' means in this context, why none are available despite firewalls/routers being present, and whether this is a limitation for DFIR workflows that depend on log correlation.
minor comments (6)
- [Abstract and Section 1] The paper does not state the dataset DOI in the abstract, only in a footnote in Section 1 (https://doi.org/10.5281/zenodo.15536351). For a dataset paper, the availability statement with DOI and a brief description of contents and formats should appear prominently, ideally in the abstract.
- [Section 2.2 and Figure 1] Figure 1 is dense and contains labels that appear cut off or ambiguous (e.g., 'Local HW-Engineering' and 'Field device admin' labels are incomplete in context). Consider redrawing at higher resolution or listing components in a table so that all component names are legible.
- [Table 1] Table 1 step 9 mentions triggering an emergency power cutoff to the 'weline01' train, but this identifier is unexplained. Additionally, step 6 says the spyTrojan 'securely copies the FCI module to the PLC machine using SCP,' while Figure 5 shows PLCs as VMs in the Engineer Department; clarify whether the FCI module runs on the PLC VM or on a separate host.
- [Section 3.1] The phrase 'non-conflicting information' used to describe CTI report reconciliation is ambiguous. Clarify what constitutes a conflict (e.g., different IPs for the same C2 server, different malware names) and how conflicts were resolved, since reproducibility of the scenario design depends on this step.
- [Section 4.3 and Figure 7] Figure 7 is used as evidence of C2 communication but the figure is not readable in the current draft; the decoded string is described in the text but not shown clearly. Please enlarge the figure or include a text transcript of the decoded URL and command.
- [Section 5.2] The related work section lists existing datasets (SWaT, WADI, HAI, EPIC, ICS-ADD) but does not compare them in terms of evidence types, attack coverage, or public availability. A short comparison table would better position the proposed railway datasets and make the novelty claim easier to assess.
Circularity Check
No significant circularity: the dataset construction is self-contained; the case study's recovered indicators are planted artifacts, a consistency check rather than a prediction.
full rationale
The paper's central contribution is the construction and public release of two railway ICS attack datasets (Sec. 1). It contains no equations, no fitted parameters, and no derivation whose conclusion is equivalent to an input. The attack scenarios are designed first (Sec. 3.1) from CTI-derived historical patterns, then executed on the cyber range (Sec. 3.2), and the evidence is collected as datasets (Sec. 4.2). The case study in Sec. 4.3 shows that indicators planted in the attack design, e.g. the ZoomMeetingInstaller.exe process in Listing 1.2 and the base64-encoded web shell in Listing 1.1, are recoverable from memory and log evidence. This is an artifact-consistency check against the ground-truth attack script, not a 'prediction' derived from fitted data; the paper does not claim the recovered IoCs validate the realism of the range. The only self-citation is use of the authors' Cluster User Emulation System [19] for traffic generation; CUE is an open-source tool whose use is descriptive, and no load-bearing argument reduces to that citation. Concerns that the miniature range may not faithfully approximate real railway OT are external-validity limitations, not circular reasoning. Therefore no circularity is found.
Assumptions & free parameters
assumptions (4)
- domain assumption The railway cyber range emulates real railway systems closely enough to produce representative attack data.
- domain assumption CTI reports describing historical ICS attacks are accurate, non-conflicting, and sufficient to define realistic attack scenarios.
- domain assumption The 2015 Ukraine power grid incident is accurately represented by the paper's characterization as a BlackEnergy-based DDoS causing power disruption.
- domain assumption CUE-generated benign user activity is realistic enough to avoid experimental artifacts in the captured data.
Cite this review
Pith. "Pith review of Signals and Symptoms: ICS Attack Dataset From Railway Cyber Range." pith.science (2026). https://pith.science/paper/C7MADWPC
@misc{pith2026250701768,
author = {Pith},
title = {Pith review of: Signals and Symptoms: ICS Attack Dataset From Railway Cyber Range},
year = {2026},
howpublished = {\url{https://pith.science/paper/C7MADWPC}},
note = {Machine review of arXiv:2507.01768}
}
read the original abstract
The prevalence of cyberattacks on Industrial Control Systems (ICS) has highlighted the necessity for robust security measures and incident response to protect critical infrastructure. This is prominent when Operational Technology (OT) systems undergo digital transformation by integrating with Information Technology (IT) systems to enhance operational efficiency, adaptability, and safety. To support analysts in staying abreast of emerging attack patterns, there is a need for ICS datasets that reflect indicators representative of contemporary cyber threats. To address this, we conduct two ICS cyberattack simulations to showcase the impact of trending ICS cyberattacks on a railway cyber range that resembles the railway infrastructure. The attack scenario is designed to blend trending attack trends with attack patterns observed from historical ICS incidents. The resulting evidence is collected as datasets, serving as an essential resource for cyberattack analysis. This captures key indicators that are relevant to the current threat landscape, augmenting the effectiveness of security systems and analysts to protect against ICS cyber threats.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
In: Abie, H., Ranise, 18 Yusof et al
Ahmed, C.M., Kandasamy, N.K.: A comprehensive dataset from a smart grid testbed for machine learning based cps security research. In: Abie, H., Ranise, 18 Yusof et al. S., Verderame, L., Cambiaso, E., Ugarelli, R., Giunta, G., Praça, I., Battisti, F. (eds.) Cyber-Physical Security for Critical Infrastructures Protection. pp. 123–135. Springer Internationa...
work page 2021
-
[2]
Ahmed, C.M., Palleti, V.R., Mathur, A.P.: Wadi: a water distribution testbed for research in the design of secure cyber physical systems. In: Proceedings of the 3rd International Workshop on Cyber-Physical Systems for Smart Wa- ter Networks. p. 25–28. CySWATER ’17, Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3055...
arXiv 2017
-
[3]
Akbarzadeh, A., Erdodi, L., Houmb, S.H., Soltvedt, T.G.: Two-stage advanced persistent threat (apt) attack on an iec 61850 power grid substation. Int. J. Inf. Secur. 23(4), 2739–2758 (May 2024). https://doi.org/10.1007/s10207-024-00856-6, https://doi.org/10.1007/s10207-024-00856-6
-
[4]
In: Proceedings of the 18th International Conference on Availability, Reliability and Security
Allison, D., Smith, P., Mclaughlin, K.: Digital twin-enhanced incident response for cyber-physical systems. In: Proceedings of the 18th International Conference on Availability, Reliability and Security. ARES ’23, Association for Computing Machinery, New York, NY, USA (2023). https://doi.org/10.1145/3600160.3600195
arXiv 2023
-
[5]
IEEE Communications Surveys & Tutorials 23(4), 2248–2294 (2021)
Conti, M., Donadel, D., Turrin, F.: A survey on industrial control system testbeds and datasets for security research. IEEE Communications Surveys & Tutorials 23(4), 2248–2294 (2021). https://doi.org/10.1109/COMST.2021.3094360
-
[6]
Computers in Industry 148, 103906 (2023)
Dehlaghi-Ghadim, A., Balador, A., Moghadam, M.H., Hansson, H., Conti, M.: Icssim — a framework for building industrial control sys- tems security testbeds. Computers in Industry 148, 103906 (2023). https://doi.org/https://doi.org/10.1016/j.compind.2023.103906, https: //www.sciencedirect.com/science/article/pii/S0166361523000568
-
[7]
IEEE Access 11, 107982–107996 (2023)
Dehlaghi-Ghadim, A., Moghadam, M.H., Balador, A., Hansson, H.: Anomaly de- tection dataset for industrial control systems. IEEE Access 11, 107982–107996 (2023). https://doi.org/10.1109/ACCESS.2023.3320928
-
[8]
Computers & Chemical Engineering 17(3), 245–255 (1993)
Downs, J., Vogel, E.: A plant-wide industrial process control prob- lem. Computers & Chemical Engineering 17(3), 245–255 (1993). https://doi.org/https://doi.org/10.1016/0098-1354(93)80018-I, https: //www.sciencedirect.com/science/article/pii/009813549380018I, indus- trial challenge problems in process control
Show all 25 references
-
[9]
d’Ambrosio, N., Capodagli, G., Perrone, G., Romano, S.P.: Scass: Break- ing into scada systems security. Comput. Secur. 151(C) (Apr 2025). https://doi.org/10.1016/j.cose.2025.104315
2025
-
[10]
Green Energy and Intelligent Transportation p
Fernandes, T., Magalhães, J.P., Alves, W.: Cybersecurity in smart rail- ways: exploring risks, vulnerabilities and mitigation in the data com- munication services. Green Energy and Intelligent Transportation p. 100305 (2025). https://doi.org/https://doi.org/10.1016/j.geits.202...
2025
-
[11]
In- ternational Journal of Critical Infrastructure Protection 36, 100487 (2022)
Firoozjaei, M.D., Mahmoudyar, N., Baseri, Y., Ghorbani, A.A.: An evaluation framework for industrial control system cyber incidents. In- ternational Journal of Critical Infrastructure Protection 36, 100487 (2022). https://doi.org/https://doi.org/10.1016/j.ijcip.2021.100487, ht...
2022
-
[12]
In: 2024 IEEE 9th European Sym- posium on Security and Privacy (EuroS&P)
Flood, R., Engelen, G., Aspinall, D., Desmet, L.: Bad design smells in benchmark nids datasets. In: 2024 IEEE 9th European Sym- posium on Security and Privacy (EuroS&P). pp. 658–675 (2024). https://doi.org/10.1109/EuroSP60621.2024.00042 Signals and Symptoms: ICS Attack Dataset...
2024
-
[13]
Fraunhofer: Safety4rails eu project.https://safety4rails.eu/ (2020), [Accessed 25-05-2025]
2020
-
[14]
IEEE Access 12, 64140–64149 (2024)
Gaggero, G.B., Armellin, A., Portomauro, G., Marchese, M.: Industrial con- trol system-anomaly detection dataset (ics-add) for cyber-physical security mon- itoring in smart industry environments. IEEE Access 12, 64140–64149 (2024). https://doi.org/10.1109/ACCESS.2024.3395991
2024
-
[15]
Applied Sciences 13(6) (2023)
Genge, B., Haller, P., Roman, A.S.: E-aptdetect: Early advanced persistent threat detection in critical infrastructures with dynamic attestation. Applied Sciences 13(6) (2023). https://doi.org/10.3390/app13063409,https://www.mdpi. com/2076-3417/13/6/3409
2023 doi
-
[16]
In: Havarneanu, G., Setola, R., Nas- sopoulos, H., Wolthusen, S
Goh, J., Adepu, S., Junejo, K.N., Mathur, A.: A dataset to support research in the design of secure water treatment systems. In: Havarneanu, G., Setola, R., Nas- sopoulos, H., Wolthusen, S. (eds.) Critical Information Infrastructures Security. pp. 88–99. Springer International...
2017
-
[17]
Electronics 13(10) (2024)
Kim, M., Jeon, S., Cho, J., Gong, S.: Data-driven ics network sim- ulation for synthetic data generation. Electronics 13(10) (2024). https://doi.org/10.3390/electronics13101920, https://www.mdpi.com/ 2079-9292/13/10/1920
2024 doi
-
[18]
In: 2025 IEEE Symposium on Security and Privacy (SP)
Liu, J., Inam, M.A., Goyal, A., Riddle, A., Westfall, K., Bates, A.: What We Talk About When We Talk About Logs: Understanding the Effects of Dataset Quality on Endpoint Threat Detection Research . In: 2025 IEEE Symposium on Security and Privacy (SP). pp. 112–129. IEEE Compute...
2025
-
[19]
LiuYuancheng: Cluster User Emulation System, https://github.com/ LiuYuancheng/Cluster_User_Emulation_System
-
[20]
In: 2024 29th Inter- national Conference on Automation and Computing (ICAC)
Lo, C., Win, T.Y., Rezaeifar, Z., Khan, Z., Legg, P.: Digital twins of cy- ber physical systems in smart manufacturing for threat simulation and de- tection with deep learning for time series classification. In: 2024 29th Inter- national Conference on Automation and Computing ...
2024
-
[21]
In: Pro- ceedings of the 7th Symposium on Hot Topics in the Science of Security
Neema, H., Koutsoukos, X., Potteiger, B., Tang, C., Stouffer, K.: Simulation testbed for railway infrastructure security and resilience evaluation. In: Pro- ceedings of the 7th Symposium on Hot Topics in the Science of Security. Hot- SoS ’20, Association for Computing Machiner...
2020
-
[22]
In: 13th USENIX Workshop on Cyber Security Experimentation and Test (CSET 20)
Shin, H.K., Lee, W., Yun, J.H., Kim, H.: HAI 1.0: HIL-based augmented ICS security dataset. In: 13th USENIX Workshop on Cyber Security Experimentation and Test (CSET 20). USENIX Association (Aug 2020),https://www.usenix.org/ conference/cset20/presentation/shin
2020
-
[23]
In: Computer Security
Srivastava, K., Köpke, C., Walter, J., Faist, K., Berry, J.M., Porretti, C., Stolz, A.: Modelling and simulation of railway networks for resilience analysis. In: Computer Security. ESORICS 2022 International Workshops: CyberICPS 2022, SECPRE 2022, SPOSE 2022, CPS4CIP 2022, CDT...
2022 doi
-
[24]
In: 2016 IEEE Region 10 Conference (TENCON)
Teo, Z.T., Tran, B.A.N., Lakshminarayana, S., Temple, W.G., Chen, B., Tan, R., Yau,D.K.Y.:Securerails:Towardsanopensimulationplatformforanalyzingcyber- physical attacks in railways. In: 2016 IEEE Region 10 Conference (TENCON). pp. 95–98 (2016). https://doi.org/10.1109/TENCON.2...
2016
-
[25]
IEEE Transactions on Dependable and Secure Computing 22(01), 16–33 (Jan 2025)
Yang, Z., He, L., Cheng, P., Chen, J.: Mismatched Control and Monitoring Frequencies: Vulnerability, Attack, and Mitigation . IEEE Transactions on Dependable and Secure Computing 22(01), 16–33 (Jan 2025). https://doi.org/10.1109/TDSC.2024.3384146, https://doi. ieeecomputersoci...
2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.