REVIEW 3 major objections 6 minor 37 references
A web platform that automates cybersecurity tabletop exercises can scale team training in universities, backed by 25 runs and 24 lessons from 743 participants.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-31 15:28 UTC pith:2SXCBNSV
load-bearing objection Solid FIE practice paper: real scale (25 runs, 743 people) and usable lessons on an open TTX platform; causal “increased engagement” language outruns the evidence. the 3 major comments →
Technology-Enhanced Tabletop Exercises for Cybersecurity Education: Lessons Learned
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Technology-enhanced tabletop exercises delivered through an automated web platform are a scalable and replicable model for university cybersecurity education: automating inject delivery, milestone-driven scenario flow, and interaction logging reduces instructor workload, raises realism, and yields actionable insight into team decision-making, as shown across 25 exercises with 743 participants and distilled into 24 lifecycle lessons.
What carries the argument
The INJECT Exercise Platform (IXP) plus the INJECT Process: a web environment that drives scenarios via timed or milestone-triggered injects, simulated tools and email, and logged actions, structured across understanding, specification, preparation, execution, and reflection phases.
Load-bearing premise
That post-exercise questionnaires and small instructor focus groups are enough to show that engagement, collaboration, and learning improved because of the digital format, and that those gains will hold outside the authors’ courses and scenarios.
What would settle it
Run the same scenarios for matched cohorts with and without the platform (or with paper TTXs), using pre/post skill measures and blinded ratings of team decisions; if engagement, collaboration quality, and learning gains do not differ, the central scalability-and-benefit claim fails.
If this is right
- Instructors can reuse scenario definitions across cohorts with little rework once milestone logic and content are stable.
- Real-time milestone dashboards let facilitators spot stuck teams during a run instead of waiting for paper debriefs.
- On-demand fully automated exercises can reach large enrollments, at the cost of less flexible free-form assessment and live facilitation.
- Structured reflection that ties logged decisions to next-step commitments becomes the main lever for lasting behavior change.
- The same digital TTX pattern can transfer to other team-based problem-solving courses beyond cybersecurity.
Where Pith is reading between the lines
- The bottleneck will shift from delivery logistics to scenario design skill—especially milestone logic—so faculty development may matter more than more platform features.
- If AI-assisted scoring of free-text replies matures, instructor-in-the-loop evaluation could scale without forcing every exercise into multiple-choice form.
- Institutions that treat scenario libraries as shared curriculum assets, not one-off events, will capture most of the claimed reuse benefit.
- Comparative studies against other active-learning formats (not only paper TTXs) would clarify whether the gains are format-specific or mainly from structured teamwork time.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This innovative-practice paper reports on integrating technology-enhanced cybersecurity tabletop exercises (TTXs) via the open-source INJECT Exercise Platform (IXP). The authors describe the INJECT Process (understanding–specification–preparation–execution–reflection), platform capabilities (milestone-driven injects, simulated tools/email, dashboards, YAML/editor authoring, on-demand and multi-tenant runs), and 25 deliveries (2024–2026) totaling 743 participants across courses and extracurricular events (Table I). From post-exercise trainee questionnaires and instructor focus groups (8 instructors), they distill 24 lessons spanning the lifecycle and argue that digital TTXs are a scalable, replicable model that increases engagement/collaboration, reduces instructor workload, and improves visibility into team decision-making.
Significance. For computing/cybersecurity education practice, the contribution is substantial and usable: a documented multi-year deployment at non-trivial scale, an open platform with exercise definitions and tooling, and concrete facilitation/design lessons organized by lifecycle phase. Strengths that should be credited include open-source IXP and exercise library, explicit milestone/tool design guidance, real-time instructor views, and exportable logs that enable research reuse. Even if causal effectiveness claims are tempered, the operational feasibility evidence and practitioner guidance are valuable for FIE-style innovative practice and for curriculum designers adopting digital TTXs.
major comments (3)
- [Abstract, §I, §V opening, §VI] Abstract, §I, and §VI claim the practice “increased engagement and collaboration,” yielded “actionable insight into student learning,” and that the authors “demonstrate” a “scalable and replicable model.” Section V states the evidence base is post-exercise trainee questionnaires plus focus groups with 8 instructors; Table I shows concentration in the authors’ own repeated course scenarios plus selected extracurriculars. There is no baseline arm (pen-and-paper/SharePoint as in prior work [13]), no validated engagement/collaboration instruments, and no pre/post learning measures. Platform logs support process-visibility claims but are not used comparatively for “increased” outcomes. The load-bearing causal and generalizability language should be revised to match the design: demonstrated operational feasibility and instructor-perceived benefits under author facilitation, with transfer treat
- [Abstract, Table I, §VI] The replicability claim (Abstract/§VI) rests on volume plus lessons, but external independent delivery sites, non-author facilitators running full scenarios without the design team, and non-cyber domains are not reported. Table I’s repeated scenario families and single primary institution make “scalable and replicable model for … others requiring team-based problem-solving” stronger than the evidence. Either report external reuse data if available, or narrow the claim to “a scalable delivery model in our setting, with lessons intended to transfer,” and state boundary conditions (facilitator skill, scenario quality, institutional context) explicitly in §VI.
- [§V.E, §VI] §V.E and the conclusions correctly emphasize that reflection is where learning happens and that on-demand automation trades away structured debrief. Given that stance, the paper’s own outcome claims still lean on immediate post-exercise self-report rather than structured reflection products (action commitments, decision comparisons tied to milestones, delayed follow-up). Strengthening the manuscript does not require a new RCT, but it does require aligning claims with what was measured—or briefly reporting any debrief artifacts/scores actually used—so the “learning” language is not carried only by engagement impressions.
minor comments (6)
- [Table I] Table I header glyphs (discussion vs simulation) are hard to parse in plain text/print; add an explicit column legend and spell out exercise type in the table body.
- [Title block / References] DOI is still “TODO”; fix before camera-ready. Several URLs are dated 2026 access—ensure consistency with the proceedings timeline.
- [§V] The paper says “24 lessons” but the enumerated lettered items under §V are easy to miscount; add a compact numbered inventory (or appendix checklist) mapping each lesson to a phase for reuse by instructors.
- [§V.C–§V.E] Figures 2–6 are helpful but depend on screenshots; ensure captions stand alone (what milestone clustering distance means for a practitioner; what an instructor should do when a team is an outlier in Fig. 5).
- [§II.B] Related work is current; still, briefly contrast IXP’s milestone///tool model with Watkins et al.’s AI-inject proposal on what is actually deployed vs. proposed, to sharpen novelty for readers.
- [Throughout] Minor copyediting: spacing anomalies (“first -time”, “human -readable”, “Y AML”), and consistent expansion of TTX/IXP on first use in each major section.
Circularity Check
No derivation-chain circularity: experiential lessons-learned paper with no fitted-as-prediction or definitional loops
full rationale
This is an innovative-practice / lessons-learned paper, not a first-principles or predictive derivation. The load-bearing content is 24 qualitative lessons distilled from 25 IXP-delivered runs (743 participants, 2024–2026), structured by the authors’ INJECT Process phases and supported by post-exercise questionnaires and instructor focus groups. There are no equations, fitted parameters renamed as predictions, uniqueness theorems, or ansatzes smuggled via self-citation. Self-citations to the authors’ prior IXP/TTX work ([10], [13], platform docs) are normal platform-evolution context and do not force the new observational claims by construction. Claims of ‘increased engagement/collaboration’ and a ‘scalable and replicable model’ are empirical/self-reported assertions whose evidential weakness is a validity issue, not circular reduction of outputs to inputs. No step reduces Eq. X to Eq. Y or a fit to a ‘prediction.’ Score 0; steps empty.
Axiom & Free-Parameter Ledger
axioms (4)
- domain assumption Simulation-based / experiential learning (including structured debrief) improves collaborative incident-response skills relative to purely didactic instruction.
- domain assumption Post-exercise trainee questionnaires and instructor focus groups are adequate instruments to detect engagement, collaboration, and exercise-design quality.
- domain assumption Milestone/trigger logic in a digital platform can faithfully encode intended pedagogical scenario flow for TTX learning objectives.
- ad hoc to paper Lessons from mostly one university’s cybersecurity and related cohorts transfer to other institutions and non-cyber team problem-solving courses.
invented entities (1)
-
INJECT Process (five phases: understanding, specification, preparation, execution, reflection)
no independent evidence
read the original abstract
This innovative practice full paper examines the integration of technology-enhanced tabletop exercises (TTXs) into computing education, focusing on cybersecurity curricula. The motivation is to better prepare students for complex, collaborative problem solving typical of incident response and IT governance, where coordination, communication, and timely decision-making are essential. Although TTXs are well-established in professional practice, they remain underused in universities. We address this gap by augmenting TTX delivery and evaluation through the INJECT Exercise Platform (IXP), a web-based environment that automates scenario flow and enables data-driven assessment. Our practice implements IXP to automatically deliver scenario updates, facilitate team discussions, and collect interaction data to support automated assessment. This combination enhances realism, reduces instructor workload, and provides actionable insight into student learning. From 2024 to 2026, we ran 25 exercises with 743 participants in multiple university courses and extracurricular events. We observed increased engagement and collaboration among students, and clearer visibility for instructors into how teams navigate complex scenarios. This paper shares 24 lessons learned from these exercises. Instructors and curriculum designers may benefit from concrete guidance for integrating technology-enhanced TTXs. We demonstrate that digital TTXs provide a scalable and replicable model for cybersecurity courses and others requiring team-based problem-solving.
Figures
Reference graph
Works this paper leans on
-
[1]
Lelewski and J
R. Lelewski and J. Hollenberger,Cybersecurity Tabletop Exercises: From Planning to Execution. San Francisco, USA: No Starch Press, 2025, ISBN: 978-1718503823
2025
-
[2]
Guide to test, training, and exercise programs for it plans and capabilities,
T. Grance, T. Nolan, K. Burke, R. Dudley, G. White, and T. Good, “Guide to test, training, and exercise programs for it plans and capabilities,” NIST, Tech. Rep., 09 2006
2006
-
[3]
The ENISA Cybersecurity Exercise Methodology,
A. Zacharis, A. Sarri, C. Van Heurck, F. Fanourakis, G. Fernández, N. Christoforatos, and R. Arcus, “The ENISA Cybersecurity Exercise Methodology,” European Union Agency for Cybersecurity (ENISA), Technical Report, 2 2026. [Online]. Available: https://www.enisa.europa.eu/sites/default/files/2026-02/The% 20ENISA%20Cybersecurity%20Exercise%20Methodology.pdf
2026
-
[4]
Enhancing civil-military cyber resilience lessons from the ecybridge tabletop exercise
M. Preda, V . Popescu, C. Argint, N. Iancu, G. Raicu, and G. Ene, “Enhancing civil-military cyber resilience lessons from the ecybridge tabletop exercise.”International Journal of Information Security & Cybercrime, vol. 14, no. 1, 2025. [Online]. Available: https://www.ceeol.com/search/article-detail?id=1349715
2025
-
[5]
Simulating cyber-resilience: The strategic role of the locked shields exercise in enhancing international cyber preparedness,
C. A. Ramezan, L. C. Schaupp, E. A. Vitullo, and W. J. Walker, “Simulating cyber-resilience: The strategic role of the locked shields exercise in enhancing international cyber preparedness,”Journal of Cybersecurity Education, Research and Practice, vol. 2026, no. 1, p. 5,
2026
-
[6]
The Value of Wargames and Tabletop Exercises as Naturalistic Tools,
S. L. Dorton, T. Fersch, E. Barrett, A. Langone, M. Seip, S. Bilsborough, C. B. Hudson Jr, P. Ward, and K. J. Neville, “The Value of Wargames and Tabletop Exercises as Naturalistic Tools,” inProceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 67, no. 1. SAGE Publications Sage CA: Los Angeles, CA, 2023, pp. 2454–2459. [Online]. Ava...
-
[7]
Challenges in IT security preparedness exercises: A case study,
M. Bartnes and N. B. Moe, “Challenges in IT security preparedness exercises: A case study,”Computers & Security, vol. 67, pp. 280–290,
-
[8]
Cyber-Security at OSI Layer 1: Defence-in-Depth for Energy Grids,
P. J. Lenk, S. Kines, E. Taube, M. Münzer, T. Kuusk, and S. Alberico, “Cyber-Security at OSI Layer 1: Defence-in-Depth for Energy Grids,” 2025, MP-SAS-190 Technical Evaluation Report. [Online]. Available: https://publications.sto.nato.int/publications/STO% 20Meeting%20Proceedings/STO-MP-SAS-190/MP-SAS-190-07.pdf
2025
-
[9]
Directive (EU) 2022/2555 of the European Parliament and of the Council of 14 December 2022 on measures for a high common level of cybersecurity across the Union,
European Parliament and Council of the European Union, “Directive (EU) 2022/2555 of the European Parliament and of the Council of 14 December 2022 on measures for a high common level of cybersecurity across the Union,” Official Journal of the European Union, L 333/80,
2022
-
[11]
Light Weight Tabletop Exercise for Cybersecurity Education,
R. Ottis, “Light Weight Tabletop Exercise for Cybersecurity Education,” Journal of Homeland Security and Emergency Management, vol. 11, pp. 579–592, 12 2014. [Online]. Available: https://doi.org/10.1515/ jhsem-2014-0031
2014
-
[12]
G. Angafor, I. Yevseyeva, and L. Maglaras, “MalAware: A tabletop exercise for malware security awareness education and incident response training,”Internet of Things and Cyber-Physical Systems, vol. 4, pp. 280–292, 2024. [Online]. Available: https: //doi.org/10.1016/j.iotcps.2024.02.003
-
[13]
From Paper to Platform: Evolution of a Novel Learning Environment for Tabletop Exercises,
V . Švábenský, J. Vykopal, M. Horák, M. Hofbauer, and P. ˇCeleda, “From Paper to Platform: Evolution of a Novel Learning Environment for Tabletop Exercises,” inInnovation and Technology in Computer Science Education. New York, NY , USA: ACM, 2024, pp. 213–219. [Online]. Available: https://doi.org/10.1145/3649217.3653639
arXiv 2024
-
[14]
Simulation-Based Learning in Higher Education: A Meta- Analysis,
O. Chernikova, N. Heitzmann, M. Stadler, D. Holzberger, T. Seidel, and F. Fischer, “Simulation-Based Learning in Higher Education: A Meta- Analysis,”Review of educational research, vol. 90, no. 4, pp. 499–541,
-
[15]
P. Hallinger and R. Wang, “The Evolution of Simulation-Based Learning Across the Disciplines, 1965–2018: A Science Map of the Literature,” Simulation & Gaming, vol. 51, no. 1, pp. 9–32, 2020. [Online]. Available: https://doi.org/10.1177/1046878119888246
-
[16]
A Framework for Developing Tabletop Cybersecurity Exercises,
N. Chowdhury and V . Gkioulos, “A Framework for Developing Tabletop Cybersecurity Exercises,” inComputer Security. ESORICS 2022 International Workshops. Cham: Springer International Publishing, 2023, pp. 116–133. [Online]. Available: https://doi.org/10.1007/ 978-3-031-25460-4_7
2022
-
[17]
Tabletop Exercise for Ransomware Negotiations,
L. Müller, “Tabletop Exercise for Ransomware Negotiations,” in Augmented Cognition, D. D. Schmorrow and C. M. Fidopiastis, Eds. Cham: Springer Nature Switzerland, 2024, pp. 166–184. [Online]. Available: https://doi.org/10.1007/978-3-031-61572-6_12
-
[18]
Tabletop exercise for ransomware negotiations,
——, “Tabletop exercise for ransomware negotiations,” Ger- many, February 2024, Bachelor’s thesis. [Online]. Avail- able: https://www.researchgate.net/publication/381290646_Tabletop_ Exercise_for_Ransomware_Negotiations
arXiv 2024
-
[19]
Using Tabletop Exercises to Raise Cybersecurity Awareness of Decision-Makers,
J. Kävrestad, S. Johansson, and E. Bergström, “Using Tabletop Exercises to Raise Cybersecurity Awareness of Decision-Makers,” in Critical Information Infrastructures Security, G. Oliva, S. Panzieri, B. Hämmerli, F. Pascucci, and L. Faramondi, Eds. Cham: Springer Nature Switzerland, 2025, pp. 231–248. [Online]. Available: https://doi.org/10.1007/978-3-031-...
-
[20]
Collaborate, design, and generate cybercrime script tabletop exercises for cybersecurity education,
J. Dwight, “Collaborate, design, and generate cybercrime script tabletop exercises for cybersecurity education,” inInternational Conference on Computers in Education, 2023. [Online]. Available: https://library.apsce.net/index.php/ICCE/article/view/1406/1300
2023
-
[21]
AI-Driven Immersive Emulation for Tabletop Scenarios,
T. Watkins, B. Davis, R. B. Ponnuru, and M. Azab, “AI-Driven Immersive Emulation for Tabletop Scenarios,” in2026 IEEE 16th Annual Computing and Communication Workshop and Conference, 2026, pp. 234–240
2026
-
[22]
Digitalization of Table- Top Exercises: An Emergency Response Training Showcase,
A. Sumereder, B. Bürger, and R. Woitsch, “Digitalization of Table- Top Exercises: An Emergency Response Training Showcase,” in Information Technology in Disaster Risk Reduction, W. Seböck, T. J. Lampoltshammer, J. Dugdale, and I. Zeller, Eds. Cham: Springer Nature Switzerland, 2026, pp. 49–65. [Online]. Available: https://doi.org/10.1007/978-3-031-97115-0_4
-
[23]
INJECT Exercise Platform,
INJECT Team, “INJECT Exercise Platform,” https://inject.muni.cz, 2026, Open-source platform for tabletop exercises, accessed: March 20, 2026
2026
-
[24]
Platform Changelog,
——, “Platform Changelog,” https://docs.inject.muni.cz/changelog/, 2026, accessed: March 11, 2026
2026
-
[25]
IXP-Definition – MS Visual Studio Code Extension,
——, “IXP-Definition – MS Visual Studio Code Extension,” https://marketplace.visualstudio.com/items?itemName=inject-muni. ixp-definition, 2026, accessed: March 11, 2026
2026
-
[26]
INJECT Process,
——, “INJECT Process,” https://docs.inject.muni.cz/INJECT_process/ intro/overview/, 2026, Documentation of designing tabletop security exercises with the INJECT Exercise Platform, accessed: March 20, 2026
2026
-
[27]
NIST Special Publication 800-84: Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities,
K. Scarfone, T. Grance, and R. Sexton, “NIST Special Publication 800-84: Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities,” https://csrc.nist.gov/pubs/sp/800/84/final, Sep 2006, accessed: March 11, 2026
2006
-
[28]
Lessons learned from complex hands-on defence exercises in a cyber range,
J. Vykopal, M. Vizvary, R. Oslejsek, P. Celeda, and D. Tovarnak, “Lessons learned from complex hands-on defence exercises in a cyber range,” in2017 IEEE Frontiers in Education Conference (FIE), 2017, pp. 1–8. [Online]. Available: https://doi.org/10.1109/FIE.2017.8190713
arXiv 2017
-
[29]
Homeland Security Ex- ercise and Evaluation Program (HSEEP),
Federal Emergency Management Agency, “Homeland Security Ex- ercise and Evaluation Program (HSEEP),” https://preptoolkit.fema. gov/documents/1269813/1269861/HSEEP_Revision_Jan20_Final.pdf, jan 2020, accessed: March 11, 2026
arXiv 2020
-
[30]
A framework for competence development and assessment in hybrid cybersecurity exercises,
A. Brilingait ˙e, L. Bukauskas, and A. Juozapavi ˇcius, “A framework for competence development and assessment in hybrid cybersecurity exercises,”Computers & Security, vol. 88, p. 101607, 2020. [Online]. Available: https://doi.org/10.1016/j.cose.2019.101607
arXiv 2020
-
[31]
Available Exercise Definitions,
INJECT Team, “Available Exercise Definitions,” https://docs.inject.muni. cz/INJECT_process/available-definitions/, 2026, accessed: March 11, 2026
2026
-
[32]
H. H. Hu, C. Kussmaul, B. Knaeble, C. Mayfield, and A. Yadav, “Results from a Survey of Faculty Adoption of Process Oriented Guided Inquiry Learning (POGIL) in Computer Science,” inProceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education, ser. ITiCSE ’16. New York, NY , USA: Association for Computing Machinery, 201...
arXiv 2016
-
[33]
Experiential learning in engineering education: A systematic literature review,
G. Tembrevilla, A. Phillion, and M. Zeadin, “Experiential learning in engineering education: A systematic literature review,”Journal of Engineering Education, vol. 113, no. 1, pp. 195–218, 2024. [Online]. Available: https://doi.org/10.1002/jee.20575
-
[34]
D. T. Rover, H. J. Duwe, M. Mina, N. D. Fila, P. H. Jones, and L. S. Sleeth, “Learning and Professional Development Through Integrated Reflective Activities in Electrical and Computer Engineering Courses,” in2021 IEEE Frontiers in Education Conference (FIE), 2021, pp. 1–9. [Online]. Available: https://doi.org/10.1109/FIE49875.2021.9637478
arXiv 2021
-
[2017]
Available: https://doi.org/10.1016/j.cose.2016.11.017
[Online]. Available: https://doi.org/10.1016/j.cose.2016.11.017
-
[2020]
Available: https://doi.org/10.3102/0034654320933544
[Online]. Available: https://doi.org/10.3102/0034654320933544
-
[2022]
Available: http://data.europa.eu/eli/dir/2022/2555/oj
[Online]. Available: http://data.europa.eu/eli/dir/2022/2555/oj
2022
-
[2026]
Available: https://doi.org/10.62915/2472-2707.1260
[Online]. Available: https://doi.org/10.62915/2472-2707.1260
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.