Pith. sign in

REVIEW 4 major objections 6 minor 59 references

The paper claims that clustering activity logs yields instructor-aligned team groupings for tabletop cybersecurity exercises in minutes, whereas GPT-4o's rubric-based communication scores disagree with human raters 55% of the time and GPT-5

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:04 UTC pith:EJYPYT7M

load-bearing objection Useful empirical paper on TTX assessment, but the clustering validation is circular and the abstract overstates it; the LLM negative result is the real takeaway. the 4 major comments →

arxiv 2607.19209 v1 pith:EJYPYT7M submitted 2026-07-21 cs.CY cs.AIcs.LG

Assessment in Team Problem-Solving Exercises in Computing Education

Valdemar \v{S}v\'abensk\'y , Jan Vykopal , Sukrit Leelaluk , Pavel \v{C}eleda , Fumiya Okubo , Atsushi Shimada This is my paper
classification cs.CY cs.AIcs.LG
keywords tabletop exerciseteam assessmentclusteringlarge language modelslearning analyticscybersecurity educationrubric-based gradingincident response
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that automatic team assessment in tabletop exercises (TTXs) is feasible using learning analytics, and that the two candidate methods are not equally trustworthy. It claims that clustering teams by which milestones they reached, and when, groups teams whose instructor-assigned scores are similar—better than the expected baseline across all possible partitions—and can return results in under two minutes, enabling fast cluster-level feedback. It also claims that a general-purpose LLM prompted with a standard rubric does not reliably grade teams' written communication: GPT-4o disagreed with instructors on 55% of scores (essentially chance), and GPT-5.2 on 35%, which the paper deems insufficiently aligned. The paper argues that instructors should adopt clustering for post-exercise assessment and treat LLM output with caution, and it provides the dataset and tools to let others test both methods.

Core claim

The paper's central claim is that in two authentic cybersecurity tabletop exercises (data exfiltration and phishing scenarios, 23 consenting teams across two countries), clustering team activity logs—specifically which milestones each team completed and when—produces groupings with better internal score cohesion than the average over all possible Bell partitions. RMSE within clusters was 0.45 vs 0.53 baseline in the EXF exercise and 0.79 vs 1.12 in PHI; the authors call this 'sufficiently aligns with instructor scores and outperforms the baseline.' The same analysis ran in under two minutes on a standard laptop. On the LLM side, when the same rubric used by human graders was given to GPT-4o,

What carries the argument

The load-bearing object is the milestone score vector: each team gets an m-dimensional vector with 2 for an important milestone completed, 1 for an optional milestone, and 0 for a missed one (m = 17 or 18). Clustering uses these same milestone IDs and timestamps as features through DBSCAN with Unsupervised Consensus Clustering over 50 iterations, then the paper measures cluster validity as the average pairwise RMSE of the milestone score vectors within a cluster, compared against the average RMSE over all Bell partitions of the teams. For the LLM arm, the key mechanism is the rubric prompt built from a cybersecurity competency framework's writing criteria (succinct organization, audience app

Load-bearing premise

The evaluation treats the instructor's milestone score vectors as an independent ground truth even though the same milestone IDs and timestamps feed the clustering features; if that overlap inflates within-cluster similarity, the RMSE comparison does not by itself prove educationally valid groupings.

What would settle it

A concrete test: have independent experts score team performance from a source that does not include milestone logs—e.g., live observation or oral debriefing—and rerun the clustering comparison. If the within-cluster RMSE drops to the Bell-partition baseline, the clustering's apparent validity is an artifact of shared data.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Instructors can identify clusters of teams that approached tasks similarly within minutes of a TTX, then deliver feedback tailored to each cluster instead of waiting days.
  • Because clustering runs locally and uses only activity logs, it avoids sending student data to external services, addressing privacy concerns that come with LLM-based grading.
  • Teams that are outliers—in no cluster—can be flagged as having unusual approaches, which the paper suggests may indicate either unique strengths or missed protocol.
  • General-purpose LLMs, even the newer GPT-5.2, should not be used alone for rubric-based assessment of TTX communication; the paper recommends human review or domain-specific models.
  • The same assessment pipeline can be adapted to other open-ended team exercises (e.g., Capture the Flag or cyber defense exercises) if activity logs record milestone-like checkpoints.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The near-chance GPT-4o result suggests the failure is not just model weakness but a mismatch between rubric criteria (grammar, audience, concision) and what short incident-response emails reveal; a rubric focusing on incident-handling decision content might perform differently.
  • Editorial inference: Because the clustering features and the benchmark scores both come from the same milestone logs, a convincing next experiment would use an independent performance measure—such as expert observation or oral debrief scores—to validate the clusters; the paper does not provide that.
  • Editorial inference: The two methods could complement each other: clustering could select which teams need what feedback, and a domain-adapted LLM could draft the wording of that feedback; the paper's own future-work section gestures at such an ensemble.
  • Editorial inference: If clustering is adopted broadly, the interpretations of clusters (e.g., 'missed stakeholder contact') depend on the exercise designer being able to read the logs; generalizing to less structured TTXs may require hand-labeled cluster descriptions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This research-to-practice paper compares two automated methods for assessing student teams in cybersecurity tabletop exercises (TTXs): unsupervised clustering of activity logs and LLM-based rubric scoring of team communication. Using data from 24 teams (23 after consent) across two countries, the authors evaluate each method against instructor scores. They report that clustering achieves low within-cluster RMSE on milestone score vectors and outperforms a random-partition baseline, while GPT-4o's communication scores disagree with human raters at roughly chance level and GPT-5.2 shows lower but still substantial disagreement. The methods are integrated into the open-source INJECT platform, and datasets, code, and exercise definitions are released.

Significance. The LLM negative result is a credible and useful cautionary data point for using general-purpose LLMs in domain-specific assessment: GPT-4o at 55% disagreement on a 3-point scale is a robust failure, and the paper honestly reports it. The open dataset, code, and platform integration are valuable resources for the community. However, the positive clustering claim—the abstract states the method 'was valid and reliable'—is undermined by a circular evaluation: the ground-truth milestone score vectors are computed from the same milestone IDs used as clustering features. Thus the current paper's main contribution is the negative LLM finding and the released artifacts, not the validated clustering method.

major comments (4)
  1. [IV-B1, V-A1, Table III] The clustering evaluation is circular. The ground-truth milestone score vectors (§IV-A4) are produced by a script that assigns 2/1/0 based on milestone completion; the clustering features (§IV-B1) are explicitly 'which milestones teams reached (milestone IDs) and when.' The scores are therefore a deterministic function of a subset of the clustering features. Low within-cluster RMSE (Table III) is largely guaranteed for any clustering that groups by milestone IDs, and the baseline—the average RMSE over all Bell partitions—is a weaker null than a feature-dependent null. Beating this baseline only shows that clusters capture milestone IDs better than chance; it does not demonstrate 'valid' alignment with instructor judgment. Please re-run clustering without milestone IDs (e.g., using timestamps, sequence, and tool usage only) and compare against the same score vectors, or use a permutation
  2. [IV-A4, Abstract] The manuscript calls the benchmark 'instructor-assigned scores based on standardized rubrics,' but the milestone scores are assigned by a script, not by human judgment; the instructors manually scored only the communication rubric. This conflation is misleading and should be corrected throughout (e.g., 'script-derived milestone scores' vs. 'instructor-rated communication scores').
  3. [V-A1, IV-B1] The claim that clustering 'was valid and reliable' is not supported by any reliability measure. The use of Unsupervised Consensus Clustering over 50 iterations is mentioned, but no stability index (e.g., adjusted Rand index across runs) is reported. Please report a quantitative stability metric or remove the reliability claim.
  4. [V-A1, Table III] The comparison of the clustering RMSE to the average over all Bell partitions lacks a distribution: the reader cannot assess whether the improvement over the baseline is statistically meaningful for n=13 and n=10. A permutation test or a report of the standard deviation of the baseline RMSE distribution should be added.
minor comments (6)
  1. [I.A] Typo: 'throughtabletop' should be 'through tabletop'.
  2. [Table II] The symbol for 'Partially' appears corrupted ('/adjus◎'); check the rendering of the grading symbols.
  3. [V-A2] The statement that GPT-4o's disagreement 'essentially matched chance' should be backed by an explicit chance-level baseline, such as the expected RMSE from random ratings with the observed marginal distribution of human scores.
  4. [IV-B1] Provide more detail on the feature encoding (how timestamps, sequences, and tool usage are vectorized) and on the DBSCAN hyperparameters (eps, min_samples) for reproducibility.
  5. [V-B1] The qualitative cluster interpretations are post-hoc; consider explicitly labeling them as illustrative rather than confirmatory, especially given the circularity concern in the quantitative evaluation.
  6. [References] Reference [11] (LLMs for tabletop exercises) is cited only in the TTX introduction; it should also be discussed in the related work section on LLMs for team assessment.

Circularity Check

1 steps flagged

Clustering 'validity' is measured against milestone score vectors that are deterministic functions of the same milestone IDs used as cluster features; the LLM evaluation is independent and credible.

specific steps
  1. self definitional [§IV-A4 (Instructor-Assigned Scores), §IV-B1 (Clustering), §V-A1 (RQ1 results)]
    "a simple script ... scored each team's achievement of TTX milestones ... we assigned a score of 2 for completing an important milestone, 1 for completing a secondary (optional) milestone, and 0 for missing a milestone ... Features were based on IXP logs of which milestones teams reached (milestone IDs) and when (timestamps) ... we examined team similarity within each cluster using manually assigned score vectors. We used the score vectors based on the milestones (not communication)."

    The benchmark score vector is a deterministic transform of the milestone IDs that are also cluster features. Clusters that group teams by milestone completion therefore have low within-cluster RMSE on those score vectors by construction. The baseline is the average RMSE over all Bell partitions, so 'outperforms the baseline' only shows the clusters are non-random with respect to milestone IDs, not that they align with instructor judgment or are educationally valid. The abstract's claim that clustering 'was valid and reliable' rests on this circular metric.

full rationale

The clustering evaluation reduces to construction: the ground-truth milestone vectors assigned 2/1/0 per milestone come from the same milestone IDs used to build the DBSCAN/consensus-clustering features. Low within-cluster RMSE and beating a random-partition baseline are therefore expected consequences of grouping by milestone achievement, not independent evidence of validity or alignment with instructor scores. The LLM comparison is not circular: human communication scores (with inter-rater RMSE 0.32) are compared against GPT-4o/GPT-5.2 rubric ratings on email texts, which are external inputs. This negative result stands independently. Because the paper's central positive claim about clustering validity is supported only by the self-overlapping evaluation, a score of 6 is appropriate; this is partial circularity in the main positive claim, not a complete collapse of the paper.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or theoretical entities. Its load-bearing assumptions are domain assumptions about logs-as-performance and benchmark validity; the most consequential is the circular use of milestone-derived scores to validate clustering that uses milestone data as features. Free parameters include unreported clustering hyperparameters/feature encoding and LLM prompt/decoding settings.

free parameters (4)
  • DBSCAN hyperparameters (eps, min_samples) = not reported
    Cluster granularity depends directly on density parameters; without them the clustering result is not fully reproducible and the reported clusters could change.
  • Feature encoding for clustering (milestone timestamps, tool usage, sequence) = not fully specified
    The paper lists feature families but not the exact vectorization/normalization; these choices determine cluster structure.
  • LLM prompt (wording and rubric mapping) = final prompt in linked code
    The prompt was iteratively revised until output format matched expectations; this tuning on the evaluation dataset can inflate apparent performance and is a free design choice.
  • LLM decoding settings (temperature, max tokens, API version) = not reported
    LLM outputs are stochastic; without decoding settings the reported RMSE values cannot be reproduced or given error bars.
axioms (5)
  • domain assumption IXP activity logs are a valid representation of team problem-solving behavior
    Used as sole input for clustering (§IV-A3); if logs miss key aspects, clusters do not reflect true approaches.
  • domain assumption The milestone-score vector (script-assigned 0/1/2 per milestone) is a valid ground truth for team performance
    Serves as benchmark for clustering alignment (§IV-A4); the script is derived from the same milestone data, so this axiom is also the source of circularity.
  • domain assumption EXF and PHI exercises can be analyzed jointly despite different scenarios, participant populations, and instructors
    Stated in §III-B; pooling is justified only by assumed shared learning objectives.
  • domain assumption A 3-point ordinal rubric can be treated as numeric for RMSE comparison
    RMSE assumes interval scale; used for both human-human and human-LLM agreement (§IV-B, Table III).
  • domain assumption Clusters from DBSCAN on 10-13 teams are stable enough for qualitative interpretation
    Cluster interpretation in §V-B1 relies on small-sample density clustering; consensus over 50 iterations mitigates but does not remove instability.

pith-pipeline@v1.3.0-alltime-deepseek · 15703 in / 15109 out tokens · 146623 ms · 2026-08-01T13:04:58.665435+00:00 · methodology

0 comments
read the original abstract

This full paper in the research-to-practice track presents methods for assessing student teams in tabletop exercises (TTXs). TTXs enable learner teams to prepare for workplace tasks and practice crisis responses, such as resolving cybersecurity incidents. While assessment is essential for determining how well teams achieve learning objectives, the complex, open-ended nature of TTXs often leads to delayed or incomplete feedback. TTX learning platforms can record teams' actions and communication; yet, leveraging these data to assess performance is underexplored. To address this gap, we compared two post-TTX team assessment methods -- clustering and large language models (LLMs) -- using an original dataset from 81 participants across two countries. We evaluated these methods against instructor-assigned scores based on standardized rubrics. Clustering grouped teams that approached TTX tasks similarly, enabling instructors to deliver faster, targeted feedback to teams within a cluster. This method was valid and reliable, with low computational requirements. LLMs used the standardized rubrics to assess teams' communication. While GPT-4o frequently disagreed with instructor scores, GPT-5.2 demonstrated considerably lower error. The researched methods have been integrated into INJECT, an open-source TTX learning platform, to support scalability and teaching practice. To encourage community adoption, we publicly share all datasets, software tools, and a full-fledged TTX scenario.

Figures

Figures reproduced from arXiv: 2607.19209 by Atsushi Shimada, Fumiya Okubo, Jan Vykopal, Pavel \v{C}eleda, Sukrit Leelaluk, Valdemar \v{S}v\'abensk\'y.

Figure 1
Figure 1. Figure 1: Conceptual illustration of the research methods. Two distinct groups of [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Clustering in EXF. All 13 teams are identified by 3-digit IDs and [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 16 canonical work pages

  1. [1]

    Use of tabletop exercises for healthcare education: a scoping review protocol,

    A. Frégeau, A. Cournoyer, M.-A. Maheu-Cadotte, M. Iseppon, N. Soucy, J. S.-C. Bourque, S. Cossette, V . Castonguay, and R. Fleet, “Use of tabletop exercises for healthcare education: a scoping review protocol,”BMJ open, vol. 10, no. 1, 2020. [Online]. Available: https://doi.org/10.1136/bmjopen-2019-032662

    work page doi:10.1136/bmjopen-2019-032662 2020

  2. [2]

    Tabletop exercise to prepare institutions of higher education for an outbreak of covid-19,

    A. M. Wendelboe, J. Amanda Miller, D. Drevets, L. Salinas, E. Miller, D. Jackson, A. Chouet al., “Tabletop exercise to prepare institutions of higher education for an outbreak of covid-19,”Journal of emergency management, vol. 18, no. 2, 2020. [Online]. Available: https://doi.org/10.5055/jem.2020.0463

    arXiv 2020

  3. [3]

    Does tabletop exercise enhance knowledge and attitude in preparing disaster drills?

    C. Husna, H. Kamil, M. Yahya, T. Tahlil, and D. Darmawati, “Does tabletop exercise enhance knowledge and attitude in preparing disaster drills?”Nurse Media Journal of Nursing, vol. 2, no. 10, pp. 182–190,

  4. [4]

    Training of public health personnel in handling cbrn emergencies: A table-top exercise card concept,

    B. E. Sandström, H. Eriksson, L. Norlander, M. Thorstensson, and G. Cassel, “Training of public health personnel in handling cbrn emergencies: A table-top exercise card concept,”Environment International, vol. 72, pp. 164–169, 2014. [Online]. Available: https://doi.org/10.1016/j.envint.2014.03.009

    work page doi:10.1016/j.envint.2014.03.009 2014

  5. [5]

    Simulation-based learning in higher education: A meta- analysis,

    O. Chernikova, N. Heitzmann, M. Stadler, D. Holzberger, T. Seidel, and F. Fischer, “Simulation-based learning in higher education: A meta- analysis,”Review of educational research, vol. 90, no. 4, pp. 499–541,

  6. [6]

    New York, NY , USA: ACM, 2024

    The Joint Task Force on Computer Science Curricula,Computing Curricula 2023. New York, NY , USA: ACM, 2024. [Online]. Available: https://doi.org/10.1145/3664191

    doi:10.1145/3664191 2023

  7. [7]

    Available: https://doi.org/10.3102/0034654320933544

    [Online]. Available: https://doi.org/10.3102/0034654320933544

    doi:10.3102/0034654320933544

  8. [8]

    Professional competencies in computing education: Pedagogies and assessment,

    R. Raj, M. Sabin, J. Impagliazzo, D. Bowers, M. Daniels, F. Hermans, N. Kiesler, A. N. Kumar, B. MacKellar, R. McCauley, S. W. Nabi, and M. Oudshoorn, “Professional competencies in computing education: Pedagogies and assessment,” inWorking Group Reports on Innovation and Technology in Computer Science Education. ACM, 2022, p. 133–161. [Online]. Available:...

    arXiv 2022

  9. [9]

    Cybersecurity curricular guideline,

    Joint Task Force on Cybersecurity Education, “Cybersecurity curricular guideline,” 2017. [Online]. Available: http://cybered.acm.org

    2017

  10. [10]

    Malaware: A tabletop exercise for malware security awareness education and incident response training,

    G. Angafor, I. Yevseyeva, and L. Maglaras, “Malaware: A tabletop exercise for malware security awareness education and incident response training,”Internet of Things and Cyber-Physical Systems, vol. 4, 2024. [Online]. Available: https://doi.org/10.1016/j.iotcps.2024.02.003

    doi:10.1016/j.iotcps.2024.02.003 2024

  11. [12]

    Tabletop exercise for ransomware negotiations,

    L. Müller, “Tabletop exercise for ransomware negotiations,” in Augmented Cognition, vol. 14695. Springer, 2024, pp. 166–184. [Online]. Available: https://doi.org/10.1007/978-3-031-61572-6_12

    doi:10.1007/978-3-031-61572-6_12 2024

  12. [13]

    Using LLMs for Tabletop Exercises within the Security Domain,

    S. Hays and J. White, “Using LLMs for Tabletop Exercises within the Security Domain,” 2024. [Online]. Available: https: //arxiv.org/abs/2403.01626

    Pith/arXiv arXiv 2024

  13. [14]

    Using tabletop exercises to raise cybersecurity awareness of decision-makers,

    J. Kävrestad, S. Johansson, and E. Bergström, “Using tabletop exercises to raise cybersecurity awareness of decision-makers,” inCritical Information Infrastructures Security. Springer Nature Switzerland, 2025, pp. 231–

    2025

  14. [15]

    Analyzing important factors in cybersecurity incidents using table-top exercise,

    K. Nakayama, I. Koshijima, and K. Watanabe, “Analyzing important factors in cybersecurity incidents using table-top exercise,”Human Factors in Cybersecurity, vol. 127, pp. 105–114, 2024. [Online]. Available: https://doi.org/10.54941/ahfe1004770

    work page doi:10.54941/ahfe1004770 2024

  15. [16]

    Cybersecurity,

    U.S. Environmental Protection Agency, “Cybersecurity,” 2022. [Online]. Available: https://ttx.epa.gov/CyberSecurity7.html

    2022

  16. [18]

    Cyber exercises in computer science education,

    M. Gafic, S. Tjoa, P. Kieseberg, O. Hellwig, and G. Quirchmayr, “Cyber exercises in computer science education,” inProceedings of the 8th International Conference on Information Systems Security and Privacy,

  17. [19]

    Exercise design for interagency collaboration training: The case of maritime nuclear emergency management tabletop exercises,

    R. Elvegård and N. Andreassen, “Exercise design for interagency collaboration training: The case of maritime nuclear emergency management tabletop exercises,”Journal of Contingencies and Crisis Management, vol. 32, no. 1, 2024. [Online]. Available: https://doi.org/10.1111/1468-5973.12517

    arXiv 2024

  18. [20]

    Towards giving timely formative feedback and hints to novice programmers,

    J. Jeuring, H. Keuning, S. Marwan, D. Bouvier, C. Izu, N. Kiesler, T. Lehtinen, D. Lohr, A. Peterson, and S. Sarsa, “Towards giving timely formative feedback and hints to novice programmers,” inProceedings of the ITiCSE 2022 Working Group Reports. ACM, 2022. [Online]. Available: https://doi.org/10.1145/3571785.3574124

    arXiv 2022

  19. [21]

    Using terminal histories to monitor student progress on hands-on exercises,

    J. Mirkovic, A. Aggarwal, D. Weinman, P. Lepe, J. Mache, and R. Weiss, “Using terminal histories to monitor student progress on hands-on exercises,” inACM Technical Symposium on Computer Science Education, ser. SIGCSE. New York, NY , USA: ACM, 2020, p. 866–872. [Online]. Available: https://doi.org/10.1145/3328778.3366935

    arXiv 2020

  20. [23]

    Romero, S

    C. Romero, S. Ventura, M. Pechenizkiy, and R. S. Baker,Handbook of educational data mining. USA: CRC Press, 2010. [Online]. Available: https://doi.org/10.1201/b10274

    doi:10.1201/b10274 2010

  21. [24]

    Automated assessment in computer science education: A state-of-the-art review,

    J. C. Paiva, J. P. Leal, and A. Figueira, “Automated assessment in computer science education: A state-of-the-art review,”ACM Trans. Comput. Educ., vol. 22, no. 3, 2022. [Online]. Available: https://doi.org/10.1145/3513140

    doi:10.1145/3513140 2022

  22. [26]

    C. Lang, G. Siemens, A. Wise, D. Gaševi ´c, and A. Merceron,The Handbook of Learning Analytics, 2nd ed. Canada: SoLAR, 2022. [Online]. Available: https://doi.org/10.18608/hla22

    doi:10.18608/hla22 2022

  23. [29]

    Robust team communication analytics with transformer-based dialogue modeling,

    J. Pande, W. Min, R. D. Spain, J. D. Saville, and J. Lester, “Robust team communication analytics with transformer-based dialogue modeling,” in Artificial Intelligence in Education, vol. 13916. Springer, 2023, pp. 639–

    2023

  24. [30]

    The history of computing education research,

    M. Guzdial and B. du Boulay, “The history of computing education research,” inThe Cambridge Handbook of Computing Education Research. Cambridge University Press, 2019, ch. 1, pp. 11–39. [Online]. Available: https://doi.org/10.1017/9781108654555

    work page doi:10.1017/9781108654555 2019

  25. [31]

    Practical cybersecurity education: A course model using experiential learning theory,

    S. Narain, P. Rayavaram, C. Morales-Gonzalez, M. Harper, M. Abbasalizadeh, K. Vellamchety, and X. Fu, “Practical cybersecurity education: A course model using experiential learning theory,” in56th ACM Tech. Symposium on Comp. Sci. Educ.ACM, 2025, p. 819–825. [Online]. Available: https://doi.org/10.1145/3641554.3701922

    arXiv 2025

  26. [32]

    A cybersecurity summer camp for high school students using autonomous r/c cars,

    M. Won, L. R. Carrington, D. M. Espinoza, M. H. Ali, and D. Dasgupta, “A cybersecurity summer camp for high school students using autonomous r/c cars,” inTech. Symposium on Comp. Sci. Educ., ser. SIGCSE. New York, NY , USA: ACM, 2024, p. 1435–1441. [Online]. Available: https://doi.org/10.1145/3626252.3630758

    arXiv 2024

  27. [33]

    Tackling the gender gap in cybersecurity education,

    G. Costa, S. De Francisci, M. Renieri, and S. Valiani, “Tackling the gender gap in cybersecurity education,” inProceedings of the 56th ACM Technical Symposium on Computer Science Education, ser. SIGCSE. New York, NY , USA: ACM, 2025, p. 234–240. [Online]. Available: https://doi.org/10.1145/3641554.3701807

    arXiv 2025

  28. [34]

    Remote controlled cyber: Toward engaging and educating a diverse cybersecurity workforce,

    C. Gough, C. Mann, C. Ficke, M. Namukasa, M. Carroll, and T. OConnor, “Remote controlled cyber: Toward engaging and educating a diverse cybersecurity workforce,” inProceedings of the 55th ACM Technical Symposium on Computer Science Education. ACM, 2024, p. 394–400. [Online]. Available: https://doi.org/10.1145/3626252.3630917

    arXiv 2024

  29. [36]

    Student Assessment in Cybersecurity Training Automated by Pattern Mining and Clustering,

    V . Švábenskýet al., “Student Assessment in Cybersecurity Training Automated by Pattern Mining and Clustering,”Educ. and Inf. Tech.,

  30. [37]

    Clustering and sequential pattern mining of online collaborative learning data,

    D. Perera, J. Kay, I. Koprinska, K. Yacef, and O. R. Zaïane, “Clustering and sequential pattern mining of online collaborative learning data,”IEEE Transactions on knowledge and Data Engineering, vol. 21, no. 6, pp. 759– 772, 2008. [Online]. Available: https://doi.org/10.1109/TKDE.2008.138

    work page doi:10.1109/tkde.2008.138 2008

  31. [38]

    Online collaboration: Collaborative behavior patterns and factors affecting globally distributed team performance,

    F. C. Serçe, K. Swigger, F. N. Alpaslan, R. Brazile, G. Dafoulas, and V . Lopez, “Online collaboration: Collaborative behavior patterns and factors affecting globally distributed team performance,”Computers in human behavior, vol. 27, no. 1, pp. 490–503, 2011. [Online]. Available: https://doi.org/10.1016/j.chb.2010.09.017

    work page doi:10.1016/j.chb.2010.09.017 2011

  32. [39]

    Characterizing team orientations and academic performance in cooperative project-based learning environments,

    A. Jaiswal, T. Karabiyik, P. Thomas, and A. J. Magana, “Characterizing team orientations and academic performance in cooperative project-based learning environments,”Education Sciences, vol. 11, no. 9, p. 520, 2021. [Online]. Available: https://doi.org/10.3390/educsci11090520

    work page doi:10.3390/educsci11090520 2021

  33. [40]

    A systematic review on educational data mining,

    A. Dutt, M. A. Ismail, and T. Herawan, “A systematic review on educational data mining,”IEEE Access, vol. 5, pp. 15 991–16 005, 2017. [Online]. Available: https://doi.org/10.1109/access.2017.2654247

    arXiv 2017

  34. [41]

    Enhancing student focus and problem-solving with real-time llm feedback on compiler errors,

    M. Pankiewicz and R. S. Baker, “Enhancing student focus and problem-solving with real-time llm feedback on compiler errors,” inTwo Decades of TEL. From Lessons Learnt to Challenges Ahead. Cham: Springer Nature Switzerland, 2025, pp. 412–426. [Online]. Available: https://doi.org/10.1007/978-3-032-03870-8_28

    work page doi:10.1007/978-3-032-03870-8_28 2025

  35. [42]

    Available: https://doi.org/10.1007/s10639-022-10954-4

    [Online]. Available: https://doi.org/10.1007/s10639-022-10954-4

    work page doi:10.1007/s10639-022-10954-4

  36. [44]

    Comparing few-shot prompting of gpt-4 llms with bert classifiers for open-response assessment in tutor equity training,

    S. Kakarla, C. Borchers, D. Thomas, S. Bhushan, and K. R. Koedinger, “Comparing few-shot prompting of gpt-4 llms with bert classifiers for open-response assessment in tutor equity training,” 2025. [Online]. Available: https://arxiv.org/abs/2501.06658

    Pith/arXiv arXiv 2025

  37. [46]

    ChatGPT for good? On opportunities and challenges of large language models for education,

    E. Kasneci, K. Seßler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasseret al., “ChatGPT for good? On opportunities and challenges of large language models for education,”Learning and individual differences, vol. 103, 2023. [Online]. Available: https://doi.org/10.1016/j.lindif.2023.102274

    arXiv 2023

  38. [47]

    Automatic Short Answer Grading in the LLM Era: Does GPT-4 with Prompt Engineering beat Traditional Models?

    R. Ferreira Mello, C. Pereira Junior, L. Rodrigues, F. D. Pereira, L. Cabral, N. Costa, G. Ramalho, and D. Gasevic, “Automatic Short Answer Grading in the LLM Era: Does GPT-4 with Prompt Engineering beat Traditional Models?” inProceedings of the 15th International Learning Analytics and Knowledge Conference. ACM, 2025, p. 93–103. [Online]. Available: http...

    arXiv 2025

  39. [48]

    On Assessing the Faithfulness of LLM-generated Feedback on Student Assignments,

    Q. Jia, J. Cui, R. Xi, C. Liu, P. Rashid, R. Li, and E. Gehringer, “On Assessing the Faithfulness of LLM-generated Feedback on Student Assignments,” inProceedings of the 17th International Conference on Educational Data Mining. MA, USA: IEDMS, 2024, pp. 491–499. [Online]. Available: https://doi.org/10.5281/zenodo.12729868

    work page doi:10.5281/zenodo.12729868 2024

  40. [50]

    Self-regulated learning processes in secondary education: A network analysis of trace-based measures,

    Y . Cheng, R. Guan, T. Li, M. Rakovi ´c, X. Li, Y . Fan, F. Jin, Y .-S. Tsai, D. Gaševi ´c, and Z. Swiecki, “Self-regulated learning processes in secondary education: A network analysis of trace-based measures,” inProceedings of the 15th International Learning Analytics and Knowledge Conference. ACM, 2025, p. 260–271. [Online]. Available: https://doi.org/...

    arXiv 2025

  41. [51]

    Analytics of scaffold compliance for self-regulated learning,

    J. Saint, Y . Fan, and D. Gasevic, “Analytics of scaffold compliance for self-regulated learning,” inProceedings of the 14th Learning Analytics and Knowledge Conference. New York, NY , USA: Association for Computing Machinery, 2024, p. 326–337. [Online]. Available: https://doi.org/10.1145/3636555.3636887

    arXiv 2024

  42. [52]

    Feedback on feedback: Student’s perceptions for feedback from teachers and few-shot llms,

    S. Rüdian, J. Podelo, J. Kužílek, and N. Pinkwart, “Feedback on feedback: Student’s perceptions for feedback from teachers and few-shot llms,” in Proc. of the 15th Intl. Learning Analytics and Knowledge Conf.ACM,

  43. [53]

    From Paper to Platform: Evolution of a Novel Learning Environment for Tabletop Exercises,

    V . Švábenský, J. Vykopal, M. Horák, M. Hofbauer, and P. ˇCeleda, “From Paper to Platform: Evolution of a Novel Learning Environment for Tabletop Exercises,” inInnovation and Technology in Computer Science Education. New York, NY , USA: ACM, 2024, pp. 213–219. [Online]. Available: https://doi.org/10.1145/3649217.3653639

    arXiv 2024

  44. [54]

    Automatic detection of collaborative states in small groups using multimodal features,

    M. Bradford, I. Khebour, N. Blanchard, and N. Krishnaswamy, “Automatic detection of collaborative states in small groups using multimodal features,” inArtificial Intelligence in Education. Springer, 2023, pp. 767–773. [Online]. Available: https://doi.org/10.1007/ 978-3-031-36272-9_69

    2023

  45. [55]

    Analytics of planning behaviours in self- regulated learning: Links with strategy use and prior knowledge,

    T. Li, Y . Fan, N. Srivastava, Z. Zeng, X. Li, H. Khosravi, Y .-S. Tsai, Z. Swiecki, and D. Gaševi ´c, “Analytics of planning behaviours in self- regulated learning: Links with strategy use and prior knowledge,” inProc. of the 14th Learning Analytics and Knowledge Conf.ACM, 2024, p. 438–449. [Online]. Available: https://doi.org/10.1145/3636555.3636900

    arXiv 2024

  46. [56]

    Consensus clusterings,

    N. Nguyen and R. Caruana, “Consensus clusterings,” in7th IEEE International Conference on Data Mining. IEEE, 2007, pp. 607–612. [Online]. Available: https://doi.org/10.1109/ICDM.2007.73

    work page doi:10.1109/icdm.2007.73 2007

  47. [57]

    CyLLM-DAP: Cybersecurity Domain-Adaptive Pre-Training Framework of Large Language Models,

    K. Mai, R. Beuran, and N. Inoue, “CyLLM-DAP: Cybersecurity Domain-Adaptive Pre-Training Framework of Large Language Models,” in11th Int. Conf. on Inf. Systems Security and Privacy. SCITEPRESS, 2025, pp. 24–35. [Online]. Available: https://www.scitepress.org/Papers/ 2025/130948/130948.pdf

    2025

  48. [58]

    Dojo: Applied cybersecurity education in the browser,

    C. Nelson and Y . Shoshitaishvili, “Dojo: Applied cybersecurity education in the browser,” inTechnical Symposium on Computer Science Education, ser. SIGCSE. New York, NY , USA: ACM, 2024, p. 930–936. [Online]. Available: https://doi.org/10.1145/3626252.3630836

    arXiv 2024

  49. [59]

    Incident response,

    National Initiative for Cybersecurity Careers and Studies (NICCS), “Incident response,” 2020. [Online]. Available: https://niccs.cisa.gov/ tools/nice-framework/work-role/incident-response

    2020

  50. [60]

    Available Exercise Definitions,

    INJECT Team, “Available Exercise Definitions,” 2026. [Online]. Avail- able: https://docs.inject.muni.cz/INJECT_process/available-definitions

    2026

  51. [61]

    Dataset From Cybersecurity Tabletop Exercises in the INJECT Platform,

    Vykopal, Jan and Švábenský, Valdemar and ˇCeleda, Pavel, “Dataset From Cybersecurity Tabletop Exercises in the INJECT Platform,” 2026, v. 1.1.0. [Online]. Available: https://doi.org/10.5281/zenodo.21396276

    work page doi:10.5281/zenodo.21396276 2026

  52. [62]

    Competency model,

    U.S. Office of Personnel Management, “Competency model,”

  53. [67]

    Using online learner trace data to understand the cohesion of teams in higher education,

    A. Zamecnik, V . Kovanovíc, S. Joksimovíc, G. Grossmann, D. Ladjal, R. Marshall, and A. Pardo, “Using online learner trace data to understand the cohesion of teams in higher education,”Journal of Computer Assisted Learning, vol. 39, no. 6, pp. 1733–1750, 2023. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/jcal.12829

    work page doi:10.1111/jcal.12829 2023

  54. [70]

    Results, visualizations, and analytical toolset,

    Paper authors, “Results, visualizations, and analytical toolset,” 2026. [On- line]. Available: https://gitlab.fi.muni.cz/inject/papers/2026-fie-assessment

    2026

  55. [248]

    Available: https://doi.org/10.1007/978-3-031-84260-3_14

    [Online]. Available: https://doi.org/10.1007/978-3-031-84260-3_14

    doi:10.1007/978-3-031-84260-3_14

  56. [650]

    Available: https://doi.org/10.1007/978-3-031-36272-9_52

    [Online]. Available: https://doi.org/10.1007/978-3-031-36272-9_52

    work page doi:10.1007/978-3-031-36272-9_52

  57. [2011]

    Available: https://www.opm.gov/chcoc/transmittals/ 2011/competency-model-cybersecurity_02-16-2011_508.pdf

    [Online]. Available: https://www.opm.gov/chcoc/transmittals/ 2011/competency-model-cybersecurity_02-16-2011_508.pdf

    2011

  58. [2020]

    Available: https://doi.org/10.14710/nmjn.v10i2.29117

    [Online]. Available: https://doi.org/10.14710/nmjn.v10i2.29117

    work page doi:10.14710/nmjn.v10i2.29117

  59. [2022]

    Available: https://doi.org/10.5220/0010845800003120

    [Online]. Available: https://doi.org/10.5220/0010845800003120

    work page doi:10.5220/0010845800003120