REVIEW 4 major objections 7 minor 3 references
Quantifying Loss Aversion in Cyber Adversaries via LLM Analysis
T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read This paper claims that large language models can extract behavioral evidence of loss aversion from hackers' operational notes by counting persistence techniques, with lower self-reported risk propensity predicting more persistence.
desk verdict A transparent proof-of-concept for LLM-based annotation of hacker notes, but the paper's own statistics do not support the loss-aversion claim it leads with. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the two-stage LLM annotation pipeline built on GPT-4o. First, the model segments each participant's operational notes into discrete timestamped actions; second, it classifies each action against the MITRE ATT&CK framework's persistence techniques, producing explicit reasoning before assigning a label. The output is a per-participant count of persistence techniques, which the paper treats as a behavioral proxy for loss aversion and feeds into Pearson correlations and a multivariate linear regression with GRiPS, ADMC RC1, ADMC RC2, and participant division as predictors. This machinery is what converts unstructured self-report text into a testable quantitative signal.
What would settle it
A reader could settle this by having two expert human coders independently label the same 17 sets of operational notes for persistence techniques and for stated motivations, then comparing their tallies to the LLM's counts; if the LLM counts diverge substantially, or if a larger preregistered sample fails to reproduce the negative GRiPS coefficient, the central claim would be falsified.
Extended reading notes
Core claim
The paper reports that LLM-based analysis of operational notes from a controlled red-team exercise can extract temporal action sequences, tag actions with MITRE ATT&CK persistence techniques, and link those tags to psychometric indicators of loss aversion. The flagship quantitative result is a negative relationship between General Risk Propensity Scale (GRiPS) scores and persistence counts: participants with lower self-reported risk propensity used more persistence techniques ($\beta = -4.42$, $p = 0.045$; $r = -0.43$, $p = 0.065$). The two Adult Decision-Making Competence (ADMC) resistance-to-framing subscales, RC1 and RC2, did not significantly predict persistence usage ($p = 0.121$ and $p = 0.290$), and division membership was only marginal ($p = 0.109$). The model explained 37.6 percent of the variance (adjusted $R^2 = 0.198$) but was not significant overall, and the authors interpret the GRiPS finding as consistent with loss aversion while acknowledging the need for replication with larger datasets.
Load-bearing premise
The load-bearing premise is that the count of persistence techniques an LLM extracts from a hacker's notes is a valid behavioral stand-in for loss aversion, with no independent check that those actions reflect fear of losing access rather than routine tradecraft or model artifacts.
Editorial extensions
If this is right
- If the central claim holds, defender systems can process attacker-authored notes and treat persistence-heavy behavior as a real-time indicator of loss aversion, enabling adaptive countermeasures.
- The significant GRiPS coefficient implies that a standard psychometric risk-propensity questionnaire, combined with LLM-extracted behavioral counts, can anticipate how much effort an attacker will invest in maintaining access.
- The null ADMC framing results imply that susceptibility to gain/loss framing is a weaker signal than general risk propensity in operational cyber settings, steering future measurement toward risk-propensity instruments.
- Because the annotation pipeline is modular, the same segmentation-and-classification approach can be applied to other MITRE tactics and to other cognitive biases, such as the sunk-cost fallacy or confirmation bias.
Reading between the lines
- A validation study comparing LLM-extracted persistence counts against independent expert coders and against a direct behavioral loss-aversion task (e.g., mixed gambles) would settle whether the counts measure loss aversion rather than routine attacker tradecraft.
- The gap between raw and adjusted R-squared (0.376 vs 0.198) and the sample size of 17 suggest the effect size is likely optimistic; a preregistered replication would clarify how much of the association is real.
- If the proxy is validated, the same pipeline could run on machine-generated network logs instead of self-reported notes, allowing near-real-time cognitive inference in live intrusions.
- The combination of a significant GRiPS effect and null framing effects hints that what the paper calls loss aversion may actually be general risk avoidance rather than framing-specific loss aversion; a targeted experiment with loss-framed scenarios could resolve this.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper describes a proof-of-concept study in which GPT-4o is used to parse operational notes (OPNOTES) written by 17 penetration testers during a controlled red-team exercise and to label actions against MITRE ATT&CK persistence techniques. The authors then compute counts of persistence techniques per participant and relate these counts to three psychometric measures: GRiPS (general risk propensity) and two ADMC Resistance to Framing subscales (RC1 for gains, RC2 for losses). The results show a moderate negative correlation between GRiPS and persistence count (r = -0.43, p = 0.065) and a nominally significant negative regression coefficient for GRiPS (β = -4.42, p = 0.045); the ADMC subscales are non-significant, and the overall regression model is non-significant (F(4,14) = 2.11, p = 0.133). The authors conclude that the approach demonstrates that LLMs can reliably extract temporal action sequences, identify MITRE persistence techniques, and associate these actions with cognitive traits such as loss aversion.
Significance. If the central claim were established, the work would offer a scalable method for extracting behavioral indicators of cognitive bias from off-the-shelf text artifacts, with potential applications in active cyber defense. The paper has several strengths: the use of externally validated psychometric instruments (GRiPS, ADMC) and the transparency of the statistical reporting (confidence intervals, model fit statistics). The attempt to ground a cognitive construct in observable attack behaviors is a valuable direction for the ReSCIND program. However, the evidence presented does not support the specific claim of loss aversion: the scale designed to measure loss-framing resistance (ADMC RC2) is non-significant, the only significant predictor is a generic risk propensity measure, and the proxy linking persistence counts to loss aversion is asserted rather than validated. As presently stated, the title and conclusion outrun the data.
major comments (4)
- [Cognitive Bias Hypothesis] The central premise that 'an adversary exhibiting loss aversion will show a heightened focus on maintaining access through the use of persistence mechanisms' is asserted without supporting evidence or a literature citation. No data are provided to show that persistence behavior is driven by loss aversion rather than by mission requirements, expertise, or habits. This is load-bearing because the dependent variable in the analysis is persistence count; without construct validation, the statistical results cannot be interpreted as evidence about loss aversion. The paper should provide external validation (e.g., expert ratings of whether each persistence action was motivated by loss aversion) or visibly restrict the claims to 'risk aversion' or 'persistence behavior'.
- [Data Processing & Annotation Pipeline] The paper claims that LLMs can 'reliably extract temporal action sequences' and that classifications were 'verified during manual review', but no reliability metrics are reported. In particular, the reader cannot assess precision or recall of the LLM's MITRE persistence technique annotations against human expert annotations, inter-annotator agreement, or the consistency of the segmentation. Since the persistence counts that form the dependent variable come entirely from this pipeline, a systematic bias in the LLM's labeling could create the observed correlations. Report agreement statistics on a held-out subset or a comparison against a human-labeled baseline.
- [Results and Findings / Table 2] The regression evidence is too weak to support the paper's conclusion. The overall model is non-significant (F(4,14) = 2.11, p = 0.133), the adjusted R² is only 0.198, and the single nominally significant coefficient (GRiPS, p = 0.045) would not survive a conservative multiple-comparison correction across the three psychometric predictors plus division. With n = 17, the estimate is fragile. The paper should present the results as exploratory, report effect sizes with confidence intervals, and avoid causal or strong confirmatory language in the abstract and conclusion.
- [Analysis Approach] The interpretation of GRiPS as a surrogate for loss aversion is not defended. GRiPS measures general risk propensity, and the loss-framing subscale RC2, which is the operationalization most directly tied to loss aversion, is non-significant. To claim loss aversion specifically, the authors need a theoretical or empirical justification for why low GRiPS should be equated with high loss aversion in this population, or they should reframe the paper to be about risk aversion rather than loss aversion. Without this, the title and abstract overstate the construct.
minor comments (7)
- [Abstract] The sentence 'We process the hacker generated notes using LLMs using it to segment the various actions' is grammatically awkward and should be reworded.
- [Related Work] In the Related Work section, 'take palace' should be 'take place'.
- [About the Authors] The author biography contains 'mulitmodal', which should be 'multimodal'.
- [Table 1] The 'LA_' prefix in the column headers is undefined; please define the abbreviation or rename the columns for clarity.
- [Figure 1] The axes of Figure 1 are not labeled; the y-axis presumably denotes occurrence counts but this is not identified.
- [References] The reference 'Kahneman, D., & Tversky, A. (2013)' is a reprint; consider citing the original 1979 Econometrica article or clearly noting that it is a reprint, and check consistency of author name ordering elsewhere (e.g., 'Nir, D.' versus 'Daniel et al.' in the text).
- [Research Methods / Experimental Setup] The paper does not mention institutional review board approval or ethical approval for the human-subjects experiment, despite recruiting human participants as penetration testers; this should be stated.
Circularity Check
No circularity found: the derivation chain is self-contained, with independent LLM labeling, fixed psychometric instruments, and no fitted-constant feedback loop.
full rationale
The paper's derivation chain contains no fitted-constant feedback loop and no load-bearing self-citation. Persistence counts are generated from participant OPNOTES through a GPT-4o annotation pipeline keyed to predefined MITRE ATT&CK persistence techniques; psychometric scores (GRiPS, ADMC RC1, RC2) come from fixed, externally validated instruments collected independently of the OPNOTES annotation. The regression model regresses persistence counts on those fixed psychometric scores, so no predictor is constructed from the outcome and no parameter is fit to a subset and then 'predicted' for a closely related quantity. The paper's central inferential weakness—that persistence counts are an unvalidated behavioral proxy for loss aversion, as evidenced by the non-significant ADMC RC2 loss-framing result—is a construct-validity/correctness concern rather than a circular derivation: the authors do not define loss aversion and persistence as mutually equivalent by construction, nor do they import a uniqueness theorem or ansatz from prior self-citation. The external psychometric scales provide independent, non-circular evidence relative to the behavioral outcome, even if the proxy assumption is not established. Accordingly, the circularity score is 0.
Assumptions & free parameters
assumptions (5)
- domain assumption LLM annotations of action segmentation and MITRE persistence classification are accurate enough for statistical analysis.
- ad hoc to paper Persistence technique count is a valid behavioral proxy for loss aversion.
- ad hoc to paper GRiPS measures a construct close enough to loss aversion to serve as a surrogate.
- domain assumption Self-reported OPNOTES reliably capture the actions and decisions of the hackers.
- standard math Standard statistical assumptions for correlation and multiple regression hold with n = 17.
Cite this review
Pith. "Pith review of Quantifying Loss Aversion in Cyber Adversaries via LLM Analysis." pith.science (2026). https://pith.science/paper/OQFV2ZUW
@misc{pith2026250813240,
author = {Pith},
title = {Pith review of: Quantifying Loss Aversion in Cyber Adversaries via LLM Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/OQFV2ZUW}},
note = {Machine review of arXiv:2508.13240}
}
read the original abstract
Understanding and quantifying human cognitive biases from empirical data has long posed a formidable challenge, particularly in cybersecurity, where defending against unknown adversaries is paramount. Traditional cyber defense strategies have largely focused on fortification, while some approaches attempt to anticipate attacker strategies by mapping them to cognitive vulnerabilities, yet they fall short in dynamically interpreting attacks in progress. In recognition of this gap, IARPA's ReSCIND program seeks to infer, defend against, and even exploit attacker cognitive traits. In this paper, we present a novel methodology that leverages large language models (LLMs) to extract quantifiable insights into the cognitive bias of loss aversion from hacker behavior. Our data are collected from an experiment in which hackers were recruited to attack a controlled demonstration network. We process the hacker generated notes using LLMs using it to segment the various actions and correlate the actions to predefined persistence mechanisms used by hackers. By correlating the implementation of these mechanisms with various operational triggers, our analysis provides new insights into how loss aversion manifests in hacker decision-making. The results demonstrate that LLMs can effectively dissect and interpret nuanced behavioral patterns, thereby offering a transformative approach to enhancing cyber defense strategies through real-time, behavior-based analysis.
Figures
Reference graph
Works this paper leans on
-
[1]
Al-Sada, B., Sadighian, A., & Oligeri, G. (2024). Mitre att&ck: Stat e of the art and way forward. ACM Computing Surveys, 57(1), 1-37. Roy, S., Panaousis, E., Noakes, C., Laszka, A., Panda, S. , & Loukas, G. (2023). SoK: The MITRE ATT&CK Framework in Research and Practice
work page 2024
-
[23]
Zhou, X., Zhang, T., & Lo, D. (2024, April). Large language model for vulnerability detection: Emerging results and future directions. In Proceedings of the 2024 ACM/IEEE 44th International Confer ence on Software Engineering: New Ideas and Emerging Results (pp. 47-51). Wang, G., Sun, Z., Gong, Z., Ye, S., Chen, Y., Zhao, Y., ... & Hao, D. (2024). Do adva...
arXiv 2024
-
[2023]
URL https://arxiv. org/abs/2304.07411. Oosthoek, K., & Doerr, C. (2019). Sok: Att&ck techniques and trends in windows malware. In Security and Privacy in Communication Networks: 15th EAI International Conference, SecureComm 2019, Orlando, FL, USA, October 23- 25, 2019, Proceedings, Part I 15 (pp. 406-425). Springer International Publishing. Ferguson-Walte...
arXiv 2019
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.