Pith. sign in

REVIEW 2 major objections 4 minor 35 references

Non-Degenerate Risk Certification for Automated Security Decisions: A Decision-Contract Theory with ATT\&CK-Aligned Triage as a Worked Instance

T0 review · 2 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A risk bound satisfied by doing nothing certifies nothing; the paper's decision-contract theory fixes it by adding an action-rate floor.

desk verdict A sound and honest formal treatment of why unconditional risk bounds are vacuous without an action-rate floor, with a useful capacity diagnostic—but the ATT&CK label mapping behind the empirical instantiation is unvalidated, so the applied claims are category-level, not true ATT&CK triage. read the letter →

arxiv 2608.12444 v1 pith:7TZS6HZK submitted 2026-08-12 cs.CR cs.LG

Zhenpeng Li This is my paper · ORCID
classification cs.CRcs.LG
keywords decisioncontractriskcertificationconformalcontrolselectivepredictionfalseautomatedattributionsingletoncapacityLLMintrusiondetectionalerttriage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

An unconditional risk bound — 'the probability of a wrong automated output is at most $\alpha$' — is not a diagnostic of a good predictor: a system that never acts, or that reports at a coarser label resolution, satisfies it without automating anything useful. The paper formalizes this as a decision-contract theory, in which any risk certificate is defined over a selector (what the system acts on) and a correctness relation (what counts as correct), and weakening either hides base-classifier error. The central accounting law is the error-conservation identity $B(h)=R(C)+D(C)+M(C)$: a base classifier's error is only reassigned among harmful automation, human deferral, and semantic masking, never eliminated. The practical claim is that a non-degenerate certificate must jointly enforce a risk bound $R(C)\le \alpha$ and an action-rate floor $A(C)\ge \rho$, which the paper demonstrates on LLM-based ATT&CK-aligned alert triage, where 90.3% of configurations meet the risk target and the capacity diagnostic explains every low-utility configuration.

What carries the argument

The load-bearing object is the decision contract $C=(g,E)$: a selector $g$ that decides for each input whether the system acts, and a reflexive acceptance relation $E$ that decides when an automated output is counted as correct. Three derived constructs carry the argument: the error-conservation identity $B(h)=R(C)+D(C)+M(C)$ (Theorem 3), which shows that abstention and semantic coarsening are the only two ways to hide base-classifier error from a risk bound; the singleton capacity $\kappa(f)=\sup_\tau [F_2^-(\tau)-F_1^-(\tau)]$ (Theorem 6), a label-free upper bound on the action rate attainable by any global threshold, together with its labeled refinement $\kappa_\alpha(f)$ (Definition 4) that distinguishes recoverable threshold misalignment from risk-constrained incapacity; and the $(\alpha,\rho)$-actionability certificate (Theorem 8 with Proposition 5), which pairs the conformal risk guarantee with a finite-sample Hoeffding lower bound on the deployed threshold's action rate.

What would settle it

Have independent security analysts annotate a held-out sample of the same IDS alerts directly with ATT&CK technique identifiers, then compare with the paper's $\phi$ mapping: material disagreement would show that the reported false-attribution risk is a four-class category error rather than ATT&CK misattribution risk. A complementary check on any dataset carrying both fine and coarse labels would verify the accounting by computing $R_{\mathrm{fine}}(g)$, $R_\phi(g)$, and $M_\phi(g)$ and testing whether $R_\phi=R_{\mathrm{fine}}-M_\phi$ holds.

Watch

Extended reading notes

Core claim

The paper's central claim is that an unconditional risk bound such as $\mathrm{FAR}\le\alpha$ is structural, not diagnostic: it certifies a decision contract $C=(g,E)$ — the set of inputs the selector $g$ acts on, plus the semantic relation $E$ under which an output counts correct — rather than the underlying predictor. Two mechanisms that are unrelated to predictive competence can drive the measured risk to zero, namely abstaining on more inputs and reporting at a coarser label resolution, and the paper shows these are the only two. The argument is carried by the error-conservation law $B(h)=R(C)+D(C)+M(C)$ (Theorem 3), which accounts for every base-classifier error as harmful automation, human deferral, or semantic masking; by the exact fine-to-coarse risk-transfer identity $R_\phi(g)=R_{\mathrm{fine}}(g)-M_\phi(g)$ (Theorem 4), under which coarsening reduces measured risk by exactly the within-fiber confusion mass $M_\phi(g)$, with a reverse-transfer impossibility result (Theorem 5); and by the $(\alpha,\rho)$-actionability certificate (Definition 5, Theorem 8), which converts a risk bound into both a correct-automation lower bound and a conditional-error upper bound while excluding all-abstain solutions by construction. On the empirical side, across 3 IDS datasets, 6 LLMs, and 4 error-rate thresholds, the label-free singleton capacity $\kappa(f)$ rules out structural incapacity for every configuration, while the risk-feasible refinement $\kappa_\alpha(f)$ separates two risk-constrained-incapacity cases from one genuinely recoverable threshold misalignment.

Load-bearing premise

The whole empirical instantiation rests on a hand-built bijection from four IDS attack categories to four ATT&CK techniques that is not independently sourced; if that mapping is semantically invalid, the reported risk numbers are four-class category error rates, not ATT&CK misattribution rates, and the triage conclusions do not transfer to the real ATT&CK taxonomy.

Editorial extensions

If this is right

  • A deployment that certifies only an unconditional risk bound is potentially vacuous: an all-abstain system or a coarser label report satisfies it while automating nothing, so practitioners should treat FAR-only certification as insufficient.
  • With an $(\alpha,\rho)$-actionability certificate, a deployer obtains a correct-automation lower bound $U\ge \rho-\alpha$ and a conditional-error upper bound $Q\le \alpha/\rho$ at a stated confidence on the action-rate side.
  • The label-free capacity check $\kappa(f)$ can be run before any labeled calibration: if $\kappa(f)<\rho$, no single global threshold can reach the automation floor, and the remedy is a better base classifier, not more calibration data.
  • When automation falls short despite adequate $\kappa(f)$, comparing $\kappa_\alpha(f)$ with the realized action rate $A(\hat{\tau})$ decides whether recalibration can close the gap (threshold misalignment) or whether the risk target itself is the binding constraint (risk-constrained incapacity).
  • Coarsening a label space transfers risk rather than removing it, by exactly the within-fiber confusion mass $M_\phi(g)$; technique-level guarantees should not be promoted to tactic-level guarantees without reporting this masked mass.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond IDS triage, any risk-controlled automation layer — clinical decision support, LLM tool use, content moderation — faces the same all-abstain vacuity, so requiring an action-rate floor should be the default wherever an unconditional risk bound gates actions.
  • The coarsening-transfer identity suggests a practical audit: whenever a system is evaluated at a coarser label resolution than the labels it was trained on, measuring $M_\phi(g)$ directly would reveal how much fine-grained error is being hidden — a quantity that is small in this benchmark set but not guaranteed to stay small in richer taxonomies such as the full ATT&CK matrix.
  • Proposition 4's counterexample implies that score-variance or accuracy-based deployability heuristics should be replaced by capacity-style diagnostics computed from top-two probability gaps; this is an inference about neighbouring practice, not a claim the paper tests empirically.
  • An editorial reading of the retracted LightGBM episode: structural-incapacity verdicts about a stochastic base learner should survive a multi-seed training-stability check before a system is redesigned or abandoned.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper argues that an unconditional risk bound on automated decisions is vacuous, since a selector that never acts trivially satisfies it. It develops a decision-contract theory in which a certificate is defined over a pair (selector, correctness relation), proves an error-conservation law (Theorem 3) showing that base-classifier error is only reassigned among harmful automation, deferral, and semantic masking, and derives a coarsening-transfer identity (Theorem 4) with a reverse impossibility result (Theorem 5). It then introduces a label-free singleton capacity κ(f) (Theorem 6) as a necessary condition for a target action rate, a risk-feasible refinement κα(f) (Definition 4) that separates threshold misalignment from risk-constrained incapacity, and an (α,ρ)-actionability certificate (Theorem 8) with a finite-sample Hoeffding lower bound on the action rate (Proposition 5). The framework is instantiated on ATT&CK-aligned alert triage for LLM-based intrusion detection across 3 IDS datasets, 6 LLMs, and 4 error thresholds, reporting that empirical false-attribution risk stays at or below target in 90.3% of configurations, with 83.4% mean correct automation. The capacity diagnostics are used to classify the three lowest-utility configurations, and a coarsening-transfer identity is verified under a genuine many-to-one map using fine-grained attack-subtype labels.

Significance. The formal theory is a valuable generalization of selective classification and conformal risk control: it makes the correctness relation part of the certified object and provides an exact accounting identity connecting harmful automation, deferral, and semantic masking. The finite-sample capacity certificate (Proposition 3) and the action-rate lower-bound certificate (Proposition 5) are practical tools, and Theorem 8 gives a clean bridge from unconditional risk to correct-automation and conditional-error bounds. The paper is unusually transparent: it explicitly acknowledges the proxy-label limitation of the ATT&CK mapping, retracts an initially reported structural-incapacity instance after a 20-seed training-stability check, and confirms the one recoverable misalignment case with an out-of-sample threshold evaluation. If the ATT&CK mapping were independently validated, the empirical instantiation would be a strong demonstration; as it stands, the theoretical contribution is sound, self-contained, and likely to be influential regardless of the empirical caveat.

major comments (2)
  1. [Section III-A, Section III-B, Table I] The central applied claim is that the paper instantiates the theory on ATT&CK-aligned alert triage, but the only bridge from IDS category labels to ATT&CK techniques is the hand-built bijection φ in Table I (DoS→T1498, CredentialAccess→T1110, Exploitation→T1190, Probe→T1046). Section III-A explicitly says this is 'not an independently sourced ATT&CK ground-truth annotation' and that y(x) is used in an 'operational sense.' Consequently, the conformal risk bound of Theorem 1, the empirical 90.3% pass rate, and all triage conclusions certify risk with respect to these proxy labels. If φ is semantically wrong — for example, if 'Exploitation' alerts are heterogeneous and span multiple ATT&CK techniques, or if any IDS category maps to a technique other than the one assumed — the reported false-attribution risk is a four-class category error, not an ATT&CK misattribution rate, and the triage conclusions do not transfer to the real ATT&CK taxonomy. The authors should either validate φ against independently sourced ATT&CK annotations on a subset of alerts, or reframe the title, abstract, and empirical claims in terms of coarse IDS-category attribution, explicitly dropping the ATT&CK-alignment claim for the empirical part while retaining it as an operational simplifying assumption.
  2. [Section IV-F, Proposition 5, Theorem 8] The actionability certificate is presented as a single non-degenerate certificate, but it actually pairs two guarantees of different character: the risk side is the standard marginal conformal statement of Theorem 1 (a probability over the joint draw of calibration set and test point, not a confidence interval on the realized population risk), while the action-rate side is a (1−δ) finite-sample lower bound for the fixed, already-realized threshold τ̂. The bounds U ≥ Aδ − α and Q ≤ α/Aδ in Theorem 8 hold on the probability-(1−δ) event {A(τ̂) ≥ Aδ}, and the risk inequality R ≤ α is not conditional on that event. The paper acknowledges that it does not combine the two into a single joint confidence level, but the abstract and contribution list present this as one 'non-degenerate actionability certificate.' To make the certificate's probabilistic content precise for practitioners, the authors should add an explicit union-bound or PAC-style joint statement (e.g., P(R ≤ α and A ≥ Aδ) ≥ 1 − α − δ under a high-probability risk bound), or clearly state in the abstract and in Section IV-F that the certificate provides two separate guarantees that are not simultaneously certified at a single stated level.
minor comments (4)
  1. [Table II, Section VII-C0e] The column labeled 'Test Samples' in Table II appears to denote total attack samples, since the certification split for CIC-IDS-2018 in Section VII-C0e is m = 1,288 = 1,488 − 200; please rename the column to 'Total Attack Samples' or clarify the relationship between the tabulated counts, the 200-sample calibration set, and the certification/test split.
  2. [Table IV, HIKARI-2021 rows] For HIKARI-2021, the stated test set size after removing 200 calibration samples is 26, so per-seed utility must be k/26 and the 5-seed mean must be a multiple of 1/130. The reported utility 0.985 with standard deviation 0.000 is not compatible with this: any configuration yielding mean 0.985 requires varying per-seed counts (e.g., 26,26,26,25,25), which would give a nonzero standard deviation. Please clarify the exact denominator used for utility and the per-seed test sizes.
  3. [Section VII-D0e] The explanation for the single one-off LightGBM collapse attributes the failure to 'floating-point nondeterminism in multi-threaded histogram construction,' which is speculative; the retraction and stability re-run are appropriate, but the specific mechanism should be labeled as a hypothesis rather than a confirmed cause.
  4. [Section IV-G, Section VII-C] Predictions P1, P2, P4, and P7 are accounting identities that must hold on any correctly computed data, and the paper already labels them as implementation checks. To avoid the appearance of circular validation, the abstract should not present these identity checks as verification of substantive predictions; consider moving them to an appendix or explicitly scoping the 'verified predictions' claim to P3, P5, and P6.

Circularity Check

4 steps flagged · score 6.0 of 10

Several 'verified predictions' are definitional identities, and the ATT&CK instantiation is a relabeling of IDS categories; the conformal and out-of-sample checks remain genuinely independent.

  1. self definitional [Section IV-G (Testable predictions); Section VII-C (RQ2.5, item a)]
    "The framework yields pre-specified predictions checked against existing D2 data with no new inference runs... P1, P2, P4, and P7 check exact accounting identities (Propositions 2 and 1, Theorems 6, 3, and 7) that must hold by construction on any correctly computed data; we report them briefly, as an implementation check."

    P1, P2, P4, and P7 are direct restatements of definitions: the automation rate is F2^-(τ)-F1^-(τ) (Theorem 6), utility is A-R (Definition 2), C(τ)=R(τ)+S(τ) with S defined as the deferred residual (Theorem 7), and B=R+D is the M≡0 case of Theorem 3. On correctly computed data these identities cannot fail, so the reported 'zero violations' only checks arithmetic. Listing them as 'pre-specified predictions' and then verifying them makes the verification equivalent to the input definitions.

  2. self definitional [Section VII-C0b (P5); Theorem 4]
    "The identity R_phi(g)=R_fine(g)-M_phi(g) holds exactly at all 4 α levels (difference<10^-9 in every case), confirming Theorem 4 under a real, non-synthetic many-to-one map rather than only its trivial single-fiber boundary case."

    In Theorem 4, M_phi(g) is defined as the within-fiber confusion mass, i.e., exactly R_fine(g)-R_phi(g). The equality is therefore true by definition. The empirical content is only the estimated magnitude of the masking mass (2.1e-5), which is a measurement, not a confirmation of the identity. Presenting the exact match as 'confirming Theorem 4' treats a definitional identity as an empirical prediction.

2 more flagged steps
  1. other [Definition 4; Section VII-C0d (Table V discussion)]
    "κ̂α = 0.380 is almost exactly A(τ̂) = 0.380—the calibrated threshold is already the best any threshold on this split can do without exceeding the risk target, and this is risk-constrained incapacity, not a fixable misalignment."

    κ_α(f) is defined as sup{A(τ): τ∈[0,1], R(τ)≤α}, so saying the realized A(τ̂) is essentially equal to κ̂_α merely restates that the realized threshold attains the defining supremum on that split. The label 'risk-constrained incapacity' is the case κ_α<ρ by Definition 4, so the diagnostic's 'separation' of regimes is a classification by definition rather than an independent explanation. The genuinely non-circular recoverability evidence is the separate out-of-sample alternative-threshold check for CIC×Gemma-2.

  2. renaming known result [Section III-A and Section III-B (Table I)]
    "An IDS alert x∈X has a benchmark-derived, ATT&CK-aligned operational technique label y(x)∈T, obtained by mapping the dataset’s own attack-category annotation through φ... this is not an independently sourced ATT&CK ground-truth annotation... Since φ is a bijection, the classification problem over C_attack and the attribution problem over T are equivalent."

    The reported 'ATT&CK misattribution risk' is, by construction, the four-class IDS-category error relabeled through the bijection φ; no ATT&CK-specific technique information enters the risk computation. This makes the ATT&CK-aligned empirical claim a renaming of the benchmark's coarse category labels rather than an independently grounded ATT&CK evaluation. The paper acknowledges the proxy status, so this is primarily a validity limitation, but it is also a definitional reduction of the applied claim to the input labels.

full rationale

The formal decision-contract theory (Theorems 2–8) is self-contained: the error-conservation, capacity, and actionability results are proved from the definitions of R, A, U, Q, Γ, and κ, and the conformal risk bound is imported from the independent CRC literature. There is no load-bearing self-citation chain: the author cites no prior work of their own, and the references to 'an earlier version' are retractions within this manuscript, not external authority. The empirical core contains genuinely non-circular components: the 90.3% pass rate tests the standard conformal guarantee on held-out splits; the CIC×Gemma-2 recoverability claim is verified by an alternative threshold selected only on the calibration split and evaluated on a disjoint test split; and Proposition 5's Hoeffding lower bound is an independent finite-sample statement. However, several items labeled 'testable predictions' are actually definitional identities. P1 (automation rate equals the CDF-gap), P2 (utility equals A−R), P4 (C=R+S), and P7 (B=R+D under E_fine) cannot fail on correctly computed data, and P5 is the same identity as Theorem 4's definition of M_phi. The paper itself concedes these 'must hold by construction' and calls them implementation checks, which mitigates but does not remove the presentation of them as pre-specified predictions that are then verified. The κ_α regime separation is likewise definitional: κ_α is the supremum over risk-feasible thresholds, so concluding that A(τ̂) is near-optimal, or that κ_α<ρ means incapacity, applies the definition rather than testing a hypothesis. Finally, the ATT&CK instantiation reduces by construction to the four-class IDS category labels through the bijective φ, so the empirical 'ATT&CK misattribution risk' is a relabeling of the benchmark category error; the paper is transparent about this, but it limits the external claim. Overall, the central derivation is not forced by self-citation or by a fitted parameter renamed as a prediction, but a substantial subset of the reported verifications reduce to definitions, giving partial circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The theory itself is built from probability definitions and standard conformal/exchangeability assumptions. The load-bearing domain choices are the operational ATT&CK mapping, the assumption that LLM score rankings are meaningful, and the user-set floors for the certificate. No completely new physical or mathematical entities are introduced.

free parameters (3)
  • bijective mapping φ from IDS categories to ATT&CK techniques = Table I (4 hand-assigned mappings)
    Chosen by the author; semantic validity assumed and not externally verified. All empirical FAR statements in the ATT&CK instantiation depend on this mapping.
  • action-rate floor ρ = 0.5 (user-chosen)
    Representative deployment floor used throughout Section VII and VIII; the paper notes a stricter floor (0.9) would reclassify RT-IoT×Gemma-2 as structural incapacity. Pass counts depend on this choice.
  • Hoeffding confidence δ = 0.1 (user-chosen)
    Sets the action-rate lower bound A_δ in Proposition 5; the 16/18 certification count depends on this choice.
assumptions (5)
  • domain assumption Exchangeability of calibration and test draws (Theorem 1)
    Standard split-conformal requirement invoked for the FAR bound; violated under distribution shift, acknowledged in Section VIII-D(e).
  • domain assumption The operational label y(x) is the ATT&CK-aligned ground truth via the bijective map φ of Table I
    Paper states this is not independently sourced ATT&CK ground truth (Section III-A); if the mapping is semantically wrong, the empirical FAR is not ATT&CK misattribution risk.
  • domain assumption LLM softmax probabilities over attack classes induce a valid ranking over ATT&CK techniques through φ (Section V-B)
    The nonconformity score and singleton capacity rely on the score p_k ordering matching technique plausibility; paper checks rank correlation ρ=0.672 on average.
  • standard math Hoeffding bound applies because τ is fixed before the certification sample is drawn (Proposition 5)
    The i.i.d. Bernoulli argument requires no dependency between τ and the certification split.
  • standard math DKW inequality margin for two CDFs (Proposition 3)
    Used to certify κ from unlabeled readiness samples.
invented entities (1)
  • operational ATT&CK label mapped from IDS categories via φ
    purpose: Allows the paper to claim ATT&CK-aligned triage despite benchmark datasets lacking true ATT&CK labels
    The bijection is hand-constructed (Table I); no external validation against real ATT&CK technique annotations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Non-Degenerate Risk Certification for Automated Security Decisions: A Decision-Contract Theory with ATT\&CK-Aligned Triage as a Worked Instance." pith.science (2026). https://pith.science/paper/7TZS6HZK

@misc{pith2026260812444,
  author       = {Pith},
  title        = {Pith review of: Non-Degenerate Risk Certification for Automated Security Decisions: A Decision-Contract Theory with ATT\&CK-Aligned Triage as a Worked Instance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7TZS6HZK}},
  note         = {Machine review of arXiv:2608.12444}
}
read the original abstract

An unconditional risk bound on automated decisions can be satisfied without automating anything, since a selector that never acts drives the bound to zero. We show this is structural: any risk certificate is defined over a decision contract, the inputs a system acts on plus the semantic relation under which an output counts correct, and weakening either hides base-classifier error. We develop a decision-contract theory: an error-conservation law showing error is only reassigned among harmful automation, human deferral, and semantic masking; a label-free singleton capacity certifying structural incapacity, with a risk-feasible refinement separating recoverable threshold misalignment from risk-constrained incapacity; and a non-degenerate actionability certificate excluding all-abstain solutions by construction. We instantiate this on ATT\&CK-aligned alert triage for LLM-based intrusion detection, the setting that exposed the vacuity failure. Across 3 IDS datasets, 6 LLMs, and 4 error-rate thresholds, empirical false-attribution risk stays at or below target in 90.3% of configurations, with 83.4% mean correct automation. The capacity diagnostic explains every low-utility configuration; its refinement separates genuine misalignment from risk-constrained incapacity, confirmed by an exhibited alternative threshold; a training-stability re-run finds no confirmed structural-incapacity instance; and real fine-grained attack-subtype labels confirm the coarsening-transfer identity under a genuine many-to-one map, with small but non-zero masking mass.

Figures

Figures reproduced from arXiv: 2608.12444 by the authors.

Figure 1
Figure 1. CRC-based ATT&CK alert triage framework. The abstention layer [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. FAR–Utility tradeoff on CIC-IDS-2018 for three representative LLMs, [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 20 canonical work pages

  1. [1]

    99% false positives: A qualitative study of soc analysts’ perspectives on security alarms,

    B. A. Alahmadi, L. Axon, and I. Martinovic, “99% false positives: A qualitative study of soc analysts’ perspectives on security alarms,” inUSENIX Security Symposium, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:245130092

    work page 2022

  2. [2]

    Revolutionizing cyber threat detection with large language models: A privacy-preserving bert-based lightweight model for iot/iiot devices,

    M. A. Ferrag, M. Ndhlovu, N. Tihanyi, L. C. Cordeiro, M. Debbah, T. Lestable, and N. S. Thandi, “Revolutionizing cyber threat detection with large language models: A privacy-preserving bert-based lightweight model for iot/iiot devices,” 2024. [Online]. Available: https://arxiv.org/abs/2306.14263

    arXiv 2024

  3. [3]

    Conformal risk control,

    A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster, “Conformal risk control,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https: //openreview.net/forum?id=33XGfHLtZg

    2024

  4. [4]

    Automated discovery and mapping att&ck tactics and techniques for unstructured cyber threat intelligence,

    L. Li, C. Huang, and J. Chen, “Automated discovery and mapping att&ck tactics and techniques for unstructured cyber threat intelligence,” Comput. Secur., vol. 140, no. C, May 2024. [Online]. Available: https://doi.org/10.1016/j.cose.2024.103815

    arXiv 2024

  5. [5]

    V ovk, A

    V . V ovk, A. Gammerman, and G. Shafer,Algorithmic Learning in a Random World, 01 2005

    2005

  6. [6]

    Learn then test: Calibrating predictive algorithms to achieve risk control,

    A. N. Angelopoulos, S. Bates, E. J. Cand `es, M. I. Jordan, and L. Lei, “Learn then test: Calibrating predictive algorithms to achieve risk control,”The Annals of Applied Statistics, vol. 19, no. 2, pp. 1641 – 1662, 2025. [Online]. Available: https://doi.org/10.1214/24-AOAS1998

    doi:10.1214/24-aoas1998 2025

  7. [7]

    Fair conformal predictors for applications in medical imaging,

    C. Lu, A. Lemay, K. Chang, K. Hoebel, and J. Kalpathy-Cramer, “Fair conformal predictors for applications in medical imaging,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 12 008– 12 016, 06 2022

    work page 2022

  8. [8]

    Safe planning in dynamic environments using conformal prediction,

    L. Lindemann, M. Cleaveland, G. Shim, and G. Pappas, “Safe planning in dynamic environments using conformal prediction,”IEEE Robotics and Automation Letters, vol. 8, pp. 5116–5123, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:252992582

    work page 2022

Show all 35 references
  1. [9]

    Conformal language modeling,

    V . Quach, A. Fisch, T. Schuster, A. Yala, J. H. Sohn, T. Jaakkola, and R. Barzilay, “Conformal language modeling,”ArXiv, vol. abs/2306.10193, 2023. [Online]. Available: https://api.semanticscholar. org/CorpusID:259203582

    2023 arXiv

  2. [10]

    Transcend: detecting concept drift in malware clas- sification models,

    R. Jordaney, K. Sharad, S. K. Dash, Z. Wang, D. Papini, I. Nouretdinov, and L. Cavallaro, “Transcend: detecting concept drift in malware clas- sification models,” inProceedings of the 26th USENIX Conference on Security Symposium, ser. SEC’17. USA: USENIX Association, 2017, p. 625–642

    2017

  3. [11]

    Smith, N

    R. Smith, N. Japkowicz, M. Dondo, and P. Mason,Using Unsupervised Learning for Network Alert Correlation, 05 2008, vol. 5032, pp. 308– 319

    2008

  4. [12]

    Selective classification for deep neural networks,

    Y . Geifman and R. El-Yaniv, “Selective classification for deep neural networks,” inProceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17. Red Hook, NY , USA: Curran Associates Inc., 2017, p. 4885–4894

    2017

  5. [13]

    On the foundations of noise-free selective classification,

    R. El-Yaniv and Y . Wiener, “On the foundations of noise-free selective classification,”J. Mach. Learn. Res., vol. 11, p. 1605–1641, Aug. 2010

    2010

  6. [14]

    Selectivenet: A deep neural network with an integrated reject option,

    Y . Geifman and R. El-Yaniv, “Selectivenet: A deep neural network with an integrated reject option,” inInternational Conference on Machine Learning, 2019. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 16

    2019

  7. [15]

    Selective classification via one-sided prediction,

    A. Gangrade, A. Kag, and V . Saligrama, “Selective classification via one-sided prediction,” 2021. [Online]. Available: https://arxiv.org/abs/ 2010.07853

    2021 arXiv

  8. [16]

    Decision-making under selective labels: Optimal finite-domain policies and beyond,

    D. Wei, “Decision-making under selective labels: Optimal finite-domain policies and beyond,” inProceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18–24 Jul 2021, pp. ...

    2021

  9. [17]

    Entropy alone is insufficient for safe selective prediction in llms,

    E. Phillips, F. K. Gustafsson, S. Wu, A. Thakur, and D. A. Clifton, “Entropy alone is insufficient for safe selective prediction in llms,”

  10. [18]

    When can conformal risk control certify llm outputs? bounds, impossibility, and adaptation for structured generation,

    V . Kotte, “When can conformal risk control certify llm outputs? bounds, impossibility, and adaptation for structured generation,” 2026. [Online]. Available: https://arxiv.org/abs/2606.29054

    2026 arXiv

  11. [19]

    Conformal selective acting: Anytime- valid risk control for rlvr-trained llms,

    H. Khosravi and X. Huo, “Conformal selective acting: Anytime- valid risk control for rlvr-trained llms,”ArXiv, vol. abs/2605.20270,

    arXiv

  12. [20]

    Class-conditional conformal prediction with many classes,

    T. Ding, A. Angelopoulos, S. Bates, M. Jordan, and R. J. Tibshirani, “Class-conditional conformal prediction with many classes,” inAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Asso...

    2023

  13. [21]

    Available: https://api.semanticscholar.org/CorpusID: 288654956

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 288654956

  14. [22]

    Generating network intrusion detection dataset based on real and encrypted synthetic attack traffic,

    A. Ferriyan, A. H. Thamrin, K. Takeda, and J. Murai, “Generating network intrusion detection dataset based on real and encrypted synthetic attack traffic,”Applied Sciences, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:239692344

    2021

  15. [23]

    Toward generating a new intrusion detection dataset and intrusion traffic characterization,

    I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward generating a new intrusion detection dataset and intrusion traffic characterization,” in International Conference on Information Systems Security and Privacy, 2018

    2018

  16. [24]

    Gemma 2: Improving open language models at a practical size,

    G. T. M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ram’e, J. Ferret, P. Liu, P. D. Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar, C. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Sta ´nczyk, S. Girgin, N. Momch...

    2024 arXiv

  17. [25]

    Quantized autoencoder (qae) intrusion detection system for anomaly detection in resource-constrained iot devices using rt-iot2022 dataset,

    B. S. Sharmila and R. Nagapadma, “Quantized autoencoder (qae) intrusion detection system for anomaly detection in resource-constrained iot devices using rt-iot2022 dataset,”Cybersecurity, vol. 6, pp. 1– 15, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID: 261516162

    2023

  18. [26]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7b,” 2023. [Online]. Available: ht...

    2023 arXiv

  19. [27]

    The llama 3 herd of models,

    A. Dubeyet al., “The llama 3 herd of models,” 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:271571434

    2024

  20. [28]

    Xgboost: A scalable tree boosting system,

    T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” inProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD ’16. New York, NY , USA: Association for Computing Machinery, 2016, p. 785–794

    2016

  21. [29]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. L...

  22. [30]

    Adaptive conformal inference under dis- tribution shift,

    I. Gibbs and E. J. Cand `es, “Adaptive conformal inference under dis- tribution shift,” inProceedings of the 35th International Conference on Neural Information Processing Systems, ser. NIPS ’21. Red Hook, NY , USA: Curran Associates Inc., 2021

    2021

  23. [31]

    Non- exchangeable conformal risk control,

    A. Farinhas, C. Zerva, D. T. Ulmer, and A. Martins, “Non- exchangeable conformal risk control,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=j511LaqEeP

    2024

  24. [32]

    Lightgbm: a highly efficient gradient boosting decision tree,

    G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y . Liu, “Lightgbm: a highly efficient gradient boosting decision tree,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17. Red Hook, NY , USA: Curran Ass...

    2017

  25. [35]

    Adaptive conformal predictions for time series,

    M. Zaffran, A. Dieuleveut, O. F’eron, Y . Goude, and J. Josse, “Adaptive conformal predictions for time series,” inInternational Conference on Machine Learning, 2022. [Online]. Available: https: //api.semanticscholar.org/CorpusID:246863519

    2022

  26. [2025]

    Available: https://arxiv.org/abs/2505.09388

    [Online]. Available: https://arxiv.org/abs/2505.09388

    arXiv

  27. [2026]

    Available: https://arxiv.org/abs/2603.21172

    [Online]. Available: https://arxiv.org/abs/2603.21172

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.