{"id":"575bb111-7b9e-45e6-acd0-e9d3b9eeab0f","arxiv_id":"2608.12444","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper formalizes why unconditional risk bounds on automated decisions are vacuous and proposes an actionability certificate that jointly bounds error and floors automation, demonstrated on LLM security triage.","lead":"A security automation system can 'satisfy' a risk guarantee by automating nothing, so this paper builds a theory that forces certified systems to also do useful work. It tests the framework on LLM-based alert triage, and most configurations keep false-attribution risk low while automating most alerts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ATT&CK grounding of the worked instance is unvalidated: the hand-built bijection φ (Table I) is the sole bridge from IDS category labels to ATT&CK techniques, and without independent validation the reported FAR is four-class category error, not ATT&CK misattribution risk.","rationale":"The reader's weakest assumption identifies exactly the point on which the empirical contribution hinges: the unvalidated hand-built bijection φ from IDS attack categories to ATT&CK techniques. The paper is transparent about the operational status of this mapping, but transparency does not remove the need for external validation. The formal theory is independent of φ and is correct, so the verdict remains CONDITIONAL rather than REJECT: the abstract and title claim an 'ATT&CK-aligned triage as a worked instance,' and that specific claim is unsupported until the mapping is checked. A positive result on the proposed independent annotation test would strengthen the empirical section and could support movement toward ACCEPT (subject to the other minor issues the reader noted, such as the lack of released code/data and the one-sided confidence wording). A negative result would require substantial reframing of the applied claims. The reader's verdict already captures this uncertainty, so no adjustment is needed.","tokens_in":26739,"tokens_out":20169,"duration_ms":172246,"concrete_test":"Independently annotate a random sample (e.g., 200 alerts per dataset) with ATT&CK techniques using a validated process—two security analysts with ATT&CK expertise working independently, or a documented CTI-mapping pipeline—blinded to the paper's φ. Compute the agreement rate between φ(c(x)) and the independent technique label. If agreement falls below a pre-specified threshold (e.g., 90%), or if any IDS category is found to map to multiple distinct techniques, then the paper's FAR numbers are category-error rates rather than ATT&CK misattribution risks, and the worked instance must be revised either by using finer-grained validated labels or by reframing the claims as category-level rather than ATT&CK-level.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's formal theory—the error-conservation law, the vacuity of unconditional risk bounds, and the (α,ρ)-actionability certificate—is sound and self-contained. The load-bearing weakness is the empirical instantiation: the 90.3% pass rate, capacity diagnostics, and the claimed ATT&CK-aligned triage all rest on the deterministic bijection φ in Table I that maps each IDS attack category to exactly one ATT&CK technique (DoS→T1498, CredentialAccess→T1110, Exploitation→T1190, Probe→T1046). Section III-A explicitly states this is 'not an independently sourced ATT&CK ground-truth annotation' and that y(x) is used only in an 'operational sense.' This is a serious limitation: the conformal risk bound of Theorem 1 certifies risk with respect to these proxy labels. If φ is semantically wrong—for example, if 'Exploitation' alerts in CIC-IDS-2018 correspond to techniques other than T1190, or if any IDS category is heterogeneous and actually spans multiple ATT&CK techniques—then the reported false-attribution risk is a four-class category error, not an ATT&CK misattribution rate, and the triage conclusions do not transfer to the real ATT&CK taxonomy. The paper's own limitation section acknowledges the granularity gap (4 techniques vs. 600+ in full ATT&CK) but does not validate the 4-technique mapping itself. Without such validation, the central applied claim is unsupported, even though the formal theory stands.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that an unconditional risk bound on automated decisions is vacuous, since a selector that never acts trivially satisfies it. It develops a decision-contract theory in which a certificate is defined over a pair (selector, correctness relation), proves an error-conservation law (Theorem 3) showing that base-classifier error is only reassigned among harmful automation, deferral, and semantic masking, and derives a coarsening-transfer identity (Theorem 4) with a reverse impossibility result (Theorem 5). It then introduces a label-free singleton capacity κ(f) (Theorem 6) as a necessary condition for a target action rate, a risk-feasible refinement κα(f) (Definition 4) that separates threshold misalignment from risk-constrained incapacity, and an (α,ρ)-actionability certificate (Theorem 8) with a finite-sample Hoeffding lower bound on the action rate (Proposition 5). The framework is instantiated on ATT&CK-aligned alert triage for LLM-based intrusion detection across 3 IDS datasets, 6 LLMs, and 4 error thresholds, reporting that empirical false-attribution risk stays at or below target in 90.3% of configurations, with 83.4% mean correct automation. The capacity diagnostics are used to classify the three lowest-utility configurations, and a coarsening-transfer identity is verified under a genuine many-to-one map using fine-grained attack-subtype labels.","tokens_in":27044,"tokens_out":10955,"duration_ms":94042,"significance":"The formal theory is a valuable generalization of selective classification and conformal risk control: it makes the correctness relation part of the certified object and provides an exact accounting identity connecting harmful automation, deferral, and semantic masking. The finite-sample capacity certificate (Proposition 3) and the action-rate lower-bound certificate (Proposition 5) are practical tools, and Theorem 8 gives a clean bridge from unconditional risk to correct-automation and conditional-error bounds. The paper is unusually transparent: it explicitly acknowledges the proxy-label limitation of the ATT&CK mapping, retracts an initially reported structural-incapacity instance after a 20-seed training-stability check, and confirms the one recoverable misalignment case with an out-of-sample threshold evaluation. If the ATT&CK mapping were independently validated, the empirical instantiation would be a strong demonstration; as it stands, the theoretical contribution is sound, self-contained, and likely to be influential regardless of the empirical caveat.","major_comments":[{"comment":"The central applied claim is that the paper instantiates the theory on ATT&CK-aligned alert triage, but the only bridge from IDS category labels to ATT&CK techniques is the hand-built bijection φ in Table I (DoS→T1498, CredentialAccess→T1110, Exploitation→T1190, Probe→T1046). Section III-A explicitly says this is 'not an independently sourced ATT&CK ground-truth annotation' and that y(x) is used in an 'operational sense.' Consequently, the conformal risk bound of Theorem 1, the empirical 90.3% pass rate, and all triage conclusions certify risk with respect to these proxy labels. If φ is semantically wrong — for example, if 'Exploitation' alerts are heterogeneous and span multiple ATT&CK techniques, or if any IDS category maps to a technique other than the one assumed — the reported false-attribution risk is a four-class category error, not an ATT&CK misattribution rate, and the triage conclusions do not transfer to the real ATT&CK taxonomy. The authors should either validate φ against independently sourced ATT&CK annotations on a subset of alerts, or reframe the title, abstract, and empirical claims in terms of coarse IDS-category attribution, explicitly dropping the ATT&CK-alignment claim for the empirical part while retaining it as an operational simplifying assumption.","section":"Section III-A, Section III-B, Table I"},{"comment":"The actionability certificate is presented as a single non-degenerate certificate, but it actually pairs two guarantees of different character: the risk side is the standard marginal conformal statement of Theorem 1 (a probability over the joint draw of calibration set and test point, not a confidence interval on the realized population risk), while the action-rate side is a (1−δ) finite-sample lower bound for the fixed, already-realized threshold τ̂. The bounds U ≥ Aδ − α and Q ≤ α/Aδ in Theorem 8 hold on the probability-(1−δ) event {A(τ̂) ≥ Aδ}, and the risk inequality R ≤ α is not conditional on that event. The paper acknowledges that it does not combine the two into a single joint confidence level, but the abstract and contribution list present this as one 'non-degenerate actionability certificate.' To make the certificate's probabilistic content precise for practitioners, the authors should add an explicit union-bound or PAC-style joint statement (e.g., P(R ≤ α and A ≥ Aδ) ≥ 1 − α − δ under a high-probability risk bound), or clearly state in the abstract and in Section IV-F that the certificate provides two separate guarantees that are not simultaneously certified at a single stated level.","section":"Section IV-F, Proposition 5, Theorem 8"}],"minor_comments":[{"comment":"The column labeled 'Test Samples' in Table II appears to denote total attack samples, since the certification split for CIC-IDS-2018 in Section VII-C0e is m = 1,288 = 1,488 − 200; please rename the column to 'Total Attack Samples' or clarify the relationship between the tabulated counts, the 200-sample calibration set, and the certification/test split.","section":"Table II, Section VII-C0e"},{"comment":"For HIKARI-2021, the stated test set size after removing 200 calibration samples is 26, so per-seed utility must be k/26 and the 5-seed mean must be a multiple of 1/130. The reported utility 0.985 with standard deviation 0.000 is not compatible with this: any configuration yielding mean 0.985 requires varying per-seed counts (e.g., 26,26,26,25,25), which would give a nonzero standard deviation. Please clarify the exact denominator used for utility and the per-seed test sizes.","section":"Table IV, HIKARI-2021 rows"},{"comment":"The explanation for the single one-off LightGBM collapse attributes the failure to 'floating-point nondeterminism in multi-threaded histogram construction,' which is speculative; the retraction and stability re-run are appropriate, but the specific mechanism should be labeled as a hypothesis rather than a confirmed cause.","section":"Section VII-D0e"},{"comment":"Predictions P1, P2, P4, and P7 are accounting identities that must hold on any correctly computed data, and the paper already labels them as implementation checks. To avoid the appearance of circular validation, the abstract should not present these identity checks as verification of substantive predictions; consider moving them to an appendix or explicitly scoping the 'verified predictions' claim to P3, P5, and P6.","section":"Section IV-G, Section VII-C"}],"recommendation":"major_revision","confidential_remarks":"The theoretical core — the error-conservation law, the vacuity analysis, the capacity diagnostics, and the actionability certificate — is sound and the presentation is unusually candid about limitations. The main obstacle to acceptance is that the empirical instantiation's title-level claim of ATT&CK alignment is not supported by the unvalidated hand-built mapping φ. I recommend requiring either independent validation of φ on a subset of alerts or an explicit reframing of all empirical claims as coarse-category attribution. The joint-confidence issue in Proposition 5 also needs a precise statement. With these changes the paper would be a strong contribution to the selective-classification and conformal-risk-control literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The formal argument is sound: an unconditional risk bound like FAR ≤ α can be certified by a system that never acts, and the paper makes a good case that this is structural, not a calibration bug. The decision-contract framing—selector plus correctness relation—is the right generalization, and the error-conservation identity B(h) = R(C) + D(C) + M(C) is a clean way to show that error only moves between harmful automation, deferral, and semantic masking. Second, the ATT&CK instantiation rests on a hand-built bijection φ from four IDS categories to four ATT&CK techniques, explicitly not an independently sourced ground-truth annotation. So the 90.3% pass rate and the capacity diagnostics are real, but they certify four-class category error, not validated ATT&CK misattribution.\n\nWhat's good beyond that. The singleton capacity κ(f) and the risk-feasible refinement κ_α(f) form a genuinely useful diagnostic; the paper shows κ alone misreads two of three low-utility cases as recoverable when they're risk-constrained. The finite-sample action-rate bound is standard Hoeffding, but the paper is careful that it has a different probabilistic character from the conformal risk bound. The empirical work is solid for what it is: 72 configurations, ML baselines, surrogate-slack decomposition. The transparency is a real asset: the paper retracts a structural-incapacity claim after a retraining check, labels P1/P2/P4/P7 as identities that can't fail, and admits the earlier B2 baseline used test labels.\n\nSoft spots, in proportion. The φ mapping is the load-bearing weakness; the theory survives, but the applied claim of ATT&CK-aligned triage does not without external validation of the mapping. The paper acknowledges this, and the conditional verdict in the attached report is fair. Minor: the 'predictions' that are accounting identities are circular in letter but the paper explicitly frames them as implementation checks, so it's more a presentation weakness than a substance problem. The two-sided certificate doesn't come with a single joint confidence; the paper says so openly and leaves that for future work—a limitation, not a hidden flaw. No code or data released, which is a disappointment.\n\nWho this is for: researchers in conformal prediction and selective classification, and security-ML practitioners who want a principled abstention layer. The theory is the value. I'd send it to peer review—the formal content deserves referee time and the honest reporting helps. The main thing I'd require is either independent validation of the ATT&CK mapping or a reframing of the empirical claims as category-level.","headline":"A sound and honest formal treatment of why unconditional risk bounds are vacuous without an action-rate floor, with a useful capacity diagnostic—but the ATT&CK label mapping behind the empirical instantiation is unvalidated, so the applied claims are category-level, not true ATT&CK triage.","tokens_in":27607,"tokens_out":7311,"would_cite":true,"duration_ms":61912,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A risk bound satisfied by doing nothing certifies nothing; the paper's decision-contract theory fixes it by adding an action-rate floor.","keywords":["decision contract","risk certification","conformal risk control","selective prediction","false automated attribution","singleton capacity","LLM intrusion detection","alert triage"],"falsifier":"Have independent security analysts annotate a held-out sample of the same IDS alerts directly with ATT&CK technique identifiers, then compare with the paper's $\\phi$ mapping: material disagreement would show that the reported false-attribution risk is a four-class category error rather than ATT&CK misattribution risk. A complementary check on any dataset carrying both fine and coarse labels would verify the accounting by computing $R_{\\mathrm{fine}}(g)$, $R_\\phi(g)$, and $M_\\phi(g)$ and testing whether $R_\\phi=R_{\\mathrm{fine}}-M_\\phi$ holds.","tokens_in":26470,"feed_emoji":"🛡️","tokens_out":14706,"duration_ms":103109,"temperature":0.7,"pith_summary":"An unconditional risk bound — 'the probability of a wrong automated output is at most $\\alpha$' — is not a diagnostic of a good predictor: a system that never acts, or that reports at a coarser label resolution, satisfies it without automating anything useful. The paper formalizes this as a decision-contract theory, in which any risk certificate is defined over a selector (what the system acts on) and a correctness relation (what counts as correct), and weakening either hides base-classifier error. The central accounting law is the error-conservation identity $B(h)=R(C)+D(C)+M(C)$: a base classifier's error is only reassigned among harmful automation, human deferral, and semantic masking, never eliminated. The practical claim is that a non-degenerate certificate must jointly enforce a risk bound $R(C)\\le \\alpha$ and an action-rate floor $A(C)\\ge \\rho$, which the paper demonstrates on LLM-based ATT&CK-aligned alert triage, where 90.3% of configurations meet the risk target and the capacity diagnostic explains every low-utility configuration.","feed_headline":"Safe automation needs a risk cap and a floor on its action rate","feed_subtitle":"A conformal risk bound can be satisfied by a system that never acts — the paper adds an action-rate floor.","key_machinery":"The load-bearing object is the decision contract $C=(g,E)$: a selector $g$ that decides for each input whether the system acts, and a reflexive acceptance relation $E$ that decides when an automated output is counted as correct. Three derived constructs carry the argument: the error-conservation identity $B(h)=R(C)+D(C)+M(C)$ (Theorem 3), which shows that abstention and semantic coarsening are the only two ways to hide base-classifier error from a risk bound; the singleton capacity $\\kappa(f)=\\sup_\\tau [F_2^-(\\tau)-F_1^-(\\tau)]$ (Theorem 6), a label-free upper bound on the action rate attainable by any global threshold, together with its labeled refinement $\\kappa_\\alpha(f)$ (Definition 4) that distinguishes recoverable threshold misalignment from risk-constrained incapacity; and the $(\\alpha,\\rho)$-actionability certificate (Theorem 8 with Proposition 5), which pairs the conformal risk guarantee with a finite-sample Hoeffding lower bound on the deployed threshold's action rate.","core_discovery":"The paper's central claim is that an unconditional risk bound such as $\\mathrm{FAR}\\le\\alpha$ is structural, not diagnostic: it certifies a decision contract $C=(g,E)$ — the set of inputs the selector $g$ acts on, plus the semantic relation $E$ under which an output counts correct — rather than the underlying predictor. Two mechanisms that are unrelated to predictive competence can drive the measured risk to zero, namely abstaining on more inputs and reporting at a coarser label resolution, and the paper shows these are the only two. The argument is carried by the error-conservation law $B(h)=R(C)+D(C)+M(C)$ (Theorem 3), which accounts for every base-classifier error as harmful automation, human deferral, or semantic masking; by the exact fine-to-coarse risk-transfer identity $R_\\phi(g)=R_{\\mathrm{fine}}(g)-M_\\phi(g)$ (Theorem 4), under which coarsening reduces measured risk by exactly the within-fiber confusion mass $M_\\phi(g)$, with a reverse-transfer impossibility result (Theorem 5); and by the $(\\alpha,\\rho)$-actionability certificate (Definition 5, Theorem 8), which converts a risk bound into both a correct-automation lower bound and a conditional-error upper bound while excluding all-abstain solutions by construction. On the empirical side, across 3 IDS datasets, 6 LLMs, and 4 error-rate thresholds, the label-free singleton capacity $\\kappa(f)$ rules out structural incapacity for every configuration, while the risk-feasible refinement $\\kappa_\\alpha(f)$ separates two risk-constrained-incapacity cases from one genuinely recoverable threshold misalignment.","pith_inferences":["Beyond IDS triage, any risk-controlled automation layer — clinical decision support, LLM tool use, content moderation — faces the same all-abstain vacuity, so requiring an action-rate floor should be the default wherever an unconditional risk bound gates actions.","The coarsening-transfer identity suggests a practical audit: whenever a system is evaluated at a coarser label resolution than the labels it was trained on, measuring $M_\\phi(g)$ directly would reveal how much fine-grained error is being hidden — a quantity that is small in this benchmark set but not guaranteed to stay small in richer taxonomies such as the full ATT&CK matrix.","Proposition 4's counterexample implies that score-variance or accuracy-based deployability heuristics should be replaced by capacity-style diagnostics computed from top-two probability gaps; this is an inference about neighbouring practice, not a claim the paper tests empirically.","An editorial reading of the retracted LightGBM episode: structural-incapacity verdicts about a stochastic base learner should survive a multi-seed training-stability check before a system is redesigned or abandoned."],"forward_implications":["A deployment that certifies only an unconditional risk bound is potentially vacuous: an all-abstain system or a coarser label report satisfies it while automating nothing, so practitioners should treat FAR-only certification as insufficient.","With an $(\\alpha,\\rho)$-actionability certificate, a deployer obtains a correct-automation lower bound $U\\ge \\rho-\\alpha$ and a conditional-error upper bound $Q\\le \\alpha/\\rho$ at a stated confidence on the action-rate side.","The label-free capacity check $\\kappa(f)$ can be run before any labeled calibration: if $\\kappa(f)<\\rho$, no single global threshold can reach the automation floor, and the remedy is a better base classifier, not more calibration data.","When automation falls short despite adequate $\\kappa(f)$, comparing $\\kappa_\\alpha(f)$ with the realized action rate $A(\\hat{\\tau})$ decides whether recalibration can close the gap (threshold misalignment) or whether the risk target itself is the binding constraint (risk-constrained incapacity).","Coarsening a label space transfers risk rather than removing it, by exactly the within-fiber confusion mass $M_\\phi(g)$; technique-level guarantees should not be promoted to tactic-level guarantees without reporting this masked mass."],"supporting_citations":[{"why":"Supplies the conformal risk control guarantee that forms the risk side of the certificate (Theorem 1).","marker":"[3]"},{"why":"Foundational conformal prediction framework underlying the finite-sample coverage machinery.","marker":"[5]"},{"why":"Selective classification with a reject option, establishing the risk-coverage trade-off that motivates the action-rate floor.","marker":"[12]"},{"why":"Foundations of noise-free selective classification used as the theoretical basis for the reject-option setting.","marker":"[13]"},{"why":"Class-conditional conformal prediction; its masking phenomenon is the analogue of the within-fiber confusion mass $M_\\phi(g)$.","marker":"[20]"},{"why":"Documents LLM-based network intrusion detection, the base classifiers the triage instantiation wraps with the certificate.","marker":"[2]"},{"why":"Quantifies SOC analyst alert load, motivating the triage automation setting.","marker":"[1]"}],"fun_headline_variants":["Risk certificates need an action floor, not just a bound","Why risk bounds fail: abstain and coarsen, then any bound holds","Safe automation: cap risk AND require action, or it's vacuous","Error conservation law shows how risk claims hide errors","Non-degenerate certification: no all-abstain shortcuts in risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole empirical instantiation rests on a hand-built bijection from four IDS attack categories to four ATT&CK techniques that is not independently sourced; if that mapping is semantically invalid, the reported risk numbers are four-class category error rates, not ATT&CK misattribution rates, and the triage conclusions do not transfer to the real ATT&CK taxonomy.","fun_headline_variants_meta":{"raw":{"variants":["Risk certificates need an action floor, not just a bound","Why risk bounds fail: abstain and coarsen, then any bound holds","Safe automation: cap risk AND require action, or it's vacuous","Error conservation law shows how risk claims hide errors","Non-degenerate certification: no all-abstain shortcuts in risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1821,"prompt_tokens":1184,"completion_tokens":637,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":800,"completion_tokens_details":{"reasoning_tokens":548}},"tokens_in":800,"tokens_out":637,"duration_ms":5382,"temperature":1.0,"reasoning_tokens":548,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:11:16.203088+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent security analysts annotate a held-out sample of the same IDS alerts directly with ATT&CK technique identifiers, then compare with the paper's $\\phi$ mapping: material disagreement would show that the reported false-attribution risk is a four-class category error rather than ATT&CK misattribution risk. A complementary check on any dataset carrying both fine and coarse labels would verify the accounting by computing $R_{\\mathrm{fine}}(g)$, $R_\\phi(g)$, and $M_\\phi(g)$ and testing whether $R_\\phi=R_{\\mathrm{fine}}-M_\\phi$ holds.","supporting_citations":[{"cited_title":"Selective classification for deep neural networks,","cited_arxiv_id":null,"evidence_quote":"Selective classification with a reject option, establishing the risk-coverage trade-off that motivates the action-rate floor."},{"cited_title":"On the foundations of noise-free selective classification,","cited_arxiv_id":null,"evidence_quote":"Foundations of noise-free selective classification used as the theoretical basis for the reject-option setting."},{"cited_title":"Class-conditional conformal prediction with many classes,","cited_arxiv_id":null,"evidence_quote":"Class-conditional conformal prediction; its masking phenomenon is the analogue of the within-fiber confusion mass $M_\\phi(g)$."},{"cited_title":"99% false positives: A qualitative study of soc analysts’ perspectives on security alarms,","cited_arxiv_id":null,"evidence_quote":"Quantifies SOC analyst alert load, motivating the triage automation setting."}],"review_version":1}