{"id":"443f42d9-14bd-42de-b020-2babb674af44","arxiv_id":"2608.05199","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A post-hoc audit can certify trajectory-level conformal coverage for frozen multi-stage LLM pipelines, and the naive pairwise-correlation extension to three or more stages is a lower bound, not an upper bound.","lead":"This paper shows how to certify that a multi-step LLM security agent covers the true answer end to end, using only per-step guarantees and an audit sample, even when the agent cannot be retrained. The result gives operators a way to measure when a pipeline's trajectory-level guarantee is truly tighter than the naive Bonferroni bound, and it exposes a label-design artifact that can fake strong correlation between stages.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section VI-D's audit size was chosen after a pilot look, so the reported 13.7% certificate gain may not be a valid fixed-n confidence statement.","rationale":"The paper's central theorem, Theorem 7, is mathematically sound under its stated fixed-n i.i.d. audit assumption, and the spanning-tree certificate with data-dependent tree selection is correctly derived. The reader's weakest-assumption concern about deployment distribution shift is real and acknowledged by the paper in L3 and Section VI-G, but it is an external validity limitation rather than a flaw in the certificate itself. The more load-bearing internal concern is the adaptive sample-size selection in Section VI-D: the authors observed the small-scale pilot results, then increased the audit size to 3,500/1,376 and computed standard fixed-n confidence intervals at the new size. A fixed-n Clopper-Pearson or Hoeffding bound is valid for any fixed n, but if n is chosen based on a pilot that is part of or correlated with the final sample, the final intervals do not automatically retain their coverage conditional on the selection rule. The paper itself flags time-uniform confidence sequences as the appropriate tool for data-dependent sample sizes, yet does not use them in the experiment that supports the headline improvement. The proposed fresh-audit check would settle whether the 13.7% gain is robust or an artifact of the scaling rule. This does not overturn the theoretical contribution, so the reader's CONDITIONAL verdict remains appropriate, but the empirical claim should be re-validated under a pre-registered or independent audit before full acceptance.","tokens_in":29927,"tokens_out":24416,"duration_ms":225165,"concrete_test":"Run a fresh, independently drawn audit at the same sample sizes (3,500 for CIC-IDS-2018/DoS and 1,376 for RT-IoT2022/Probe) that was not used in any way to choose the audit size, and recompute the seed-wise Clopper-Pearson U_union for each configuration. If the average U_union rises above 0.086 or fewer than 7 of 12 configurations satisfy U_union < 0.10, the reported gain is selection-dependent and the fixed-n certificate claim in Section VI-D is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The empirical headline rests on Clopper-Pearson certificates computed at the boosted audit size (3,500/1,376) that was adopted only after the authors first ran a smaller audit (397/109), observed U_union = 0.124 (worse than Bonferroni) and zero joint failures in some configurations, and then 'scaled' the audit sample. Theorems 7-10 provide valid simultaneous confidence bounds for a fixed audit size n; they do not cover an n chosen after inspecting audit data unless the final sample is independent of the pilot and was pre-specified. The paper does not state that the boosted pool is disjoint from the pilot pool or that the scale-up was pre-registered, and it does not use a time-uniform confidence sequence, although Section II-C explicitly notes that such sequences are the tool for data-dependent audit sizes. If the final sample overlaps the pilot or the decision to scale was based on the pilot estimates, the seed-wise Clopper-Pearson intervals lose their claimed 1-delta coverage conditional on the selection event. The abstract's '13.7% tighter than Bonferroni' and the 7-of-12 count of configurations with U_union < 0.10 may therefore be an artifact of adaptive sample-size selection rather than a valid certificate.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes trajectory-level risk certification for modular LLM-based security pipelines whose stages are individually calibrated but cannot be jointly recalibrated. The main theory is centered on a post-hoc certificate (Theorem 7) that combines simultaneous upper bounds on per-step miscoverage probabilities with lower bounds on pairwise failure overlaps via a spanning-tree bound, valid even when the tree is selected from the data. The paper shows that the natural K=2 inclusion-exclusion identity does not extend to K>=3 as an upper bound, proves a non-identifiability result for pairwise information once K>=3, derives fixed-n sample-complexity bounds for certifying positive dependence and for beating the nominal Bonferroni target, and proves that certain coarse-to-fine label mappings mechanically induce near-perfect measured correlation. The empirical study on a two-stage intrusion-detection pipeline across six LLMs and two datasets reports a 13.7% tighter direct union audit than Bonferroni at a boosted audit scale, a smaller 0.6% average gain for the modular decomposed certificate, evidence that residual coupling reflects shared sample difficulty rather than shared model representations, and a cross-dataset shift experiment in which single-step coverage collapses to zero despite high raw accuracy. The paper is unusually candid about its own earlier deterministic label-mapping artifact and about the limitations of pairwise information for K>=3.","tokens_in":30142,"tokens_out":7632,"duration_ms":78219,"significance":"If the results hold, the paper makes a useful conceptual contribution to conformal and risk-control composition: it separates marginal from realized-chain risk, distinguishes structural from certifiable dependence gains, supplies a valid K>=2 spanning-tree certificate with data-dependent tree selection, and gives rate-matching upper and lower bounds on the audit size needed to certify positive dependence. The main-text proofs that are included (notably Theorem 3, Theorem 6, Theorem 7, Theorem 8, Theorem 10, Theorem 13, Theorem 14, Theorem 16) are clear and internally sound. The paper also deserves credit for explicitly retracting its earlier near-1 correlation finding as a labeling artifact and for designing a same-model/cross-model/permuted protocol that separates shared-difficulty from representation-sharing accounts. The central theoretical framework is defensible, but the headline empirical certificate is compromised by an adaptive audit-size selection that the paper does not validly justify, and several substantive proofs are deferred to a supplementary file not present in this arXiv version.","major_comments":[{"comment":"The headline numerical claim, that the direct union audit U_union becomes 13.7% tighter than Bonferroni (0.086 vs 0.100), is computed at an audit sample size (3,500/1,376) that was chosen after inspecting a smaller pilot audit (397/109) in which the union bound was worse than Bonferroni and some configurations had zero joint failures. Theorems 7 and 10 are fixed-n statements: they provide valid simultaneous confidence bounds for a pre-specified n, but they do not cover an n selected after seeing pilot data unless the final audit pool is independent of the pilot and the scaling rule was fixed in advance. The text does not state that the boosted pool is disjoint from the pilot pool, does not report a pre-registered or otherwise pre-specified scaling rule, and does not use a time-uniform confidence sequence even though Section II-C explicitly identifies those sequences as the appropriate tool for data-dependent audit sizes. As written, the 13.7% gain and the 7-of-12 count of configurations below 0.10 are not valid 1-delta certificates; they should be re-derived with an always-valid bound, an independent pre-specified split, or clearly re-labeled as an exploratory observation rather than a certified gain.","section":"Section VI-D, paragraphs on the boosted audit scale"},{"comment":"Theorem 5, Theorem 9, Theorem 12, Proposition 2, and Theorem 17 are stated in the main text, but their proofs are explicitly deferred to a supplementary material that is not included in this arXiv version. These results carry load-bearing content: the second-order degeneracy threshold, the finite-sample certifiability gap, the sample-complexity lower bound for beating the nominal target, the coupling provenance decomposition, and the dependence duality. The reader cannot currently verify these claims from the manuscript, and the claim of matching upper and information-theoretic lower bounds depends in part on these missing derivations. The revised version should include the full proofs in the paper or in an available supplement, or clearly state where the supplement can be obtained.","section":"Section IV-C and Section IV-F, deferred proofs"},{"comment":"The plug-in estimate b_BIE in Eq. (12) replaces the true per-step miscoverage probabilities p1 and p2 in Theorem 3 with the nominal budgets alpha1 and alpha2, even though the theorem's identity is written in terms of p1 and p2. The paper justifies this informally by saying the per-step conformal sets are approximately saturated, but the approximation is not quantified and the plug-in is then used in comparative tables. Please make explicit that Eq. (12) is only valid as an approximation when the achieved miscoverage rates are close to their budgets, or replace alpha_k by estimates of p_k and report the effect on the residual analysis.","section":"Equation (12) and Remark 1"}],"minor_comments":[{"comment":"The phrase 'chainstraffic classification' near the start of Section I should be corrected to 'chain traffic classification'.","section":"Introduction"},{"comment":"The caption contains '1OF12 CONFIGURATIONS' without a space; it should read '1 of 12 configurations'.","section":"Table XI caption"},{"comment":"The notation for the pilot audit size (pool n=497/139, test-audit 397/109) is not connected to the Ncal/Ntest values in Table I; please define the calibration sizes used in the pilot and state how the pilot pool relates to the final pool, since this is exactly the adaptive-selection issue in the major comment above.","section":"Section VI-D, first paragraphs"},{"comment":"The sentence reporting mean |C1(x)|=0.964 'of 5 categories' should say 'mean cardinality of the prediction set across samples' to avoid the impression that set sizes are fractional; the subsequent explanation about singleton rates clarifies this, but the wording could be tightened.","section":"Section VI-B"},{"comment":"Remark 3 says U_union < U1+U2-L12 holds in every one of the 12 configurations, but the paper does not give a table of the decomposed versus direct values per configuration; please add the per-configuration comparison or state where it can be found.","section":"Section VI-D, Remark 3 reference"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is intellectually honest and the fixed-n theory appears sound, but the abstract-level claim that the direct audit becomes 13.7% tighter than Bonferroni depends on a data-dependent audit-size choice that is not validly handled. I would prefer a revised version that either supplies a valid always-valid or pre-specified-selection treatment of the audit size, or demotes the gain to an exploratory finding. The deferred supplementary proofs should also be made available before publication. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The theory is real. Theorem 7's post-hoc certificate—simultaneous per-step upper bounds, a pairwise overlap lower bound, and data-dependent tree selection—is valid because Hunter's spanning-tree bound holds for every tree at once. Theorem 10 vs. Theorem 12 (Θ(1/q) to detect positive gain, Θ(1/γ²) to beat the nominal target) is a clean distinction that matters in practice. Theorems 13–14 show sharply why pairwise information cannot identify trajectory risk for K≥3, and Theorem 16's label-nesting result is genuine: the authors caught their own near-1 correlation as a label artifact, fixed the Step-2 task, and reported the corrected numbers in the same paper. That is good scientific conduct.\n\nThe main soft spot is exactly the stress-test note. The 13.7% headline is computed at an audit size that was chosen after looking at a smaller pilot (397/109 test samples, U_union=0.124, worse than Bonferroni). Theorems 7–10 are fixed-n statements; they do not cover an n selected after inspecting pilot data unless the final sample is disjoint and pre-specified. The paper never says that, and it does not use the time-uniform confidence sequences it cites in Section II-C. So the abstract's '13.7% tighter' should not be treated as a valid 95% certificate at the reported n. The direction of the result is consistent with the theory, but the specific number is conditional on a selection event nobody has controlled. This is a load-bearing flaw in the empirical headline, not in the theory.\n\nSmaller issues: several proofs (Theorems 5, 9, 12, Proposition 2, Theorem 17) live in a supplementary that isn't in this arXiv version; the review copy needs them. No code or data is shipped. The cross-dataset shift experiment is stark—100% miscoverage with 78% accuracy—but only Step 1 is tested because Step 2's label spaces are dataset-specific. The citation pattern is honest: Bonferroni, Hunter, Šidák, PASC, and the conformal literature are all credited, and PASC is correctly positioned as the joint-recalibration alternative.\n\nWho it's for: anyone composing calibrated stages into a pipeline guarantee, and LLM security agent builders specifically. The theory deserves serious referee time; the empirical section needs a rewritten audit protocol or a retreat from the 13.7% claim. Send it to review, but require the authors to either pre-specify a disjoint fixed audit, use confidence sequences, or explicitly reframe the headline as pilot-suggested. I would cite the theory even while flagging the empirical claim.","headline":"Solid post-hoc certification theory and honest self-correction, but the headline 13.7% empirical gain is built on an audit size chosen after a pilot look and is not a valid fixed-n certificate as reported.","tokens_in":30682,"tokens_out":3503,"would_cite":true,"duration_ms":30429,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For frozen multi-stage LLM security pipelines, trajectory-level miscoverage can be certified post-hoc from per-stage audits plus pairwise failure-overlap bounds.","keywords":["trajectory coverage","conformal prediction","multi-stage LLM agents","intrusion detection","post-hoc certification","sample complexity","spanning-tree bound","label nesting artifact"],"falsifier":"Simulate a realized two-stage chain with known per-step risks $p_1,p_2$ and overlap $q_{12}$, draw audit samples at the size prescribed by the paper’s threshold, and check whether the certificate $\\min\\{1, U_1+U_2-L_{12}\\}$ holds with empirical frequency at least $1-\\delta_p-\\delta_q$; if the frequency falls below the claimed confidence, the post-hoc certificate is invalid.","tokens_in":29689,"feed_emoji":"🛡️","tokens_out":12684,"duration_ms":104515,"temperature":0.7,"pith_summary":"The paper establishes a way to turn per-step safety certificates of a frozen, multi-stage LLM security pipeline into a guarantee on the whole trajectory, using only audit data and no joint recalibration. The central claim is that trajectory miscoverage is bounded above by the sum of certified per-step risk bounds minus a certified pairwise failure-overlap gain, provided the gain is taken over a spanning tree rather than by naive inclusion–exclusion; the naive extension to three or more stages is a lower bound, not an upper bound. A second central claim is that certifying any positive dependence gain requires an audit sample of order $1/q_{12}$, where $q_{12}$ is the joint-failure rate, and the certificate can flip from worse than the Bonferroni bound to tighter than it purely as the audit reaches that scale. The paper also shows that a coarse-to-fine label design can mechanically manufacture near-perfect measured inter-stage correlation, and that correcting this artifact reveals task-dependent coupling attributable mostly to shared sample difficulty.","feed_headline":"Post-hoc audit certifies whole-pipeline failure risk","feed_subtitle":"Per-step conformal guarantees plus pairwise overlap bounds yield a valid trajectory certificate once the audit is large enough.","key_machinery":"The spanning-tree pairwise upper bound is the load-bearing object: for any spanning tree $T$ on the $K$ stages, trajectory miscoverage is at most $\\sum_k p_k - \\sum_{(i,j)\\in T} q_{ij}$, where $p_k$ is the per-step marginal risk and $q_{ij}$ is the pairwise failure-overlap probability. This bound is what makes the certificate valid for every $K$, replacing the natural pairwise inclusion–exclusion truncation, which is a lower bound once $K>2$. The certificate assembles the bound from simultaneous concentration or exact binomial confidence bounds on the $p_k$ and $q_{ij}$, and the maximizing tree may be selected after seeing the audit data. A companion diagnostic result shows that if a fine label is a deterministic function of a coarse label, the two stages’ failure events are nested and their measured correlation is forced near one by construction.","core_discovery":"Condition on the realized trained-and-calibrated chain $\\theta$. If an audit of $n$ i.i.d. trajectories yields simultaneous upper bounds $U_k$ on each stage’s miscoverage probability and lower bounds $L_{ij}$ on each pair’s failure overlap, then with probability at least $1-\\delta_p-\\delta_q$ the true trajectory miscoverage satisfies $R(\\theta) \\le \\min\\{1, \\sum_k U_k - \\max_T \\sum_{(i,j)\\in T} L_{ij}\\}$, where $T$ runs over spanning trees on the stages. The data-dependent choice of the best tree after seeing the audit does not break validity, because the underlying spanning-tree bound holds for every tree simultaneously. On a two-stage intrusion-detection pipeline over six open LLMs and two datasets, a direct one-sided binomial audit of the union event is 13.7% tighter than the Bonferroni bound once the audit reaches the sample size the theory requires, while the modular certificate that uses only per-stage certificates and one pairwise overlap recovers a 0.6% average gain; at an under-sized audit the same certificate is worse than Bonferroni, matching the derived threshold.","pith_inferences":["The paper’s own comparison implies that when full joint audit vectors are available at $K=2$, operators should deploy the direct one-sided binomial bound on the union event rather than the modular decomposition; the modular certificate is the useful object only when only pairwise summaries exist or when $K>2$.","The $\\Theta(1/q_{12})$ audit threshold suggests that for very rare joint failures, dependence-aware certification is impractical; an adaptive or time-uniform sequential audit that stops as soon as the overlap lower bound turns positive would make the procedure deployable.","The label-nesting theorem likely generalizes to any hierarchical taxonomy: whenever later stages refine earlier labels deterministically or near-deterministically, reported inter-stage correlation is inflated; this gives a testable checklist for other multi-stage evaluation pipelines.","The near-one empty-set rate under cross-dataset shift suggests an online, label-free drift alarm: monitoring the rate at which conformal prediction sets are empty or singleton could detect exchangeability failure before any ground truth arrives."],"forward_implications":["Operators of frozen pipelines can certify trajectory-level risk from audit data alone, without retraining or joint recalibration, whenever per-stage certificates and pairwise overlap lower bounds are obtainable.","An audit too small for the joint-failure rate makes dependence-aware certificates worse than the Bonferroni bound; the paper makes the required sample size explicit as $\\Theta(1/q_{12})$ for detecting any positive gain and $\\Theta(1/\\gamma^2)$ for beating the nominal target.","For pipelines with three or more stages, pairwise information alone cannot determine trajectory risk; the spanning-tree bound remains a valid upper bound, but tighter two-sided statements require higher-order overlap data.","Positive inter-stage error correlation is an asset for the union-type trajectory failure criterion; the same dependence would be a liability for an intersection-type redundant-system criterion.","Evaluations of multi-stage LLM agents must check that stage labels are not deterministic refinements of earlier stage labels, or measured inter-stage dependence can be a labeling artifact rather than a property of the model."],"supporting_citations":[{"why":"It defines conformal prediction, the per-step guarantee that the trajectory certificate composes.","marker":"[1]"},{"why":"It provides the split conformal procedure that produces the per-step prediction sets used in the experiments.","marker":"[2]"},{"why":"It generalizes conformal guarantees to monotone losses, one of the decision-rule families the composition framework subsumes.","marker":"[4]"},{"why":"It supplies the pipeline-aware joint-recalibration baseline whose deployment regime the post-hoc certificate explicitly does not require.","marker":"[12]"},{"why":"It supplies the union upper bound over a spanning tree that the paper turns into a finite-sample certificate.","marker":"[25]"},{"why":"It supplies the association concept used in the dependence-duality result.","marker":"[26]"},{"why":"It provides the first intrusion-detection dataset and its attack-variant labels used in the two-stage evaluation.","marker":"[28]"},{"why":"It provides the second intrusion-detection dataset and its attack-variant labels used in the two-stage evaluation.","marker":"[29]"}],"fun_headline_variants":["Spanning-tree bound certifies trajectory risk","Audit-size threshold flips Bonferroni vs direct","13.7% tighter than Bonferroni once audit is sized","Pairwise overlap bounds beat Bonferroni at scale","Post-hoc certificate for multi-stage pipeline risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the audit sample is drawn independently from the exact deployment distribution of the frozen chain, so the audit’s empirical failure rates stay representative of real traffic.","fun_headline_variants_meta":{"raw":{"variants":["Spanning-tree bound certifies trajectory risk","Audit-size threshold flips Bonferroni vs direct","13.7% tighter than Bonferroni once audit is sized","Pairwise overlap bounds beat Bonferroni at scale","Post-hoc certificate for multi-stage pipeline risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000429,"raw_usage":{"total_tokens":2277,"prompt_tokens":1115,"completion_tokens":1162,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":731,"completion_tokens_details":{"reasoning_tokens":1086}},"tokens_in":731,"tokens_out":1162,"duration_ms":8657,"temperature":1.0,"reasoning_tokens":1086,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:41:25.559158+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate a realized two-stage chain with known per-step risks $p_1,p_2$ and overlap $q_{12}$, draw audit samples at the size prescribed by the paper’s threshold, and check whether the certificate $\\min\\{1, U_1+U_2-L_{12}\\}$ holds with empirical frequency at least $1-\\delta_p-\\delta_q$; if the frequency falls below the claimed confidence, the post-hoc certificate is invalid.","supporting_citations":[{"cited_title":"V ovk, A","cited_arxiv_id":null,"evidence_quote":"It defines conformal prediction, the per-step guarantee that the trajectory certificate composes."},{"cited_title":"PASC: Pipeline-Aware Conformal Prediction with Joint Coverage Guarantees for Multi-Stage NLP and LLM Pipelines","cited_arxiv_id":"2605.18812","evidence_quote":"It supplies the pipeline-aware joint-recalibration baseline whose deployment regime the post-hoc certificate explicitly does not require."},{"cited_title":"An upper bound for the probability of a union,","cited_arxiv_id":null,"evidence_quote":"It supplies the union upper bound over a spanning tree that the paper turns into a finite-sample certificate."},{"cited_title":"Association of random variables, with applications,","cited_arxiv_id":null,"evidence_quote":"It supplies the association concept used in the dependence-duality result."},{"cited_title":"Toward generating a new intrusion detection dataset and intrusion traffic characterization,","cited_arxiv_id":null,"evidence_quote":"It provides the first intrusion-detection dataset and its attack-variant labels used in the two-stage evaluation."},{"cited_title":"Quantized autoencoder (qae) intrusion detection system for anomaly detection in resource-constrained iot devices using rt-iot2022 dataset,","cited_arxiv_id":null,"evidence_quote":"It provides the second intrusion-detection dataset and its attack-variant labels used in the two-stage evaluation."}],"review_version":2}