Online Conformal Anomaly Detection with Prediction-Powered Data Acquisition

Amirmohammad Farzaneh; Osvaldo Simeone

REVIEW 3 major objections 5 minor 1 cited by

Online Conformal Anomaly Detection with Prediction-Powered Data Acquisition

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read C-PP-COAD guarantees time-averaged false discovery rate control while deciding, per context, whether to acquire real calibration data or rely on synthetic data.

desk verdict A clean, valid integration of active p-values with conformal calibration for online FDR-controlled anomaly detection, but the headline data-efficiency claim is only as strong as the assumption that real calibration data appears on demand. read the letter →

arxiv 2505.01783 v2 pith:FES2IIOO submitted 2025-05-03 cs.LG cs.ITmath.ITstat.ML

Amirmohammad Farzaneh , Osvaldo Simeone This is my paper · ORCID

classification cs.LGcs.ITmath.ITstat.ML

keywords onlineanomalydetectionconformalp-valuesfalsediscoveryratecontrolsyntheticcalibrationdataactivecontext-awareLORDalgorithmacquisition

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that online anomaly detection with a formal false-discovery-rate guarantee does not need a steady stream of fresh real calibration data. It proposes C-PP-COAD, which wraps any pre-trained anomaly score and decides at each time step, using context and a synthetic-data 'proxy' p-value, whether to spend resources acquiring a real calibration batch. The core assertion, Proposition 1, is that the resulting test keeps the decaying-memory time-averaged FDR at or below the target level $\alpha$ for every time step, for any context sequence and any score function. Experiments on thyroid-disease and O-RAN conflict data show lower data acquisition than conformal baselines at comparable power.

What carries the argument

The load-bearing object is the active p-value statistic $Z_t = (1-U_t) Q_t + U_t (1-\gamma(C_t))^{-1} P_t$, where $Q_t$ is the conformal p-value computed against synthetic calibration data, $P_t$ is the conformal p-value against freshly acquired real data, and $U_t$ is drawn Bernoulli with probability $1 - \gamma(C_t) Q_t$. This is the prediction-powered step: synthetic data provides a cheap proxy that sets the probability of querying the real p-value, and the $1/(1-\gamma)$ rescaling of $P_t$ is what keeps $Z_t$ superuniform. With $Z_t$ in hand, the LORD procedure, which sets thresholds $\alpha_t$ from past discovery times, turns a sequence of valid p-values into online FDR control.

What would settle it

Run the algorithm with a simulated context in which the real calibration batch, whenever U_t=1, is drawn from an anomalous distribution instead of the nominal P(X|C_t), while test points under the null are drawn from P(X|C_t); the empirical sFDR over many runs should rise above alpha at later times, showing the guarantee depends on uncontaminated real calibration.

Watch

Extended reading notes

Core claim

For a fixed score function $s(X|C)$, and assuming the observations $X_t$ are i.i.d. given their contexts $C_t$, C-PP-COAD controls the smoothed decaying-memory FDR (sFDR) at level $\alpha$ at all times. The discovery is that this guarantee survives even though the anomaly statistic $Z_t$ is sometimes computed only from synthetic calibration data. The mechanism is the active p-value: $Z_t$ coincides with a synthetic-data conformal p-value $Q_t$ when no real data is acquired, and with a rescaled real-data conformal p-value when real data is acquired, with the acquisition probability itself a function of $Q_t$. Because $Z_t$ is a valid p-value under the null hypothesis, the LORD thresholds continue to control the sFDR.

Load-bearing premise

At every time step the system chooses to acquire real calibration data, it must actually obtain a fresh batch of n i.i.d. genuine nominal samples from P(X|C_t); if that batch is unavailable, contaminated, or drawn from the wrong context, the real conformal p-value P_t is no longer superuniform, Z_t is not a valid p-value, and Proposition 1's guarantee collapses.

Editorial extensions

If this is right

Deployments with a good digital twin can lower their cumulative data-acquisition rate well below COAD while keeping the sFDR bound; the experiments demonstrate this on both healthcare and telecom data.
The guarantee is not tied to the score function: supervised, unsupervised, and semi-supervised scores all work, so C-PP-COAD can be added on top of an existing detector.
Context-aware acquisition is more powerful per unit of real data than the context-agnostic PP-COAD, and C-COAD's higher power comes at the cost of full real-data usage.
The extension to MCAR missing data means the same FDR bound carries over when features are imputed by a fixed pretrained function.
LORD is not the only admissible threshold rule; SAFFRON and ADDIS can be swapped in, so the framework inherits other online FDR procedures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If synthetic data is perfectly calibrated for a context, $\gamma(C)$ can be set near 1, making real acquisition rare; the guarantee then holds almost for free, although Proposition 1 itself does not quantify the acquisition savings.
The method's operating point is a data-cost-versus-power tradeoff left to the user via $\gamma(C)$: the proposed per-context estimator of synthetic-data quality, based on the superuniformity gap $D(C)$, is heuristic, so an online, regret-bounded rule for tuning $\gamma(C)$ is a natural testable extension.
A failure mode the paper does not treat: real data acquisition may be most expensive exactly when $Q_t$ is small, so a practical deployment should detect when the acquisition indicator is forced to 0 and can no longer guarantee FDR.
The same active-p-value construction should transfer to e-values or set-valued conformal prediction, giving analogous data-efficient guarantees for other online decision problems.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is worth a serious look. What is genuinely new is the packaging: active p-values from Xu et al. meet conformal calibration and LORD-style online FDR control, with a context-dependent acquisition rule that actually changes how often real data is requested. The main theorem is short and leans on two external results, but I checked the active p-value step: the masking inequality holds for any proxy in [0,1], so the machinery is sound. The D(C) heuristic in Sec. IV-A is a reasonable, clearly-labeled empirical device, and the missing-data extension is a nice bonus. Experiments on two datasets show the expected pattern: sFDR is controlled, power beats plain COAD, and CDAR drops. That is real evidence the framework works under the conditions it assumes.

The soft spots are proportional. The biggest one is exactly what the stress-test note flags: Algorithm 1 step 8 assumes fresh i.i.d. real calibration samples are available whenever U_t=1. That is stated in Sec. II-A, but the experiments never exercise a failed acquisition, because the data split guarantees real calibration data at every time step. So the paper demonstrates reduced query frequency, not reduced worst-case dependency. This is a limitation, not an inconsistency, and it should be discussed explicitly—a fallback like setting Z_t=1 on failed acquisition would preserve FDR control at the cost of power, and the authors should at least mention it. Second, Proposition 1 depends on an unpublished, non-peer-reviewed preprint [37]. The construction checks out, but the paper should either prove the key step in an appendix or clearly flag the dependency. Third, manuscript hygiene: there is a stray biography and reference block from an unrelated IEEE paper appended after the reference list, plus a few mis-references (e.g., [19] is not the original LORD reference). That is sloppy but easy to fix. Finally, the figures lack error bars; with 100 repeats, there is no excuse for omitting them.

Who is this for? Researchers in conformal prediction, online FDR, or anomaly detection with scarce labels. It is not a new statistical principle, but it is a useful integration with a plausible path to real deployment. I would not desk-reject it. Send to review, with the expectation that the authors address the oracle assumption, add error bars, and clean up the manuscript.

Referee Report

3 major / 5 minor

Summary. The paper proposes C-PP-COAD, an online anomaly detection method that uses synthetic calibration data to compute a proxy conformal p-value Q_t, and then randomizes between this proxy and a real-data conformal p-value P_t through an active-p-value construction, with LORD thresholds controlling a decaying-memory smoothed FDR (sFDR). The main theoretical claim (Proposition 1, Sec. III-D) is that, for any pre-trained score function and any context sequence, if the sequence X_t is i.i.d. given the contexts, the sFDR is bounded by the target level alpha at every time step. The paper also proposes a context-dependent heuristic for the real-data query probability, extends the method to missing features, and reports experiments on the Thyroid and O-RAN conflict datasets showing reduced real-data acquisition relative to COAD while maintaining sFDR control.

Significance. If the guarantee holds, the paper contributes a clean algorithmic wrapper that imports active p-values into online conformal anomaly detection, allowing arbitrary score functions to be used with FDR control while reducing the frequency of real-data queries. The paper is careful to mark PO-COAD and C-PO-COAD as lacking guarantees, and it explicitly labels the gamma(C) selection rule in Sec. IV-A as a heuristic. The experimental work is extensive, covering three score families, context-aware and context-agnostic baselines, a missing-data extension, and an ablation on the tuning parameter lambda. The main caveats are that the central guarantee is conditional on an oracle that supplies fresh nominal data whenever a query is made, and that the active-p-value formula needs to be stated unambiguously; these issues are fixable but should be addressed before publication.

major comments (3)

[Sec. III-B, Eq. (14)] Equation (14) is not self-consistent with the stated range of gamma(C_t). The text allows gamma(C_t) in (0,1], but the displayed statistic Z_t = (1-U_t)Q_t + U_t (1-gamma(C_t))^{-1} P_t is undefined at gamma(C_t)=1. Moreover, since p_real(Q_t,C_t)=1-gamma(C_t)Q_t, the correction factor in the queried branch would naturally be (1-gamma(C_t)Q_t)^{-1}, not (1-gamma(C_t))^{-1}; the sentence after (14) saying that Z_t 'coincides with' P_t when U_t=1 is also inaccurate under either version because of the scaling. Please correct the formula, the admissible range of gamma, and the explanatory sentence. This is load-bearing because the validity of Z_t is the starting point of Proposition 1.
[Sec. II-A, Algorithm 1 step 8, Sec. V-A] Proposition 1 and the experiments presuppose that whenever U_t=1, a fresh batch D_t of n i.i.d. samples from the nominal distribution P(X|C_t) is available. The experiments enforce this by construction through the fixed data split in Fig. 3, so the claimed 'significantly reduces dependency on real calibration data' is established for query frequency under an oracle, not for scenarios in which acquisition fails or returns contaminated data. In those scenarios P_t in (15) is not superuniform and Proposition 1 does not apply. Please qualify the claims and, ideally, define and analyze a conservative fallback (e.g., taking Z_t=1 when acquisition fails) so that the method remains valid when the oracle is imperfect.
[Sec. III-D, proof of Proposition 1] The proof of Proposition 1 is a citation to [37] and [19] without stating the exact active-p-value lemma being imported. Since the proxy Q_t in (11) and the real p-value P_t in (15) are both functions of the same test point X_t, it is not immediate to a reader which conditions of [37] are satisfied, for example whether P_t needs to be valid unconditionally or conditionally on Q_t and how dependence through X_t is handled. Please state the lemma and verify these conditions explicitly; as written, the central guarantee is not checkable from the manuscript alone.

minor comments (5)

[Sec. II-B, Eq. (7)] The notation E_{C_t} is confusing because C_t is used both for the context sequence and for the expectation set; please define the expectation as being over the nominal data distribution given the contexts.
[Sec. V-A, Fig. 3] COAD and C-COAD use twice as many real calibration samples per time step as the prediction-powered methods because their data split contains no digital-twin training subset; this should be stated more prominently so that the power comparisons are not interpreted as equal-sample-size comparisons.
[Sec. IV-A, Eqs. (17)-(19)] The heuristic D(C) requires a held-out set of real inlier points V_C per context; this is itself a real-data cost and should be acknowledged in the data-efficiency discussion.
[Sec. V-C3, Eq. (20)] The statement that increasing lambda corresponds to more frequent real data queries holds only when D(C)>0; for D(C)<=0, gamma(C)=1 regardless of lambda. Please state this qualification.
[Sec. II-B, Eqs. (5)-(7)] The smoothing parameter eta is used in (7) but introduced only after R_t in (6); please move the definition of eta before its first use.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: Proposition 1 composes external active-p-value and LORD guarantees, and the paper's tuned/heuristic components are explicitly non-load-bearing.

full rationale

The central sFDR guarantee (Proposition 1, Sec. III-D) is not obtained by fitting or by self-citation. Its proof cites two external, independent results: active p-value validity from [37] (Eq. 14 as a valid p-value) and LORD's decaying-memory FDR control from [19] (Eq. 16). The paper's own tuning parameters gamma(C) and lambda enter only the real-data acquisition probability (12) and the heuristic D(C) in Sec. IV-A; the text explicitly labels D(C)-based gamma(C) selection a heuristic, and the formal validity statement in Sec. III-D does not depend on these choices. The benchmarks include externally defined COAD from [19], and the experiments evaluate the proposed method against it, so the empirical claim is not a re-description of the input. The main caveat is substantive but not circular: Proposition 1 is conditional on the oracle assumption in Sec. II-A that when U_t=1 a fresh i.i.d. real calibration batch D_t from P(X|C_t) is available (Algorithm 1, step 8), and the experiments ensure this by pre-partitioning real calibration data; failure of this oracle is a robustness gap, not an equation-level reduction of the output to the input.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The framework's guarantee is assembled from external theorems (active p-values [37], LORD [19]) plus domain assumptions about data availability; the paper's own additions (gamma heuristic, imputation) are clearly labeled and do not affect the formal guarantee.

free parameters (5)

gamma(C) = Set per context via gamma(C)=exp(-lambda max(0,D(C))), Eq. (20)
User-specified propensity to acquire real data. The sFDR guarantee holds for any gamma in (0,1]; gamma only tunes the power/data tradeoff.
lambda = 5 in most experiments, swept 1 to 10 in ablation
Tuning parameter in Eq. (20) controlling how quickly gamma drops with the superuniformity deviation metric D(C).
delta = 0.95 or 0.99
Memory decay in sFDR definitions (5)-(7), set by the experimenter.
eta = Not specified
Smoothing parameter in the denominators of (7) and (9); small positive constant.
alpha = 0.1 or 0.2
Target sFDR level, user-specified.

assumptions (6)

domain assumption Fresh real calibration data D_t is available on demand and is i.i.d. from the nominal distribution P(X|C_t).
Stated in Sec. II-A; used in Algorithm 1 step 8 and in conformal p-value (15). If this fails, Prop. 1 has no force.
standard math The conformal p-value P_t in (15) is superuniform under the null H_t.
Exchangeability of X_t with D_t; standard conformal prediction result [24], used in Prop. 1.
standard math The active p-value construction of Xu et al. [37] applies to (12)-(14).
Prop. 1 proof item (i) relies entirely on this external preprint; the paper does not re-derive it.
standard math The LORD algorithm controls the decaying-memory sFDR as shown in Rebjock et al. [19].
Prop. 1 proof item (ii); external theorem.
domain assumption The sequence {X_t} is i.i.d. conditioned on the context sequence {C_t}.
Explicit assumption in Prop. 1; needed for independence of p-values across time for LORD.
domain assumption Missing data are missing completely at random (MCAR) for the extension of Sec. IV-B.
The paper assumes masks M_t independent of X_t; if missingness is informative, imputation may break exchangeability.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Online Conformal Anomaly Detection with Prediction-Powered Data Acquisition." pith.science (2026). https://pith.science/paper/FES2IIOO

@misc{pith2026250501783,
  author       = {Pith},
  title        = {Pith review of: Online Conformal Anomaly Detection with Prediction-Powered Data Acquisition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FES2IIOO}},
  note         = {Machine review of arXiv:2505.01783}
}

read the original abstract

Online anomaly detection is essential in fields such as cybersecurity, healthcare, industrial monitoring, and telecommunications, where promptly identifying deviations from expected behavior can avert critical failures or security breaches. While numerous anomaly scoring methods based on supervised or unsupervised learning have been proposed, the only existing approach capable of providing assumption-free guarantees on the false discovery rate (FDR) rely on a continuous stream of real-world calibration data. To address this limitation, we introduce context-aware prediction-powered conformal online anomaly detection (C-PP-COAD), a novel principled framework that strategically leverages synthetic calibration data to mitigate data scarcity, while adaptively integrating real data based on contextual information. C-PP-COAD wraps around any existing anomaly detection method, leveraging any given anomaly score to construct active conformal p-value statistics. These statistics support online testing with formal FDR control, maintaining rigorous and reliable anomaly detection performance over time. Experiments conducted on both synthetic and real-world datasets, including thyroid dysfunction detection, O-RAN conflict detection, 5G network intrusion detection, and O-RAN UE throughput degradation detection, demonstrate that C-PP-COAD significantly reduces dependency on real calibration data without compromising guaranteed FDR control.

Figures

Figures reproduced from arXiv: 2505.01783 by the authors.

**Figure 2.** An overview of C-PP-COAD B. Problem Definition Using any pre-trained anomaly score s(X|C), we are interested in designing an online anomaly detection framework that (i) ensures that the fraction of false anomalies is no larger than a desired target α, while making a best effort at (ii) maximizing the fraction of true detected anomalies and (iii) reducing the reliance on real calibration data. A generic anomaly dete… view at source ↗

**Figure 3.** Data splits for schemes considered in this work. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Performance of supervised (random forest), unsuper [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Performance of COAD, C-PP-COAD, and context [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 8.** Figure 8: Average CDAR and power achieved by C-PP-COAD [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Impact of Benign Connectivity Variations on Intrusion Detection for Encrypted OPC UA Traffic in Industrial Private 5G Networks
cs.CR 2026-07 conditional novelty 4.0 of 10

Benign private-5G connectivity variations raise false positives in ML IDS for encrypted OPC UA traffic and often coincide with control-plane activity.

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.