{"id":"70636917-3df6-4dad-b220-8cc1a772f4a4","arxiv_id":"2505.01783","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A context-aware framework that combines synthetic calibration data, active p-values, and online FDR control to reduce real-data usage in anomaly detection while preserving sFDR guarantees.","lead":"This paper introduces C-PP-COAD, a framework that uses synthetic data to calibrate online anomaly detection while adaptively deciding when to buy real data. It aims to cut the cost of real-world calibration data while keeping a statistical guarantee on false alarms.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 1's FDR guarantee holds only under an oracle that supplies fresh i.i.d. nominal calibration samples whenever U_t=1; the experiments assume this oracle, so the paper's data-scarcity claim is not tested without it.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing condition: fresh real calibration data from P(X|C_t) must be available whenever U_t=1. I agree with this. The active p-value construction is not itself fragile: one can verify directly that for any proxy Q_t in [0,1] and any valid p-value P_t, E[γQ_t 1(Q_t≤u)] + E[(1-γQ_t) 1(P_t≤(1-γ)u)] ≤ u, so the reliance on the unreviewed preprint [37] is not a correctness risk. The LORD thresholds in (16) inherit the COAD result [19] provided the Z_t are valid p-values, and the fresh-data construction makes them independent across time. The remaining soft spot is the oracle assumption for D_t. The paper is explicit about it, and the theorem is valid under that assumption, so this is not a reason to reject. However, the abstract and conclusion overstate the practical contribution by omitting this caveat, and the experiments cannot distinguish between 'uses less real data' and 'works when real data is hard to get.' The reader's CONDITIONAL verdict already captures the needed revisions: add error bars, discuss availability, and clean up the stray IEEE biography section. My stress-test does not change that verdict, so I recommend UNCHANGED. I credit the paper for the direct verifiability of the active p-value step and for clearly stating the distributional assumptions in Sec. II-A.","tokens_in":19909,"tokens_out":18761,"duration_ms":195362,"concrete_test":"Modify Algorithm 1 so that when U_t=1 but real calibration data is unavailable, one of two fallbacks is used: (a) the paper's literal algorithm, which cannot compute Z_t in (14), or (b) a conservative fallback Z_t=1. Run the Thyroid experiment (Sec. V-B) with acquisition failure probabilities p = 0, 0.1, 0.5 on U_t=1, and record empirical sFDR (7) and power (9) over 100 splits with α=0.1. If fallback (b) keeps sFDR ≤ α, the theorem can be robustified; if fDR exceeds α under the literal algorithm or under a naive Q_t substitution, the practical claim requires an explicit availability guarantee.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central guarantee (Proposition 1, Sec. III-D) is conditional on Algorithm 1 step 8: whenever U_t=1, the system can collect a fresh batch D_t of n i.i.d. samples from the nominal distribution P(X|C_t). This assumption is stated in Sec. II-A but is the practical bottleneck. If real calibration data is unavailable, contaminated, or drawn from a different context, the conformal p-value P_t in (15) is not superuniform, the active p-value Z_t in (14) cannot be formed as written, and Proposition 1 does not apply. The paper's experiments never exercise this failure mode: the data splits in Sec. V-A guarantee that a pre-partitioned real calibration set exists for every time step when U_t=1, so the oracle is always available. The claim that C-PP-COAD 'significantly reduces dependency on real calibration data without compromising guaranteed FDR control' is therefore only established in a regime where real data is always obtainable on demand; the reduction is in the frequency of queries, not in the worst-case requirement. This is a substantive limitation, though not an internal inconsistency: a conservative fallback that sets Z_t=1 on failed acquisition would preserve FDR control at the cost of power, but the paper does not define or analyze such a fallback.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes C-PP-COAD, an online anomaly detection method that uses synthetic calibration data to compute a proxy conformal p-value Q_t, and then randomizes between this proxy and a real-data conformal p-value P_t through an active-p-value construction, with LORD thresholds controlling a decaying-memory smoothed FDR (sFDR). The main theoretical claim (Proposition 1, Sec. III-D) is that, for any pre-trained score function and any context sequence, if the sequence X_t is i.i.d. given the contexts, the sFDR is bounded by the target level alpha at every time step. The paper also proposes a context-dependent heuristic for the real-data query probability, extends the method to missing features, and reports experiments on the Thyroid and O-RAN conflict datasets showing reduced real-data acquisition relative to COAD while maintaining sFDR control.","tokens_in":20075,"tokens_out":24984,"duration_ms":267656,"significance":"If the guarantee holds, the paper contributes a clean algorithmic wrapper that imports active p-values into online conformal anomaly detection, allowing arbitrary score functions to be used with FDR control while reducing the frequency of real-data queries. The paper is careful to mark PO-COAD and C-PO-COAD as lacking guarantees, and it explicitly labels the gamma(C) selection rule in Sec. IV-A as a heuristic. The experimental work is extensive, covering three score families, context-aware and context-agnostic baselines, a missing-data extension, and an ablation on the tuning parameter lambda. The main caveats are that the central guarantee is conditional on an oracle that supplies fresh nominal data whenever a query is made, and that the active-p-value formula needs to be stated unambiguously; these issues are fixable but should be addressed before publication.","major_comments":[{"comment":"Equation (14) is not self-consistent with the stated range of gamma(C_t). The text allows gamma(C_t) in (0,1], but the displayed statistic Z_t = (1-U_t)Q_t + U_t (1-gamma(C_t))^{-1} P_t is undefined at gamma(C_t)=1. Moreover, since p_real(Q_t,C_t)=1-gamma(C_t)Q_t, the correction factor in the queried branch would naturally be (1-gamma(C_t)Q_t)^{-1}, not (1-gamma(C_t))^{-1}; the sentence after (14) saying that Z_t 'coincides with' P_t when U_t=1 is also inaccurate under either version because of the scaling. Please correct the formula, the admissible range of gamma, and the explanatory sentence. This is load-bearing because the validity of Z_t is the starting point of Proposition 1.","section":"Sec. III-B, Eq. (14)"},{"comment":"Proposition 1 and the experiments presuppose that whenever U_t=1, a fresh batch D_t of n i.i.d. samples from the nominal distribution P(X|C_t) is available. The experiments enforce this by construction through the fixed data split in Fig. 3, so the claimed 'significantly reduces dependency on real calibration data' is established for query frequency under an oracle, not for scenarios in which acquisition fails or returns contaminated data. In those scenarios P_t in (15) is not superuniform and Proposition 1 does not apply. Please qualify the claims and, ideally, define and analyze a conservative fallback (e.g., taking Z_t=1 when acquisition fails) so that the method remains valid when the oracle is imperfect.","section":"Sec. II-A, Algorithm 1 step 8, Sec. V-A"},{"comment":"The proof of Proposition 1 is a citation to [37] and [19] without stating the exact active-p-value lemma being imported. Since the proxy Q_t in (11) and the real p-value P_t in (15) are both functions of the same test point X_t, it is not immediate to a reader which conditions of [37] are satisfied, for example whether P_t needs to be valid unconditionally or conditionally on Q_t and how dependence through X_t is handled. Please state the lemma and verify these conditions explicitly; as written, the central guarantee is not checkable from the manuscript alone.","section":"Sec. III-D, proof of Proposition 1"}],"minor_comments":[{"comment":"The notation E_{C_t} is confusing because C_t is used both for the context sequence and for the expectation set; please define the expectation as being over the nominal data distribution given the contexts.","section":"Sec. II-B, Eq. (7)"},{"comment":"COAD and C-COAD use twice as many real calibration samples per time step as the prediction-powered methods because their data split contains no digital-twin training subset; this should be stated more prominently so that the power comparisons are not interpreted as equal-sample-size comparisons.","section":"Sec. V-A, Fig. 3"},{"comment":"The heuristic D(C) requires a held-out set of real inlier points V_C per context; this is itself a real-data cost and should be acknowledged in the data-efficiency discussion.","section":"Sec. IV-A, Eqs. (17)-(19)"},{"comment":"The statement that increasing lambda corresponds to more frequent real data queries holds only when D(C)>0; for D(C)<=0, gamma(C)=1 regardless of lambda. Please state this qualification.","section":"Sec. V-C3, Eq. (20)"},{"comment":"The smoothing parameter eta is used in (7) but introduced only after R_t in (6); please move the definition of eta before its first use.","section":"Sec. II-B, Eqs. (5)-(7)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core idea is promising. The main reason for major revision is that the central statistic in Eq. (14) needs to be stated unambiguously, and the oracle assumption behind Proposition 1 needs to be made explicit and ideally supported by a fallback analysis. I do not see a circularity or novelty problem, and I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious look. What is genuinely new is the packaging: active p-values from Xu et al. meet conformal calibration and LORD-style online FDR control, with a context-dependent acquisition rule that actually changes how often real data is requested. The main theorem is short and leans on two external results, but I checked the active p-value step: the masking inequality holds for any proxy in [0,1], so the machinery is sound. The D(C) heuristic in Sec. IV-A is a reasonable, clearly-labeled empirical device, and the missing-data extension is a nice bonus. Experiments on two datasets show the expected pattern: sFDR is controlled, power beats plain COAD, and CDAR drops. That is real evidence the framework works under the conditions it assumes.\n\nThe soft spots are proportional. The biggest one is exactly what the stress-test note flags: Algorithm 1 step 8 assumes fresh i.i.d. real calibration samples are available whenever U_t=1. That is stated in Sec. II-A, but the experiments never exercise a failed acquisition, because the data split guarantees real calibration data at every time step. So the paper demonstrates reduced query frequency, not reduced worst-case dependency. This is a limitation, not an inconsistency, and it should be discussed explicitly—a fallback like setting Z_t=1 on failed acquisition would preserve FDR control at the cost of power, and the authors should at least mention it. Second, Proposition 1 depends on an unpublished, non-peer-reviewed preprint [37]. The construction checks out, but the paper should either prove the key step in an appendix or clearly flag the dependency. Third, manuscript hygiene: there is a stray biography and reference block from an unrelated IEEE paper appended after the reference list, plus a few mis-references (e.g., [19] is not the original LORD reference). That is sloppy but easy to fix. Finally, the figures lack error bars; with 100 repeats, there is no excuse for omitting them.\n\nWho is this for? Researchers in conformal prediction, online FDR, or anomaly detection with scarce labels. It is not a new statistical principle, but it is a useful integration with a plausible path to real deployment. I would not desk-reject it. Send to review, with the expectation that the authors address the oracle assumption, add error bars, and clean up the manuscript.","headline":"A clean, valid integration of active p-values with conformal calibration for online FDR-controlled anomaly detection, but the headline data-efficiency claim is only as strong as the assumption that real calibration data appears on demand.","tokens_in":20714,"tokens_out":2724,"would_cite":true,"duration_ms":31347,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"C-PP-COAD guarantees time-averaged false discovery rate control while deciding, per context, whether to acquire real calibration data or rely on synthetic data.","keywords":["online anomaly detection","conformal p-values","false discovery rate control","synthetic calibration data","active p-values","context-aware detection","LORD algorithm","data acquisition"],"falsifier":"Run the algorithm with a simulated context in which the real calibration batch, whenever U_t=1, is drawn from an anomalous distribution instead of the nominal P(X|C_t), while test points under the null are drawn from P(X|C_t); the empirical sFDR over many runs should rise above alpha at later times, showing the guarantee depends on uncontaminated real calibration.","tokens_in":19599,"feed_emoji":"🔍","tokens_out":5075,"duration_ms":46224,"temperature":0.7,"pith_summary":"The paper claims that online anomaly detection with a formal false-discovery-rate guarantee does not need a steady stream of fresh real calibration data. It proposes C-PP-COAD, which wraps any pre-trained anomaly score and decides at each time step, using context and a synthetic-data 'proxy' p-value, whether to spend resources acquiring a real calibration batch. The core assertion, Proposition 1, is that the resulting test keeps the decaying-memory time-averaged FDR at or below the target level $\\alpha$ for every time step, for any context sequence and any score function. Experiments on thyroid-disease and O-RAN conflict data show lower data acquisition than conformal baselines at comparable power.","feed_headline":"Synthetic data cuts real calibration needs in anomaly detection","feed_subtitle":"An online test keeps false alarms below target by querying real data only when synthetic p-values look suspicious.","key_machinery":"The load-bearing object is the active p-value statistic $Z_t = (1-U_t) Q_t + U_t (1-\\gamma(C_t))^{-1} P_t$, where $Q_t$ is the conformal p-value computed against synthetic calibration data, $P_t$ is the conformal p-value against freshly acquired real data, and $U_t$ is drawn Bernoulli with probability $1 - \\gamma(C_t) Q_t$. This is the prediction-powered step: synthetic data provides a cheap proxy that sets the probability of querying the real p-value, and the $1/(1-\\gamma)$ rescaling of $P_t$ is what keeps $Z_t$ superuniform. With $Z_t$ in hand, the LORD procedure, which sets thresholds $\\alpha_t$ from past discovery times, turns a sequence of valid p-values into online FDR control.","core_discovery":"For a fixed score function $s(X|C)$, and assuming the observations $X_t$ are i.i.d. given their contexts $C_t$, C-PP-COAD controls the smoothed decaying-memory FDR (sFDR) at level $\\alpha$ at all times. The discovery is that this guarantee survives even though the anomaly statistic $Z_t$ is sometimes computed only from synthetic calibration data. The mechanism is the active p-value: $Z_t$ coincides with a synthetic-data conformal p-value $Q_t$ when no real data is acquired, and with a rescaled real-data conformal p-value when real data is acquired, with the acquisition probability itself a function of $Q_t$. Because $Z_t$ is a valid p-value under the null hypothesis, the LORD thresholds continue to control the sFDR.","pith_inferences":["If synthetic data is perfectly calibrated for a context, $\\gamma(C)$ can be set near 1, making real acquisition rare; the guarantee then holds almost for free, although Proposition 1 itself does not quantify the acquisition savings.","The method's operating point is a data-cost-versus-power tradeoff left to the user via $\\gamma(C)$: the proposed per-context estimator of synthetic-data quality, based on the superuniformity gap $D(C)$, is heuristic, so an online, regret-bounded rule for tuning $\\gamma(C)$ is a natural testable extension.","A failure mode the paper does not treat: real data acquisition may be most expensive exactly when $Q_t$ is small, so a practical deployment should detect when the acquisition indicator is forced to 0 and can no longer guarantee FDR.","The same active-p-value construction should transfer to e-values or set-valued conformal prediction, giving analogous data-efficient guarantees for other online decision problems."],"forward_implications":["Deployments with a good digital twin can lower their cumulative data-acquisition rate well below COAD while keeping the sFDR bound; the experiments demonstrate this on both healthcare and telecom data.","The guarantee is not tied to the score function: supervised, unsupervised, and semi-supervised scores all work, so C-PP-COAD can be added on top of an existing detector.","Context-aware acquisition is more powerful per unit of real data than the context-agnostic PP-COAD, and C-COAD's higher power comes at the cost of full real-data usage.","The extension to MCAR missing data means the same FDR bound carries over when features are imputed by a fixed pretrained function.","LORD is not the only admissible threshold rule; SAFFRON and ADDIS can be swapped in, so the framework inherits other online FDR procedures."],"supporting_citations":[{"why":"supplies the active p-value construction that makes Z_t valid despite the synthetic proxy.","marker":"[37]"},{"why":"gives the COAD baseline and the LORD-based decaying-memory sFDR control that Proposition 1 builds on.","marker":"[19]"},{"why":"defines the LORD thresholds used in (16) that convert valid p-values into online FDR control.","marker":"[35]"},{"why":"establishes conformal p-values as valid tests for outliers, the basis of both Q_t and P_t.","marker":"[25]"},{"why":"provides the imputation framework that lets the guarantees carry over to MCAR missing data.","marker":"[38]"},{"why":"defines conformal prediction and its p-values, the general machinery behind the calibration step.","marker":"[24]"}],"fun_headline_variants":["Synthetic calibration data suffices for online FDR control","Adaptive data acquisition cuts real calibration needs in online anomaly detection","Context-aware p-values keep FDR low with fewer real samples","Query real data only when synthetic p-values point to anomalies"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"At every time step the system chooses to acquire real calibration data, it must actually obtain a fresh batch of n i.i.d. genuine nominal samples from P(X|C_t); if that batch is unavailable, contaminated, or drawn from the wrong context, the real conformal p-value P_t is no longer superuniform, Z_t is not a valid p-value, and Proposition 1's guarantee collapses.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic calibration data suffices for online FDR control","Adaptive data acquisition cuts real calibration needs in online anomaly detection","Context-aware p-values keep FDR low with fewer real samples","Query real data only when synthetic p-values point to anomalies"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000332,"raw_usage":{"total_tokens":1810,"prompt_tokens":872,"completion_tokens":938,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":869}},"tokens_in":488,"tokens_out":938,"duration_ms":9040,"temperature":1.0,"reasoning_tokens":869,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:12:05.689896+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the algorithm with a simulated context in which the real calibration batch, whenever U_t=1, is drawn from an anomalous distribution instead of the nominal P(X|C_t), while test points under the null are drawn from P(X|C_t); the empirical sFDR over many runs should rise above alpha at later times, showing the guarantee depends on uncontaminated real calibration.","supporting_citations":[{"cited_title":"Online false discovery rate control for anomaly detection in time series,","cited_arxiv_id":null,"evidence_quote":"gives the COAD baseline and the LORD-based decaying-memory sFDR control that Proposition 1 builds on."},{"cited_title":"Online control of the false discovery rate with decaying memory,","cited_arxiv_id":null,"evidence_quote":"defines the LORD thresholds used in (16) that convert valid p-values into online FDR control."},{"cited_title":"Testing for outliers with conformal p-values,","cited_arxiv_id":null,"evidence_quote":"establishes conformal p-values as valid tests for outliers, the basis of both Q_t and P_t."},{"cited_title":"Conformal prediction with missing values,","cited_arxiv_id":null,"evidence_quote":"provides the imputation framework that lets the guarantees carry over to MCAR missing data."},{"cited_title":"Conformal prediction: A gentle introduction,","cited_arxiv_id":null,"evidence_quote":"defines conformal prediction and its p-values, the general machinery behind the calibration step."}],"review_version":1}