{"id":"f584b142-d720-48eb-82fa-da08ce3b1c88","arxiv_id":"2509.01098","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CCE is a bounded, robust, O(n) event-level metric for time series anomaly detection that combines prediction confidence with Bayesian uncertainty consistency, and RankEval is a new benchmark for comparing such metrics.","lead":"This paper proposes CCE, a new way to score time series anomaly detection models by combining how confident a model is with how consistent its uncertainty is. It also introduces RankEval, a benchmark to test which scoring metrics best rank models by true performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RankEval's expected rankings are monotone transforms of the event-level score means that CCE's confidence terms use, so the perfect Table I scores are built in rather than demonstrating that CCE is the best ranking metric.","rationale":"The reader's weakest_assumption is the Beta-distribution modeling. I agree that assumption is unvalidated, but the more load-bearing weakness is the circular construction of the synthetic expected rankings. Even if scores were perfectly Beta-distributed, the perfect Table I scores would still be predetermined because R* is a monotone function of the same event-level means on which CCE's confidence terms are based. The reader's rationale does flag this as 'partially circular,' so my concern overlaps with the reader's broader critique, but it is not the same as the stated weakest_assumption. I therefore mark agreement as partial. I also considered the Lipschitz proof: the bound on the partial derivative of n with respect to s_k in Eq. (33) is not justified and can fail when event variance is small, so the robustness proof needs repair. However, the empirical 'best ranking metric' claim is the headline contribution, and the circularity concern most directly attacks that claim. A single analytical check can settle it: if the mean-only ranking already yields 1.000/1.000/0.000 for every task, CCE's Table I result is a consequence of the benchmark definition rather than a competitive finding. Since the paper can be revised by adding non-circular benchmark tasks and fixing the proof, the reader's CONDITIONAL verdict remains appropriate; no verdict change is needed from my pass.","tokens_in":21312,"tokens_out":12962,"duration_ms":182479,"concrete_test":"Compute, for each of the five RankEval tasks in Table I, the Spearman/Kendall/MD between R* and the ranking obtained solely from the event-level mean confidence terms (Eqs. 5 and 7), ignoring the consistency factor. If this mean-only ranking is already 1.000/1.000/0.000 on every task, CCE's perfect scores are explained by construction and the benchmark is circular; if any task has a non-perfect mean-only ranking while CCE remains perfect, the circularity concern is refuted and CCE's consistency term is doing real work.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing empirical assertion is the perfect Avg. Sp=1.000, Kd=1.000, MD=0.000 in Table I. In every RankEval synthetic task, the expected ranking R* is defined by q or p, and q/p are directly encoded in the event-level mean of generated anomaly scores. For AccQ, the anomaly-event mean is q*0.95+(1-q)*0.025, strictly increasing in q; for PreQ-NegP-P, the normal-event mean is 0.05+0.5p, strictly increasing in p. CCE's anomaly/normal confidence (Eqs. 5 and 7) are thresholds on exactly these event-level means, and consistency (Eq. 9) is a positive multiplicative factor in [exp(-0.25),1]. Thus the ranking induced by CCE is essentially the ranking by score mean, which is identical to R* by construction. The benchmark therefore does not provide independent evidence that CCE is the best metric; it verifies that CCE implements the same statistic used to define quality. The Beta-distribution assumption is secondary here: U is a fixed transform of mean/variance, so even a non-Beta score distribution would still yield a bounded variance penalty; the RankEval circularity is what undermines the central 'best ranking metric' claim. The real-data results are too small and partly qualitative to substitute for that evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces CCE, a TSAD evaluation metric that combines event-level and global confidence scores based on thresholded means of anomaly scores with a consistency term exp(-U), where U is the variance of a Beta distribution fitted to each event by the method of moments. The authors prove boundedness, Lipschitz robustness, and O(n) complexity for CCE, and introduce RankEval, a benchmark with synthetic ASGM score generators and real-world datasets. They report that CCE achieves perfect ranking consistency (Sp=Kd=1.000, MD=0.000) across all RankEval scenarios and is the only metric with complete noise robustness.","tokens_in":21674,"tokens_out":11489,"duration_ms":131598,"significance":"The idea of an uncertainty-aware, threshold-independent interval-level TSAD metric is timely, and the open-source release plus the attempt to standardize metric evaluation through RankEval are useful contributions. If the theoretical properties and ranking results held, CCE would be a practical and efficient evaluation tool. However, the central ranking claim is undermined by the way RankEval constructs ground-truth rankings from the same statistics CCE uses, and the Lipschitz robustness proof contains a derivative bound that is not generally valid. The contribution is a promising first step, but the main claims need substantial revision before publication.","major_comments":[{"comment":"The proof of Theorem 6 relies on Eq. (33), which states |∂p/∂s_k|≤2/|E_i| and |∂n/∂s_k|≤2/|E_i|. The first bound is loose but harmless (p=α/(α+β)=bar{s}_{E_i}, so the true derivative is 1/|E_i|). The second is incorrect: from n=s(1-s)/m_2−1 with m_2 the sample second central moment, ∂n/∂s_k=(1−2s)/m_2 − 2s(1−s)(s_k−s)/(|E_i| m_2^2), which is unbounded as m_2→0. Thus the uniform constant C1=1.5 in Eq. (36) and L=0.75/|E_i|+1 in Eq. (41) are not established. The premise that real events have length ≥20 is also contradicted by Table S5 (Creditcard segment lengths 1–5). The Lipschitz claim may hold on a restricted domain, but the proof as written does not support it.","section":"Section IV-B, Eqs. (31)-(36)"},{"comment":"The RankEval expected ranking R* is defined directly from the ASGM generation parameters q and p. In AccQ(-R), the anomaly-event mean score is an increasing function of q, and in PreQ-NegP-P, the normal-event mean is an increasing function of p. CCE's confidence terms (Eqs. (5)–(8)) are threshold functions of exactly these event-level means, and Cons(E_i) in Eq. (9) is a positive factor in [e^{−0.25},1]. Therefore the ranking induced by CCE is essentially the ranking by score mean, i.e., the same statistic used to construct R*. The perfect Avg. Sp=1.000, Kd=1.000, MD=0.000 in Table I is a built-in consistency check rather than independent evidence that CCE is the best ranking metric. Independent benchmarks with ground truth not monotonically related to the mean score, or real model outputs with externally judged quality, are needed. The ASGM definitions in Sec. V-A2 are also self-contrad","section":"Section V-C1 and Table I"},{"comment":"The confidence definitions use max(·,0) (Eqs. (5), (7)), and Algorithm 1 returns SCCE computed from these definitions, which is nonnegative. However, Table III reports negative CCE scores (ECG LOF −0.01, A.T. −0.21; Power IForest −5.1), which can only arise from the 'relaxed' confidence forms in Eqs. (6) and (8). The paper does not state that experiments use the relaxed variant; this ambiguity affects the interpretation of Table I and all real-data comparisons, and the reported negative scores are inconsistent with the stated bound SCCE∈[0,1]. Please specify exactly which variant was implemented, justify its use, and align the definitions, algorithm, and experiments.","section":"Section III-C, Algorithm 1, Table III"},{"comment":"The entire consistency term and the Lipschitz proof depend on the assumption that anomaly scores for each event follow a Beta distribution and that the method-of-moments parameters in Eq. (3) are meaningful. No goodness-of-fit or sensitivity analysis is provided for real TSAD scores, which are often skewed, multimodal, or degenerate (e.g., constant segments). In particular, the derivative bounds in the robustness proof use the Beta variance formula (Eq. (4)) and fail or require regularization when m_2=0. The paper should either prove the property under weaker assumptions on the score distribution, add a robustness check on distributions other than Beta, or state the restricted domain. Additionally, Theorem 6 only analyzes event-level scores; the global scores used in Eq. (13) are not bounded by the same argument, although Eq. (40) applies the event-level Lipschitz constant to the sum.","section":"Section III-B and IV-B"}],"minor_comments":[{"comment":"The algorithm lists a 'scale parameter γ' that is never defined or used anywhere in the paper; please remove it or clarify its role.","section":"Algorithm 1"},{"comment":"The header is confusing: the first row reads 'Threshold 0.1 0.3 ...' and the second row reads 'τ 1/1/0 1/1/0 ...'. These should be labeled Sp/Kd/MD for each threshold value.","section":"Table IV"},{"comment":"The notation swaps Conf and Conf*: Eq. (19) uses Conf* for the constrained case and Eq. (20) uses Conf for the relaxed case, opposite to Definitions 3 and 4. Please fix the asterisk convention.","section":"Theorem 4, Eqs. (19)-(20)"},{"comment":"The statement that 'eTaPR always considers the model score to be 0' is unclear and should be substantiated with a concrete example or explanation of the metric behavior.","section":"Section V-D2"},{"comment":"In the AccQ generator, the normal-label case says 'when correctly predicted, the anomaly score is s(1)', but s(1) is the high anomaly score; this is presumably a typo (should be s(2)). Please correct because the synthetic benchmark's semantics depend on this definition.","section":"Section V-A2, AccQ definition"}],"recommendation":"major_revision","confidential_remarks":"The RankEval circularity is substantial: the perfect Table I scores are largely a consequence of CCE using the same event-level mean statistic that defines the expected ranking. Combined with the incorrect Lipschitz bound and the ambiguity about relaxed versus non-relaxed confidence (negative scores in Table III), the paper should not be accepted in its current form. The approach is salvageable if the authors redesign the benchmark, correct the robustness proof under explicit assumptions, and align the implementation with the stated definitions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nShort version: a genuinely useful, incremental idea wrapped in overclaimed validation. CCE—event-level confidence from mean scores plus a consistency term from score variance—is a new TSAD evaluation metric, and O(n) complexity plus strict boundedness are real practical selling points. Anyone doing model selection on large anomaly datasets should know about it.\n\nWhat's actually good: the motivation is solid. F1-PA and Aff-F1 are known to inflate scores, and an interval-aware metric with no buffer hyperparameter and linear time fills a real niche. The RankEval benchmark, synthetic score generators with explicit accuracy/precision parameters, and the ranking metrics (Spearman, Kendall, MD) are a reasonable systematic way to compare metrics. The open-source implementation is a plus. The boundedness of the Beta variance and the O(n) argument both hold.\n\nThe soft spots are in the validation and theory. First, the RankEval synthetic tasks are circular in a load-bearing way. R* is defined by q or p, and those same q and p are encoded directly in the event-level means that CCE's confidence term thresholds. So CCE's perfect 1.000 in Table I is the metric agreeing with the statistic used to define quality—not independent evidence that CCE ranks better than other metrics. AUC-ROC and VUS-ROC also do well, but the benchmark still can't separate a good metric from one that just uses score means.\n\nSecond, the Lipschitz proof has errors. Eq. (33) says |∂p/∂s_k| ≤ 2/|E_i|, but p is the mean, so the derivative is 1/|E_i|. The bound on ∂n/∂s_k also looks optimistic—when event variance is near zero, that derivative can blow up. The C1=1.5 constant rests on these bounds, so the robustness theorem isn't established as stated.\n\nThird, the Beta-distribution assumption is asserted, not validated. There's no goodness-of-fit test, and the per-point uncertainty U_k in Def. 5 is never defined consistently with the event-level U of Section III-B. The consistency term is just a bounded function of variance, so the Beta machinery is not clearly load-bearing, but then the 'Bayesian uncertainty' framing is weaker than claimed.\n\nThe real-data results are too small and partly qualitative to substitute. Fixing this requires (a) redefining RankEval expected rankings from a ground truth not monotonically tied to the statistics CCE uses, (b) correcting the Lipschitz derivation, and (c) checking the uncertainty model on real score distributions.\n\nWho it's for: TSAD evaluation and model selection. It deserves peer review but with major revisions; I'd send it to a careful referee and push the authors to address the circularity head-on.","headline":"A useful TSAD evaluation metric with O(n) complexity and boundedness, but the headline validation is circular and the robustness proof has real gaps.","tokens_in":22170,"tokens_out":6044,"would_cite":true,"duration_ms":66182,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes CCE, an anomaly-detection evaluation metric that multiplies prediction confidence by a Bayesian uncertainty-consistency term, and claims it is the only interval-level metric that ranks models perfectly, survives score no","keywords":["time series anomaly detection","evaluation metric","Bayesian uncertainty estimation","Beta distribution","ranking benchmark","Lipschitz robustness","model confidence"],"falsifier":"Run a Kolmogorov-Smirnov test comparing each event's empirical score distribution to the fitted Beta(alpha, beta) from Eq. (3) on outputs of standard detectors (e.g., LSTMAD, USAD, AnomalyTransformer) across MSL, SMD, PSM, and SWaT; if a substantial fraction of events reject the Beta fit, the consistency term is not measuring what the proofs assume. A direct ranking test also works: construct two detectors with identical expected accuracy but score distributions of different shape (bimodal versus Beta-like); CCE predicts the lower-variance one ranks better, and that ordering is checkable again","tokens_in":21130,"feed_emoji":"📊","tokens_out":5864,"duration_ms":63531,"temperature":0.7,"pith_summary":"This paper argues that how an anomaly detector's scores are shaped matters as much as whether they match the labels, and proposes CCE, a metric that scores a detector on both confidence and the consistency of its uncertainty. The metric is built by treating each event's scores as a Beta distribution, reading the distribution's variance as uncertainty, and converting that uncertainty into a penalty term; the authors prove the resulting score is bounded in [0,1], Lipschitz-robust to score perturbations, and computable in O(n) time. They also introduce RankEval, a benchmark that ranks the ranking metrics themselves, using Spearman, Kendall, and mean-rank-deviation against ground-truth model quality. On that benchmark, CCE is the only metric with perfect ranking consistency across every scenario and the only one insensitive to injected noise, which would make it the first fast, threshold-free, uncertainty-aware standard for choosing time series anomaly detection models.","feed_headline":"New anomaly-detection metric ranks detectors perfectly in every test","feed_subtitle":"One score folds confidence and uncertainty into model ranking, hitting zero ranking error with linear-time speed.","key_machinery":"The key machinery is the Beta-distribution uncertainty estimate. The paper treats each event's normalized anomaly scores as draws from a Beta(alpha, beta), with alpha and beta fixed by the sample mean and second central moment (method of moments), and takes the Beta variance U = alpha*beta / ((alpha+beta)^2 (alpha+beta+1)) as the event's uncertainty. Consistency of an event is exp(-mean U), so high-variance score regions are down-weighted even when their mean score is high; the same U-derived bounds (max variance 1/4, and the constant C1 = 1.5 in the robustness proof) underlie both the boundedness and perturbation-robustness theorems.","core_discovery":"The paper claims that evaluating a time series anomaly detector should not stop at whether its scores match the labels; the scores themselves carry a reliability signal. CCE models each normal and anomalous event's scores as a Beta distribution (parameters via method of moments), reads the distribution's variance as the event's uncertainty, and derives an event-level consistency score exp(-mean uncertainty). Multiplied by a confidence score (mean score minus a symmetric threshold) and combined across events and the whole series, this yields one number in [0,1] that the authors prove is bounded, Lipschitz-continuous in the scores with constant near 1, and computable in O(n). On the paper's ow","pith_inferences":["The Beta assumption is untested: since the consistency term inherits all its meaning from the variance of a fitted Beta, real score distributions that are multimodal or heavy-tailed could make CCE's uncertainty estimate diverge from genuine predictive reliability; a goodness-of-fit check on standard detectors' outputs would show how far the Lipschitz guarantee actually carries.","Because CCE consumes only labels and scores, the same confidence-times-consistency product could transfer to ranking detectors in other domains, such as classification calibration or image anomaly detection, where threshold-free uncertainty-aware ranking is also missing.","The confidence and consistency terms are both monotone functions of the same Beta variance, so part of CCE's reported edge may come from amplifying score spread rather than from true calibration; ablating the consistency term against a variance-free surrogate (e.g., discretized-score entropy) would separate the two effects.","RankEval's synthetic space could be extended with distribution shift, drift, or seasonal anomalies to test whether CCE's perfect ranking and noise immunity survive when the score-generating process itself changes over time."],"forward_implications":["If CCE is right, model selection in time series anomaly detection can use a threshold-free metric whose ranking matches ground-truth model quality without tuning buffers or thresholds, unlike Aff-F1, VUS-ROC, and PATE.","CCE's O(n) cost makes interval-aware evaluation feasible on million-point industrial series, where the paper measures VUS-ROC and PATE as orders of magnitude slower.","The consistency term introduces a new signal: models that emit confident-looking but erratic scores are penalized even when their thresholded labels look fine, making prediction reliability itself a measurable and trainable property.","The RankEval pipeline makes metric-versus-metric comparison reproducible, so future proposed metrics can be ranked against existing ones rather than defended on single datasets.","Because CCE is symmetric between normal and anomalous events, it can report which side of a detector's behavior is weaker, guiding targeted model improvement."],"supporting_citations":[{"why":"Documents that existing TSAD benchmarks and metrics are flawed and create an illusion of progress, motivating the need for a new metric.","marker":"[14]"},{"why":"Supplies the UAff-F1 baseline and the prior-limitation analysis (low discriminability, threshold dependence) that CCE extends.","marker":"[15]"},{"why":"Defines Aff-F1, the affiliation-based interval metric CCE must outperform in ranking consistency.","marker":"[20]"},{"why":"Defines VUS-ROC, the strongest threshold-free interval baseline used for both ranking comparison and latency comparison.","marker":"[21]"},{"why":"Defines PATE, the buffer-based baseline whose high cost sets the latency comparison CCE claims to beat.","marker":"[22]"},{"why":"Supplies the Bayesian estimation of Beta mixture models that grounds CCE's Beta-distribution uncertainty estimator.","marker":"[28]"},{"why":"Defines Spearman's rank correlation, one of the three ground-truth ranking indicators used by RankEval.","marker":"[38]"},{"why":"Defines Kendall's tau, the second ranking indicator used by RankEval for metric comparison.","marker":"[39]"}],"fun_headline_variants":["New metric for anomaly detectors: bounded, robust, and O(n)","Score anomaly detectors by confidence and uncertainty consistency","Anomaly detection metric with Lipschitz robustness and linear time","CCE: evaluate detectors with one number from Beta-distributed scores","Anomaly metric that twists score uncertainty into a ranking signal"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that each event's anomaly scores are well approximated by a Beta distribution, because the consistency penalty is computed from Beta-variance and the Lipschitz proof is derived from Beta-specific derivatives; if real scores are multimodal or heavy-tailed, both the uncertainty's meaning and the robustness bound can break down.","fun_headline_variants_meta":{"raw":{"variants":["New metric for anomaly detectors: bounded, robust, and O(n)","Score anomaly detectors by confidence and uncertainty consistency","Anomaly detection metric with Lipschitz robustness and linear time","CCE: evaluate detectors with one number from Beta-distributed scores","Anomaly metric that twists score uncertainty into a ranking signal"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00062,"raw_usage":{"total_tokens":2683,"prompt_tokens":684,"completion_tokens":1999,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":428,"completion_tokens_details":{"reasoning_tokens":1914}},"tokens_in":428,"tokens_out":1999,"duration_ms":19408,"temperature":1.0,"reasoning_tokens":1914,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:51:03.057155+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a Kolmogorov-Smirnov test comparing each event's empirical score distribution to the fitted Beta(alpha, beta) from Eq. (3) on outputs of standard detectors (e.g., LSTMAD, USAD, AnomalyTransformer) across MSL, SMD, PSM, and SWaT; if a substantial fraction of events reject the Beta fit, the consistency term is not measuring what the proofs assume. A direct ranking test also works: construct two detectors with identical expected accuracy but score distributions of different shape (bimodal versus Beta-like); CCE predicts the lower-variance one ranks better, and that ordering is checkable again","supporting_citations":[{"cited_title":"Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress,","cited_arxiv_id":null,"evidence_quote":"Documents that existing TSAD benchmarks and metrics are flawed and create an illusion of progress, motivating the need for a new metric."},{"cited_title":"Simad: A simple dissimilarity- based approach for time-series anomaly detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the UAff-F1 baseline and the prior-limitation analysis (low discriminability, threshold dependence) that CCE extends."},{"cited_title":"Local evalu- ation of time series anomaly detection algorithms,","cited_arxiv_id":null,"evidence_quote":"Defines Aff-F1, the affiliation-based interval metric CCE must outperform in ranking consistency."},{"cited_title":"V olume under the sur- face: a new accuracy evaluation measure for time-series anomaly detection,","cited_arxiv_id":null,"evidence_quote":"Defines VUS-ROC, the strongest threshold-free interval baseline used for both ranking comparison and latency comparison."},{"cited_title":"Pate: Proximity-aware time series anomaly evaluation,","cited_arxiv_id":null,"evidence_quote":"Defines PATE, the buffer-based baseline whose high cost sets the latency comparison CCE claims to beat."},{"cited_title":"Bayesian estimation of beta mixture models with variational inference,","cited_arxiv_id":null,"evidence_quote":"Supplies the Bayesian estimation of Beta mixture models that grounds CCE's Beta-distribution uncertainty estimator."},{"cited_title":"The proof and measurement of association between two things,","cited_arxiv_id":null,"evidence_quote":"Defines Spearman's rank correlation, one of the three ground-truth ranking indicators used by RankEval."}],"review_version":1}