Pith. sign in

REVIEW 3 major objections 5 minor 2 references

Did We Actually Fix It? An Independent Adversarial Stress-Test of Post-Point-Adjustment Evaluation Metrics for Time-Series Anomaly Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Best-of-N seed shopping decides which anomaly-detection metrics are gameable: ROC and affiliation inflate, PR-based metrics do not.

desk verdict A careful, reproducible audit that reframes metric gameability as a function of the reporting budget N; the ROC/affiliation-vs-PR split is real, and the paper deserves serious refereeing. read the letter →

arxiv 2607.11969 v2 pith:KE23MVCR submitted 2026-07-12 stat.ML cs.LG

Zongye Lyu This is my paper
classification stat.MLcs.LG
keywords time-seriesanomalydetectionevaluationmetricspointadjustmentbest-of-NreportingROCvsPRaffiliation-F1VUSrobustnesstorandomdetectors
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that whether the post-point-adjustment evaluation metrics for time-series anomaly detection are gameable by random, no-skill detectors is not a fixed property but a function of N, the number of random attempts an adversary reports the best of. On six real benchmarks, a single honest run at N=1 leaves every replacement metric robust or only mildly fragile, with no metric reaching the gameable threshold. But under best-of-N reporting, the common practice of running many random seeds and keeping the best, affiliation-F1 and every ROC-based metric inflate steeply toward state-of-the-art scores, while PR-based metrics and PA%K stay near-flat, floored near the anomaly prevalence. The paper identifies the mechanism in the order-statistic behavior of AUC under extreme class imbalance and recommends disclosing N and preferring PR-based metrics.

What carries the argument

The central object is N, the reporting budget: the number of random no-skill attempts an adversary reports the best of. It turns exploitability from a fixed property into a monotone-increasing function of N. The mechanism is the order-statistic behavior of AUC under extreme class imbalance: a random ROC-AUC has enough variance for best-of-N to climb toward 0.9, while a random PR-AUC is floored at the anomaly prevalence and cannot move; affiliation-F1 inflates through a second route, its already-high single-run value caused by its forgiveness of temporal misplacement. A secondary instrument, the forgiveness frontier, quantifies this tolerance in anomaly-length units.

What would settle it

Re-run the stress test with a reference pool of detectors taken from published leaderboards. If, with that stronger reference, best-of-N inflation of ROC or affiliation metrics on the primary benchmark falls below the 25% gameable threshold, the central claim fails. Alternatively, find any series where VUS-PR inflates but VUS-ROC does not; the paper reports a 131-to-0 one-directional split, so a single counterexample would refute that split.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the replacement metrics split cleanly along the ROC-versus-PR axis once the reporting budget is considered. At N=1 no replacement metric is gameable on any of the six benchmarks; at N=3 affiliation-F1 crosses the gameable threshold on a quarter of series and reaches 0.98 by N=41, the ROC family crosses by about N=9 to 11, and the PR family and PA%K never do on the primary benchmark. The paired comparison is one-directional: VUS-ROC is inflated on 131 series where VUS-PR is not, never the reverse. The author attributes this to the variance of random ROC-AUC versus the prevalence floor of random PR-AUC, plus affiliation's extreme single-run leniency from f

Load-bearing premise

The gameability ladder rests on an in-house pool of six real detectors standing in for state-of-the-art; if that pool is unrepresentative of true top detectors, the absolute inflation rates and the gameable/fragile/robust labels could shift, as the paper itself concedes when it names published-leaderboard outputs as the natural next strengthening.

Editorial extensions

If this is right

  • A no-skill ROC-AUC or affiliation-F1 number reported without its N is uninterpretable: the same random detector is harmless at N=1 and state-of-the-art at N=3 to 11.
  • Best-of-N affiliation-F1 can look worse than the point-adjustment control it replaced, reaching 0.98 versus about 0.76 at N=41.
  • PR-based metrics and PA%K resist best-of-N inflation on nearly every benchmark, so the field's shift toward PR-based scoring is supported.
  • VUS-ROC and VUS-PR should not be reported interchangeably; only the PR sibling retains the resistance.
  • No single metric is safe on all benchmarks: on a high-prevalence benchmark such as NAB even the PR family tips at large N, so per-dataset verification remains necessary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same best-of-N order-statistic logic should apply to any score-based evaluation under extreme class imbalance, so requiring disclosure of N could become a general norm for anomaly-detection leaderboards, not just this metric suite.
  • Editorial inference: a direct extension would be to run the released harness with a partially skilled detector that contains a small real signal; the paper argues this should inflate at least as much as pure noise, and this is testable.
  • Editorial inference: the prevalence-floor explanation predicts that a metric's crossover N changes with anomaly prevalence; a controlled sweep that resamples ground-truth prevalence could map where each metric becomes exploitable and could anticipate failures like the one seen on NAB.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper asks whether the post-point-adjustment replacement metrics used in time-series anomaly detection are themselves gameable by no-skill detectors. Its central contribution is to identify N, the number of random attempts an adversary reports the best of, as the neglected axis: at N=1 (a single honest run), none of the eleven replacement metrics reaches the GAMEABLE threshold on any of six benchmarks; under best-of-N seed-shopping, affiliation-F1 and every ROC-based metric inflate steeply (affiliation crosses the threshold by N=3 and reaches 0.98 at N=41; the ROC family crosses by N≈9–11), while PR-based metrics and PA%K stay near-flat. A paired McNemar test shows VUS-ROC inflated on 131 series where VUS-PR is not, with no reverse cases. The mechanism is attributed to the order-statistic behavior of AUC-ROC versus AUC-PR under extreme class imbalance, plus affiliation's intrinsic temporal leniency. The paper also introduces a forgiveness frontier, a rank-agreement analysis, and a released, version-controlled stress-test harness with the metric authors' own frozen code.

Significance. If the results hold, this is a genuinely useful contribution. The paper fills a real independence gap: prior adversarial checks of these metrics were mostly proposer-run, and independent audits targeted point-adjustment or dataset quality rather than the replacement metrics themselves. The harness is reproducible, with positive-control validation that reproduces published anchors (e.g., random PA-F1≈0.89, random affiliation-F1≈0.68, VUS-ROC buffer behavior), and the full N-curve is a more informative statistic than a single gameability number. The paired one-directional ROC-over-PR split is a strong, falsifiable finding. The main residual risk, as the paper itself concedes, is that the SOTA reference is an in-house six-detector pool: the gSOTA diagnostic protects the ROC/affiliation direction (real detectors score 0.97–1.00 on gamed series), but for the PR family the reference is near the floor on the few gamed series, so absolute PR rates could shift under a published-leaderboard reference. The NAB exception also shows that no metric is universally safe. These caveats weaken the absolute calibration but not the core ordering claim.

major comments (3)
  1. [§3.4, §5.4, Table 4] The headline absolute inflation rates and the ROBUST/FRAGILE/GAMEABLE labels are calibrated against an in-house six-detector SOTA pool. The gSOTA diagnostic convincingly protects the ROC/affiliation labels: on the series where those metrics are gamed, real detectors already score 0.97–1.00. However, for the PR family the gSOTA on the few gamed series is only 0.04–0.17 (near the floor), so the absolute PR rates (e.g., VUS-PR 0.08 at N=41, point-AUC-PR 0.08) are measured against a reference that barely separates real from random. The statement that their true robustness is 'likely better' is conjectural. A stronger PR reference—such as published leaderboard outputs, as Limitation 1 suggests—could change the absolute rates and even the FRAGILE/ROBUST tier of these metrics, even though the ROC/PR ordering would likely survive. Please either add such a reference or explicitly reframe the PR-f
  2. [§5.4, Table 2, Figure 2] range-AUC-PR is excluded from the 'PR is safe' conclusion after observing that it is GAMEABLE at N=41. The 'floored-reference artifact' explanation is plausible and supported (median no-skill 0.51 ≈ median SOTA 0.51, real detectors below 0.6 on all gamed series), but the exclusion is post hoc. Since the released configuration is described as fixing all outcome-defining choices, the manuscript should either pre-specify a degenerate-reference exclusion criterion or report results both including and excluding range-AUC-PR prominently. As written, a skeptical reader cannot fully separate a genuine PR-family exception from an artifact.
  3. [§5.5, Table 4] The cross-benchmark replication shows that PR-family resistance is not universal: on NAB at N=41, VUS-PR and PA%K reach 0.25 and range-based F1 reaches 0.35, all GAMEABLE. The abstract's 'stay near-flat at every N' is therefore true only outside NAB, and the recommendation to prefer PR-based metrics should be conditional on benchmark characteristics. The decision protocol already advises per-dataset verification, but the paper's central summary and abstract under-weight the NAB exception. Please make the limits of the PR-family claim more prominent, for example by reporting the N at which each PR metric crosses GAMEABLE on NAB, and by stating the prevalence/strength conditions under which the PR floor fails.
minor comments (5)
  1. [Abstract] 'Not one replacement metric is gameable' at N=1 is true relative to the 25% GAMEABLE threshold, but affiliation-F1 is already FRAGILE at 11%. Consider saying 'not GAMEABLE' to avoid the impression that all metrics are robust at N=1.
  2. [§3.3] The '41 independent attempts' include one deterministic constant detector; 'independent' therefore applies only to the stochastic generators. A one-word clarification would help.
  3. [Table 2 / Figure 2] The note that range-AUC-PR is omitted from Table 2 and excepted in Figure 2 should be cross-referenced to the §5.4 exclusion, so that a reader jumping to the results does not see an unexplained gap.
  4. [§5.3] The rank-agreement section carefully distinguishes 'not established' from 'demonstrated absent'; good. The statement that top-1 flip is 'coarse' with six detectors is appropriately hedged.
  5. [§5.4] The phrase 'applied evenhandedly to every metric so the diagnostic cannot be selectively invoked' is useful; consider adding the gSOTA values for all metrics in a small table for transparency.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: central claims are measured empirical results benchmarked against external metric code and independent reference checks, with no fitted parameter renamed as a prediction.

full rationale

This paper is an empirical stress-test, not a derivation chain, and no load-bearing step reduces to its own inputs by construction. The gameability criterion ('best of N random no-skill reaches ≥90% of SOTA') is defined against an in-house six-detector pool, but this is not circular: the results are also reported against chance expectation (Section 3.4), the gSOTA diagnostic (Section 5.4) shows real detectors score 0.97–1.00 on the exact series where ROC/affiliation gaming occurs, the ROC/PR split replicates across five additional external benchmarks (Section 5.5), and the authors explicitly concede the in-house pool as Limitation 1 (Section 7). The best-of-N inflation curve is a directly measured order-statistic effect (Section 5.1), not an imposed result; the 90%-of-SOTA and 25% thresholds are hand-chosen but shown to be cutoff-invariant at fixed N. Every metric is computed with the metric authors' frozen code (affiliation @8d84498, TSB-UAD @313f0fd, prts 1.0.0.3), which is independent external support rather than self-citation. There are no load-bearing self-citations, no equation-level reduction of an output to an input, and no fitted parameter renamed as a prediction. The residual SOTA-reference calibration risk is a correctness/validity concern about absolute labels, not a circularity. Honest non-finding is therefore appropriate.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The analysis rests on standard statistical facts (order-statistic monotonicity, PR-flooring), domain assumptions about benchmark labels and the representativeness of the in-house detector pool, and hand-chosen thresholds that are shown to be cutoff-invariant. No new physical or theoretical entities are postulated; the forgiveness frontier and gSOTA are analysis instruments, not entities.

free parameters (7)
  • gameability_fraction = 0.90
    A no-skill score must reach ≥90% of SOTA to count as inflated; tested at 0.85 and 0.95 with unchanged tier classification.
  • gameable_cutoff = 0.25
    ≥25% of series inflated defines GAMEABLE, with FRAGILE between 5–25%; tested at 20% and 30% cutoffs.
  • replicate_count_R = 10
    Ten Monte-Carlo draws per stochastic generator; yields the 41-attempt pure no-skill pool (4×10 + 1 constant).
  • length_cap = 40,000
    UCR series longer than 40k points are cropped to a 40k window centered on the anomaly; timing-driven cap that could affect absolute rates.
  • vus_threshold_resolution = 250
    Resolution for VUS ROC/PR computation.
  • near_miss_shifts = {0.1, 0.25, 0.5, 1, 2, 4} × median anomaly length
    Sweep for the forgiveness frontier.
  • sota_detector_pool = Matrix Profile, iForest, LOF, PCA reconstruction, kNN, AR residual
    In-house reference pool defining SOTA; the paper's Limitation 1 flags it as a candidate weakness.
assumptions (5)
  • standard math The best of N independent random draws is monotone non-decreasing in N
    Used to frame N as the reporting budget and interpret N=41 as a worst case (Sections 3.3, 5.1).
  • standard math A random PR-AUC is floored at the anomaly prevalence, while a random ROC-AUC has substantial variance under extreme class imbalance
    The explanatory mechanism for the ROC/PR split; cited to Davis & Goadrich (2006) and Saito & Rehmsmeier (2015), Section 5.1.
  • domain assumption The ground-truth labels of UCR, SMD, SMAP, MSL, NAB, and PSM are treated as correct
    The audit compares no-skill scores against these labels; label errors would alter rates.
  • domain assumption The in-house detector pool is a meaningful stand-in for state-of-the-art detectors
    The SOTA reference and the 90%-of-SOTA criterion depend on this; partially tested by the gSOTA diagnostic (Section 5.4) and listed as Limitation 1 (Section 7).
  • domain assumption The 41 no-skill attempts are independent draws for the best-of-N analysis
    Best-of-N is computed without replacement over the pool; if attempts were strongly correlated, the N-curve could be shallower, though the empirical curve matches the independent-draw complement (1−(1−p)^N).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Did We Actually Fix It? An Independent Adversarial Stress-Test of Post-Point-Adjustment Evaluation Metrics for Time-Series Anomaly Detection." pith.science (2026). https://pith.science/paper/KE23MVCR

@misc{pith2026260711969,
  author       = {Pith},
  title        = {Pith review of: Did We Actually Fix It? An Independent Adversarial Stress-Test of Post-Point-Adjustment Evaluation Metrics for Time-Series Anomaly Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KE23MVCR}},
  note         = {Machine review of arXiv:2607.11969}
}
read the original abstract

Point-adjustment (PA), for years the default scoring protocol in time-series anomaly detection (TSAD), was shown by Kim et al. (2022) to award near-perfect F1 to random anomaly scores. The field adopted a suite of replacement metrics (PA%K, range-based precision/recall, affiliation precision/recall, and Volume-Under-the-Surface, VUS, ROC/PR). We ask, independently and adversarially, whether these resist no-skill detectors on real benchmarks, and find the answer turns entirely on one overlooked variable: N, the number of random attempts an adversary reports the best of. Under a single honest run (N=1), not one replacement metric is gameable on any of six benchmarks (UCR, SMD, SMAP, MSL, NAB, PSM): a random detector reaches 90% of the best real detector's score on at most 11% of series for affiliation-F1, 5% for the ROC family, and 2% for the PR-based metrics and PA%K. But under best-of-N reporting, the seed-shopping endemic to ML, the metrics split sharply. affiliation-F1 and every ROC-based metric inflate steeply, affiliation crossing gameable (25% of series) by N=3 and reaching 0.98 at the full pool (N=41), the ROC family crossing by N=9-11; the PR-based metrics and PA%K stay near-flat at every N, floored near the anomaly prevalence (the lone exception is NAB at large N). A paired test finds VUS-ROC inflated on 131 series where its sibling VUS-PR is not, and never the reverse. The ROC-vs-PR split follows from the order-statistic behaviour of AUC under extreme class imbalance (a random PR-AUC is floored at prevalence); affiliation inflates by a second route, its extreme single-run leniency (already fragile at N=1). We release a pip-installable stress-test harness, and recommend reporting single-run scores or disclosing N and preferring PR-based metrics, which resist best-of-N inflation on nearly every benchmark.

Figures

Figures reproduced from arXiv: 2607.11969 by the authors.

Figure 1
Figure 1. Best-of-N inflation (the paper’s central result): the fraction of UCR series on which a pure no-skill [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. The ROC/PR split at the N=41 (full pure pool) best-of-N worst case: each ROC metric’s inflation [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Forgiveness frontier: median near-miss ratio-to-SOTA vs ground-truth shift (in median anomaly [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Inter-metric rank agreement: distribution of per-series mean pairwise Kendall [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

2 extracted references

  1. [240]

    Garg, A., Zhang, W., Samaran, J., Savitha, R., Foo, C.-S

    doi:10.1145/1143844.1143874. Garg, A., Zhang, W., Samaran, J., Savitha, R., Foo, C.-S. (2022). An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series.IEEE TNNLS33(6):2508–2517. doi:10.1109/TNNLS.2021.3105827. Ghorbani, R., Reinders, M. J. T., Tax, D. M. J. (2024). PATE: Proximity-Aware Time Series Anomaly Evaluation.KDD. doi:10.1145/...

    arXiv 2022

  2. [2787]

    Paparrizos, J., Kang, Y., Boniol, P., Tsay, R

    doi:10.14778/3551793.3551830. Paparrizos, J., Kang, Y., Boniol, P., Tsay, R. S., Palpanas, T., Franklin, M. J. (2022). TSB-UAD: An End-to-End Benchmark Suite for Univariate Time-Series Anomaly Detection.PVLDB15(8):1697–1711. doi:10.14778/3529337.3529354. Rewicki, F., Denzler, J., Niebling, J. (2023). Is It Worth It? Comparing Six Deep and Classical Method...

    arXiv 2022

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.