REVIEW 4 major objections 5 minor 43 references
Open Challenges in Time Series Anomaly Detection: An Industry Perspective
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This paper argues that the field's definitions miss how industry actually uses anomaly detection: as streaming alerting on populations of series, with side information, human feedback, and conditional anomalies.
desk verdict A well-written industry position paper that names real gaps and offers two novel formalizations, but its core 'application-specific' tenet remains an assumption rather than a demonstrated fact. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by two formal definitions. The first is the mixture model $\bar{p}(x_t \mid x_{<t}) = (1-\varepsilon) p(x_t \mid x_{<t}) + \varepsilon q(x_t \mid x_{<t})$, in which $p$ is a Markov process of unknown order and a point is anomalous when $p(x_t \mid x_{<t}) < \eta$, making anomaly relative to a learned model of normality. The second, Eq. (4), restates the detection target as the conditional probability of one series given other series' current and past values, distinguishing conditional anomalies from the joint multivariate anomalies that dominate the literature. These definitions convert practical complaints into precise problem statements: streaming evaluation as Eq. (1), subpopulation mining as finding a simple rule $f_t(\theta_i)$ over time-invariant attributes, and human-in-the-loop regret as Eq. (5), a censored-feedback objective with application-specific costs for false positives and negatives.
What would settle it
Show a dataset of time series drawn from several industries where independent domain experts, given only the raw signal and told to label anomalies for their own application, produce near-identical labels; high agreement would contradict the claim that anomaly status is application-specific rather than intrinsic to the data.
Extended reading notes
Core claim
The paper's central claim is that the most practically relevant question in time-series anomaly detection is usually not the joint multivariate question $\min_t p(x_t, y_t, z_t \mid x_{<t}, y_{<t}, z_{<t})$, but the conditional question $\min_t p(x_t \mid x_{<t}, y_{<t+1}, z_{<t+1})$: whether the current value of one series is unusual given the current and past values of other series. This conditional-anomaly formulation appears, the author argues, nowhere in the literature, despite being the most common scenario in practice. The paper also formalizes the online alerting setting as one where the algorithm sees only past data and receives labels only for points it flagged, yielding a censored-feedback regret objective, and reports a small synthetic study where simple autocorrelation heuristics outperform published periodicity-detection methods. Together, these claims say that streaming evaluation, populations, conditional anomalies, human feedback, explainability, and signal preprocessing are core to the field as practiced, not optional extras.
Load-bearing premise
The load-bearing premise is the application-specific tenet: that whether an event is an anomaly is not determined by the data itself but by the application, and the paper admits that rigorously proving this is difficult; if anomalies are largely identifiable from the signal alone, the agenda for side information, conditional anomalies, and human feedback loses much of its urgency.
Editorial extensions
If this is right
- Benchmark evaluation of TAD algorithms should switch from batch scoring, Eq. (2), to streaming scoring, Eq. (1), where each point is judged using only previously seen data, and detection delay should be reported as an explicit metric.
- Datasets and methods should incorporate side information: time-invariant attributes for subpopulation or cohort anomalies and time-varying covariates for conditional anomalies as defined in Eq. (4).
- Human feedback should be modeled as a censored-bandit online learning problem, because operators label only points the system flagged, and regret should weight false positives and negatives by application-specific costs.
- Resampling of point processes and selection of the periodicity or window size should be treated as first-class components of the TAD pipeline, since they can destroy or create anomalies.
- Anomaly labels for benchmarking need to be curated with human consistency rules and should contain multiple anomaly instances per series so that supervised and weakly-supervised settings can be studied.
Reading between the lines
- If the application-specific tenet is correct, universal leaderboards that rank algorithms on fixed labeled datasets have a ceiling: rankings can only be meaningful relative to a stated application context and cost structure, so future benchmarks may need to sample applications rather than series.
- The conditional-anomaly definition in Eq. (4) can be operationalized immediately with regression residuals on existing multivariate benchmarks, which the paper itself suggests; this would test whether the setting changes algorithm rankings.
- The censored-feedback formulation connects TAD to the bandit literature on selective labels, so regret bounds and exploration strategies from threshold bandits could be imported into anomaly alerting.
- The appendix's periodicity result hints that many window-size methods are over-engineered relative to a simple autocorrelation peak heuristic; adding this heuristic as a baseline in future TAD benchmarks would be a natural next step.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper is a position paper arguing that current time-series anomaly detection (TAD) research overlooks several aspects that are important in industrial deployments. It states two core tenets: (1) the main applications of TAD are alerting and mitigation, and (2) whether an event is an anomaly is application-specific rather than intrinsic to the data. From these tenets it derives requirements for streaming algorithms and streaming evaluation, populations of time series, conditional anomalies, human-in-the-loop approaches, explainability, point-process preprocessing, periodicity detection, and threshold setting. The paper illustrates these requirements with a food-cooling example, formalizes streaming evaluation, defines conditional anomalies via Eq. (4), casts human-in-the-loop learning as a censored bandit problem, and reports a small synthetic benchmark on periodicity detection. It is explicitly intended as a call for action and a motivation for new datasets and research directions.
Significance. If the paper's perspective is accepted, it could help reorient TAD research toward problems that matter in practice. Its concrete contributions include the explicit streaming evaluation formulation, a definition of conditional anomalies, a human-in-the-loop formalization, and a simple periodicity-detection benchmark. The paper is non-circular and does not rely on self-citation; it honestly labels its second tenet as an assumption and engages with counterarguments in Section 6. However, the empirical basis is limited: the abstract's claim of an 'investigation of systems deployed in a cloud environment' is never substantiated, the central 'Application Specific' tenet is not empirically demonstrated, and the new periodicity benchmark is presented without variance or released code. These issues make the paper a useful but not yet fully rigorous call to action.
major comments (4)
- [Section 1, Tenet 'Application Specific'; Section 6.2] The paper derives its central new requirements (human-in-the-loop, conditional anomalies, side information) from the tenet that anomaly status is application-specific, but it explicitly labels this tenet an assumption and supports it only with illustrative examples (freezer door vs. power outage, holiday sales, flu vs. temperature). Section 6.2 admits that 'Rigorously proving our tenet might be difficult' and even contemplates the alternative that 'most of the information required to determine anomalies is indeed contained in the signal itself.' Because the urgency of the proposed research agenda depends directly on this tenet, the manuscript should either reframe it as a falsifiable hypothesis with a concrete empirical test (e.g., comparing signal-only anomaly labels against application-relevant labels on a suitable dataset) or explicitly mark the derived requirements as conditional on the assumption. In its current form, the abstract's claim that current definitions 'miss critical aspects' overstates the strength of the evidence.
- [Abstract and Section 1] The abstract states that the paper is 'Based on an investigation of systems deployed in a cloud environment,' but no such investigation is described anywhere in the manuscript. The only stated basis is that the author worked with internal and external customers at a cloud provider. This is an important gap because Tenet 1 (alerting) and the claimed practical prevalence of the listed requirements rest on this experience. The authors should either describe the investigation in sufficient detail (setting, data collection, methodology, how the conclusions were drawn) or remove the phrase from the abstract and instead clearly state that the claims are based on the author's industry experience.
- [Appendix A, Table 1; Section 5.1] The conclusion that existing periodicity-detection methods 'are outperformed by simple heuristics' rests on Table 1, which reports a single accuracy value per method on 1000 synthetic series, with no variance, no repeated runs, and no description of the random seed or the distribution of the generated periods. The 'Peaks' heuristic is new but its definition ('returns the first non-dominated peak') is underspecified, and no code or detailed protocol is provided for reproduction. Since this benchmark is one of the few concrete empirical contributions of the paper, it needs error bars or multiple trials, a precise pseudocode for 'Peaks,' and release of the exact data-generation code and parameter choices.
- [Section 3.2.2, Eq. (4)] The paper makes two strong claims about Eq. (4): that it is 'the most practically relevant question' and that 'We have not found a reference to the definition in Equation (4) in the literature.' Both are empirical/literature claims that require support. There is a substantial body of work on anomaly detection with exogenous variables, regression-residual-based anomaly detection, and contextual anomalies that should be engaged with to demonstrate that the conditional formulation is indeed missing or underappreciated. The authors should either survey this related work and justify the novelty claim, or soften the phrasing to say that the formulation is underused and deserves more attention.
minor comments (5)
- [Section 3.1, Eq. (1)] The notation A(x_{<t})_t is not explicitly defined; the text says the algorithm produces a binary sequence a, but the equation would be clearer if it stated that A(x_{<t})_t denotes the label assigned to x_t when the algorithm is trained only on x_1,...,x_{t-1}.
- [Section 4, Eq. (5)] The expression in Eq. (5) is a weighted loss or total cost, not a regret in the usual sense (the difference from an optimal policy). Consider renaming it 'cost' or defining it formally as regret relative to a policy class.
- [Section 5.1] The phrase 'fixed values of 4 for PCA window size' is unclear: 4 is presumably a window length in number of time points, but it should be stated explicitly and the relevant parameter of the PCA method should be identified.
- [Figures 2 and 4] The markers on the autocorrelation panels are small and the 'True' label is not explained in the caption; increase marker visibility and add a clear legend or a note that 'True' indicates the ground-truth periodicity in the ACF.
- [Throughout] The manuscript contains several typographical errors and misspellings, including 'Febuary' in the header, 'Periocity' in the introduction list, 'likelyhood' in Section 3.2.2, 'unfurtunately' in Section 5.3, 'obivous' in Section 5.2, and 'belief' in Section 7. A thorough proofreading pass is needed.
Circularity Check
No significant circularity: the paper is an opinion/perspective piece whose stated tenets are explicit assumptions, and its only quantitative experiment is evaluated against synthetic ground truth, not against its own fitted parameters.
full rationale
The paper makes no derivation that reduces to its own inputs. Its two central tenets, Alerting and Application Specific, are introduced explicitly as 'an empirical observation' and 'an assumption about the nature of anomalies relevant in practice,' and the paper itself canvasses alternatives in Section 6, including the admission in 6.2 that 'Rigorously proving our tenet might be difficult' and the possibility that 'most of the information required to determine anomalies is indeed contained in the signal itself.' Because the requirements (streaming, populations, conditional anomalies, human-in-the-loop) are presented as consequences of these stated assumptions rather than as empirical predictions claimed to be independently derived, this is an argument from premise, not circularity. The formal definitions in Eq. (1), (2), (4), and (5) are proposed problem formulations, not fit-derived predictions. The Appendix A periodicity experiment compares existing and simple heuristic methods against a synthetic benchmark with known ground-truth periodicity 'performing methods' outputs, so the comparison is external to the methods' own assumptions. There is no parameter fitting to a subset followed by prediction of a closely related quantity, no author self-citation chain carrying a load-bearing claim, and no renamed known result presented as a derivation. The under-investigation claims rest on cited prior benchmarks, but those citations are independent external work and not load-bearing in a circular sense. The paper is self-described as 'a motivation and call for action,' and its honesty about the unproven nature of its main tenet reinforces rather than undermines the non-circularity finding.
Assumptions & free parameters
assumptions (3)
- domain assumption Whether a certain event is considered an anomaly is not intrinsic to the data, but is application specific.
- domain assumption The main applications of time-series anomaly detection are alerting and mitigation.
- ad hoc to paper A time series can be described by a mixture of a regular Markov process and an arbitrary noise distribution.
Cite this review
Pith. "Pith review of Open Challenges in Time Series Anomaly Detection: An Industry Perspective." pith.science (2026). https://pith.science/paper/56HH2DE6
@misc{pith2026250205392,
author = {Pith},
title = {Pith review of: Open Challenges in Time Series Anomaly Detection: An Industry Perspective},
year = {2026},
howpublished = {\url{https://pith.science/paper/56HH2DE6}},
note = {Machine review of arXiv:2502.05392}
}
read the original abstract
Current research in time-series anomaly detection is using definitions that miss critical aspects of how anomaly detection is commonly used in practice. We list several areas that are of practical relevance and that we believe are either under-investigated or missing entirely from the current discourse. Based on an investigation of systems deployed in a cloud environment, we motivate the areas of streaming algorithms, human-in-the-loop scenarios, point processes, conditional anomalies and populations analysis of time series. This paper serves as a motivation and call for action, including opportunities for theoretical and applied research, as well as for building new dataset and benchmarks.
Figures
Reference graph
Works this paper leans on
-
[1]
Jacob D Abernethy, Kareem Amin, and Ruihao Zhu. 2016. Threshold bandits, with and without censored feedback. Advances In Neural Information Processing Systems 29 (2016)
work page 2016
-
[2]
Paul Boniol, John Paparrizos, Themis Palpanas, and Michael J Franklin. 2021. SAND: streaming subsequence anomaly detection. Proceedings of the VLDB Endowment 14, 10 (2021), 1717–1729
work page 2021
-
[3]
Ziquan Deng, Xiwei Xuan, Kwan-Liu Ma, and Zhaodan Kong. 2024. A Reliable Framework for Human-in-the-Loop Anomaly Detection in Time Series. arXiv preprint arXiv:2405.03234 (2024)
arXiv 2024
-
[4]
Mohamed G Elfeky, Walid G Aref, and Ahmed K Elmagarmid. 2005. Periodicity detection in time series databases. IEEE Transactions on Knowledge and Data Engineering 17, 7 (2005), 875–887
work page 2005
-
[5]
Arik Ermshaus, Patrick Schäfer, and Ulf Leser. 2023. ClaSP: parameter-free time series segmentation. Data Mining and Knowledge Discovery 37, 3 (2023), 1262–1300
work page 2023
-
[6]
Arik Ermshaus, Patrick Schäfer, and Ulf Leser. 2023. Window size selection in unsupervised time series analytics: A review and benchmark. In International Workshop on Advanced Analytics and Learning on Temporal Data . Springer, 83– 101
work page 2023
-
[7]
Onat Gungor, Amanda Rios, Priyanka Mudgal, Nilesh Ahuja, and Tajana Rosing
-
[8]
Waleed Hilal, S Andrew Gadsden, and John Yawney. 2022. Financial fraud: a review of anomaly detection techniques and recent advances. Expert systems With applications 193 (2022), 116429
work page 2022
Show all 43 references
-
[9]
Tao Huang, Silas Nøstvik, Peder Bacher, Jonas Kjær Jensen, Wiebke Brix Markussen, and Jan Kloppenborg Møller. 2023. Labelled dataset for Ultra-Low Temperature Freezer to aid dynamic modelling & fault detection and diagnostics. Scientific Data 10, 1 (2023), 888
2023
-
[10]
Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. 2018. Detecting spacecraft anomalies using lstms and nonpara- metric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD interna- tional conference on knowledge discovery & data...
2018
-
[11]
Shima Imani and Eamonn Keogh. 2021. Multi-window-finder: domain agnostic window size for time series data. Proceedings of the MileTS 21 (2021)
2021
-
[12]
Nicholas A James, Arun Kejariwal, and David S Matteson. 2016. Leveraging cloud data to mitigate user experience from ‘breaking bad’. In 2016 IEEE International Conference on Big Data (Big Data) . IEEE, 3499–3508
2016
-
[13]
Siwon Kim, Kukjin Choi, Hyun-Soo Choi, Byunghan Lee, and Sungroh Yoon. 2022. Towards a rigorous evaluation of time-series anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 36. 7194–7201
2022
-
[14]
Kwei-Herng Lai, Daochen Zha, Junjie Xu, Yue Zhao, Guanchu Wang, and Xia Hu. 2021. Revisiting time series outlier detection: Definitions and benchmarks. In Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 1)
2021
-
[15]
Nikolay Laptev, Saeed Amizadeh, and Ian Flint. 2015. Generic and scalable framework for automated time-series anomaly detection. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 1939–1947
2015
-
[16]
Alexander Lavin and Subutai Ahmad. 2015. Evaluating real-time anomaly detec- tion algorithms–the Numenta anomaly benchmark. In2015 IEEE 14th international conference on machine learning and applications (ICMLA) . IEEE, 38–44
2015
-
[17]
Dapeng Liu, Youjian Zhao, Haowen Xu, Yongqian Sun, Dan Pei, Jiao Luo, Xi- aowei Jing, and Mei Feng. 2015. Opprentice: Towards practical and automatic anomaly detection through machine learning. In Proceedings of the 2015 internet measurement conference. 211–224
2015
-
[18]
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2008. Isolation forest. In 2008 eighth ieee international conference on data mining . IEEE, 413–422
2008
-
[19]
Qinghua Liu, Paul Boniol, Themis Palpanas, and John Paparrizos. 2024. Time- Series Anomaly Detection: Overview and New Trends. Proceedings of the VLDB Endowment (PVLDB) 17, 12 (2024), 4229–4232
2024
-
[20]
Qinghua Liu and John Paparrizos. 2024. The Elephant in the Room: Towards A Re- liable Time-Series Anomaly Detection Benchmark. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track
2024
-
[21]
Nesryne Mejri, Laura Lopez-Fuentes, Kankana Roy, Pavel Chernakov, Enjie Ghor- bel, and Djamila Aouada. 2024. Unsupervised anomaly detection in time-series: An extensive evaluation and analysis of state-of-the-art methods. Expert Systems with Applications (2024), 124922
2024
-
[22]
Alan V Oppenheim. 1999. Discrete-time signal processing . Pearson Education India
1999
-
[23]
John Paparrizos, Paul Boniol, Themis Palpanas, Ruey S Tsay, Aaron Elmore, and Michael J Franklin. 2022. Volume under the surface: a new accuracy evaluation measure for time-series anomaly detection. Proceedings of the VLDB Endowment 15, 11 (2022), 2774–2787
2022
-
[24]
John Paparrizos, Yuhao Kang, Paul Boniol, Ruey S Tsay, Themis Palpanas, and Michael J Franklin. 2022. TSB-UAD: an end-to-end benchmark suite for univariate Open Challenges in Time Series Anomaly Detection: An Industry Perspective Preprint, Febuary 07, 2025, Redmond, USA time-s...
2022
-
[25]
Hansheng Ren, Bixiong Xu, Yujing Wang, Chao Yi, Congrui Huang, Xiaoyu Kou, Tony Xing, Mao Yang, Jie Tong, and Qi Zhang. 2019. Time-series anomaly detec- tion service at microsoft. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mini...
2019
-
[26]
Peter J Rousseeuw and Katrien Van Driessen. 1999. A fast algorithm for the minimum covariance determinant estimator. Technometrics 41, 3 (1999), 212–223
1999
-
[27]
M Saquib Sarfraz, Mei-Yen Chen, Lukas Layer, Kunyu Peng, and Marios Koulakis
-
[28]
Sebastian Schmidl, Phillip Wenig, and Thorsten Papenbrock. 2022. Anomaly detection in time series: a comprehensive evaluation. Proceedings of the VLDB Endowment 15, 9 (2022), 1779–1797
2022
-
[29]
Jacopo Soldani and Antonio Brogi. 2022. Anomaly detection and failure root cause analysis in (micro) service-based cloud applications: A survey. ACM Computing Surveys (CSUR) 55, 3 (2022), 1–39
2022
-
[30]
Sondre Sørbø and Massimiliano Ruocco. 2024. Navigating the metric maze: A taxonomy of evaluation metrics for anomaly detection in time series.Data Mining and Knowledge Discovery 38, 3 (2024), 1027–1068
2024
-
[31]
Emmanouil Sylligardos, Paul Boniol, John Paparrizos, Panos Trahanias, and Themis Palpanas. 2023. Choose wisely: An extensive evaluation of model selection for anomaly detection in time series. Proceedings of the VLDB Endowment 16, 11 (2023), 3418–3432
2023
-
[32]
Nesime Tatbul, Tae Jun Lee, Stan Zdonik, Mejbah Alam, and Justin Gottschlich
-
[33]
Chee-Wooi Ten, Junho Hong, and Chen-Ching Liu. 2011. Anomaly detection for cybersecurity of the substations. IEEE Transactions on Smart Grid 2, 4 (2011), 865–873
2011
-
[34]
Owen Vallis, Jordan Hochenbaum, and Arun Kejariwal. 2014. A novel technique for{Long-Term} anomaly detection in the cloud. In 6th USENIX workshop on hot topics in cloud computing (HotCloud 14)
2014
-
[35]
Michail Vlachos, Philip Yu, and Vittorio Castelli. 2005. On periodicity detection and structural periodic similarity. In Proceedings of the 2005 SIAM international conference on data mining . SIAM, 449–460
2005
-
[36]
Qingsong Wen, Kai He, Liang Sun, Yingying Zhang, Min Ke, and Huan Xu. 2021. RobustPeriod: Robust time-frequency mining for multiple periodicity detection. In Proceedings of the 2021 international conference on management of data . 2328– 2337
2021
-
[37]
Renjie Wu and Eamonn J Keogh. 2021. Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress.IEEE transactions on knowledge and data engineering 35, 3 (2021), 2421–2429
2021
-
[38]
Wenzhuo Yang, Kun Zhang, and Steven CH Hoi. 2022. A causal approach to detecting multivariate time-series anomalies and root causes. arXiv preprint arXiv:2206.15033 (2022)
2022 arXiv
-
[39]
Zhun Jerry Yu, Fariborz Haghighat, and Benjamin CM Fung. 2016. Advances and challenges in building engineering and data mining applications for energy- efficient communities. Sustainable Cities and Society 25 (2016), 33–38
2016
-
[40]
Jiuqi Elise Zhang, Di Wu, and Benoit Boulet. 2021. Time series anomaly detection for smart grids: A survey. In 2021 IEEE electrical power and energy conference (EPEC). IEEE, 125–130. Algorithm Accuracy Avg Runtime (s) Random 0.007 - SuSS [5] 0.008 0.0113 MWF [11] 0.074 0.0441 ...
2021
-
[2018]
31 (2018)
Precision and Recall for Time Series. 31 (2018). https://proceedings.neurips. cc/paper_files/paper/2018/file/8f468c873a32bb0619eaeb2050ba45d1-Paper.pdf
2018
-
[2024]
In Forty-first International Conference on Machine Learning
Position: Quo Vadis, Unsupervised Time Series Anomaly Detection?. In Forty-first International Conference on Machine Learning
-
[2025]
In International Conference on Pattern Recognition
A Robust Framework for Evaluation of Unsupervised Time-Series Anomaly Detection. In International Conference on Pattern Recognition . Springer, 48–64
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.