REVIEW 4 major objections 7 minor 30 references
Exploring the impact of Optimised Hyperparameters on Bi-LSTM-based Contextual Anomaly Detector
T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read UoCAD-OH claims that automatically tuned hyperparameters let an unsupervised Bi-LSTM sliding-window detector reach 0.97 F1 on smart-home contextual anomalies.
desk verdict Max Trials = 1 means the tuner never tuned: the paper's central claim is unsupported, though the 10d2a dataset is a small useful artifact. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a sliding-window Bi-LSTM: each window is read in both temporal directions, the model predicts the next instance, and the prediction error is converted into a feature-wise Average Absolute Relative Error (AARE). A dynamic threshold of three standard deviations above the running mean of historical AARE values decides whether an instance is anomalous. UoCAD-OH adds a Hyperband-based automatic hyperparameter search over input units, activation, learning rate, optimiser, number of layers, and dropout rate, run offline on a larger unlabeled dataset; the winning configuration (160 units, ReLU, learning rate 1e-4, Adam, 2 layers, dropout 0.2) then runs online with window sizes 6 to 144 and individual or majority voting across nine features. The event-level scoring rule, where one flagged instance labels the whole anomalous segment as detected, is also part of the evaluation machinery.
What would settle it
Re-run the evaluation on the 2d1a and 10d2a datasets scoring each instance independently instead of counting a whole anomalous segment as a true positive when one of its instances is flagged; if per-instance Recall falls well below 1.0, the headline F1 scores depend on the aggregation rule.
Extended reading notes
Core claim
The paper's central claim is that UoCAD-OH, which adds offline automatic hyperparameter optimisation to the UoCAD pipeline, can detect contextual anomalies in multivariate smart-home time series with high event-level precision and recall. Using eight window sizes (6 to 144) and two voting criteria (individual and majority), the tuned Bi-LSTM achieved its best result on the 2d1a dataset under the individual criterion: Precision 0.95, Recall 1.0, and F1 0.97 at window size 24. On the 10d2a dataset, window sizes 24 and 48 were the only ones that detected both anomalies; the majority criterion failed for 2d1a and detected at most one anomaly on 10d2a. The author's interpretation is that window sizes 24 and 48 are most effective, that the upper useful window size for 2d1a is around 96, and that individual-feature voting outperforms majority voting.
Load-bearing premise
The reported scores count an anomalous segment as detected if any single instance in it is flagged, so Recall 1.0 reflects event-level detection, not per-instance classification.
Editorial extensions
If this is right
- If UoCAD-OH is right, automatic hyperparameter tuning can substitute for manual fixing in UoCAD while keeping the same sliding-window Bi-LSTM pipeline.
- Window sizes 24 and 48 are the most effective for these smart-home anomalies, and sizes above 96 stop working on the two-day dataset.
- The majority criterion (five of nine features agreeing) is generally ineffective, so individual-feature voting is the better operating point for this method.
- The same unintended-cooking anomaly is detected at similar quality in both datasets, suggesting the tuned model transfers across time periods in the same environment.
Reading between the lines
- Editorial inference: the event-level scoring rule makes the reported recall optimistic relative to per-instance expectations; per-instance precision-recall would be a stricter test and the paper does not report it.
- Editorial inference: because the tuning set and test sets come from the same room and sensor, the tuned hyperparameters may be environment-specific; testing on another home's air-quality series would show how much of the gain transfers.
- Editorial inference: the best window sizes (24 and 48) are close to the anomaly durations (24 to 28 instances), which suggests window size should be matched to expected anomaly length or made adaptive, a connection the paper does not draw.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes UoCAD-OH, an extension of the earlier UoCAD contextual anomaly detector for smart-home time series. The claimed contribution is an offline hyperparameter-optimization phase that tunes a Bi-LSTM model on a large unlabeled 5-month dataset using Keras Tuner's Hyperband, after which the tuned hyperparameters are used in an online detection phase on two smaller labeled datasets (2d1a and 10d2a). Detection performance is reported as Precision, Recall, and F1 for eight sliding-window sizes under 'individual' and 'majority' feature criteria. The headline reported result is an F1 of 0.97 for the 2d1a dataset under the individual criterion with window size 24.
Significance. If the claims were fully supported, the paper would demonstrate a practical recipe for automatically tuning a Bi-LSTM-based online anomaly detector on unlabeled data, with clear implications for deployment in smart-home monitoring. The use of real-world datasets with naturally embedded contextual anomalies is a strength, as is the transparent reporting of the search space and tuner configuration. However, the central empirical claim is not currently supported: Table IV configures the tuner with Max Trials=1, so no meaningful optimization occurs, and Table VII reports no UoCAD baseline, so the 'impact' of optimization is not demonstrated. The event-level true-positive counting rule further inflates recall in a way that is not clearly disclosed in the headline numbers.
major comments (4)
- [IV-B, Table IV] The tuner is configured with Max Trials=1 and Execution per Trial=1. In Keras Tuner's Hyperband, max_trials is the total number of hyperparameter configurations evaluated; with max_trials=1, only a single configuration is drawn and trained. The 'optimised hyperparameters' in Table VI are therefore not the result of an optimization process but a single sample from the search space. The text in Section IV-B even acknowledges that 'running multiple trials can sometimes provide more optimal results,' which makes the configuration choice internally inconsistent with the paper's stated goal. This is load-bearing: the independent variable of the study—'optimised hyperparameters'—may not actually have been manipulated.
- [IV-E, Table VII] The title and introduction claim the paper explores the impact of optimised hyperparameters on UoCAD, yet Table VII reports results only for UoCAD-OH. No Precision, Recall, or F1 values for the original UoCAD method are given for either dataset or any window size, even though the experimental setup section says the study uses the same eight window sizes 'to ensure consistency with UoCAD's experiments and allow for a fair evaluation.' Without the UoCAD baseline under identical evaluation rules, the reader cannot assess whether the tuned hyperparameters improved, degraded, or left unchanged the detection performance. This missing comparison is central to the paper's main claim.
- [IV-C, Evaluation Metrics] The event-level true-positive rule—'if p is the total number of anomalous instances and any instance i belongs to p, the whole p is considered as true positive'—converts a single flagged instance into a fully detected anomalous segment. Since each anomalous segment spans 24 to 28 instances, this rule makes Recall=1.0 achievable with very sparse detections and is therefore load-bearing for the reported high F1 values. The paper should report per-instance precision and recall alongside the segment-level numbers, or at minimum state clearly in the abstract and results that all reported recall values use this event-level definition.
- [IV-E, Results and Discussion] The discussion of window-size effects is inconsistent with Table VII. The text states that 'Ind-120 and Ind-144 are the only combinations for the 2d1a dataset that were least successful' and later concludes that 'the upper limit of window size for the 2d1a dataset is 96.' However, Table VII shows Ind-144 with Precision 0.81, Recall 1.0, and F1 0.89—a substantially better result than Ind-72 (F1 0.59) and comparable to Ind-48. Only Ind-120 yields all-zero scores. The conclusion that window sizes are effective only up to 96 is therefore not supported by the table, and the non-monotonic pattern of F1 across window sizes (0.94, 0.96, 0.97, 0.89, 0.59, 0.61, 0.00, 0.89 for Ind-6 through Ind-144) is not addressed. This undermines the claim made again in the conclusions that 'UoCAD-OH is capable of detecting anomalies for window sizes between 6 and 96.'
minor comments (7)
- [Abstract] There are repeated spelling errors in key terms: 'hyperparamteres' appears in the abstract, and 'Hyperparamnters' appears in the title of the abstract and in Section I. These should be corrected to 'hyperparameters' throughout.
- [IV-B] The tuner setup text says the option 'Max retires per Trial' and later 'Max retries per Trial' is described as enabling 'the tuner to retry the trial in case it crashes or produces invalid results.' The first phrase is a typo, and the explanation is vague; please specify what invalid results mean and how retries affect the objective.
- [IV-E] In the sentence 'Hence, we can deduce that the upper limit of window size for the 2d1a dataset is 96 for UoCAD=OH,' the 'UoCAD=OH' is a typo and should be 'UoCAD-OH.'
- [Table II] Table II uses a comma as the decimal separator in the Humidity standard deviation ('9,50') while all other tables and rows use a period. Please standardize the decimal notation.
- [References] Reference [27] gives the URL as 'view-source:https://www.airthings.com/en/view-plus,' which is not a proper citation URL. Please provide the stable page URL or a DOI if available.
- [IV-E, Figures 1 and 2] Figures 1 and 2 are referenced in the text but their content is not described in the caption or body. Please clarify what is shown (e.g., predicted anomaly windows over time vs. ground truth) so the reader can interpret the visualizations.
- [IV-C] The notation in the event-level TP rule is confusing: 'if p is the total number of anomalous instances and any instance i belongs to p, the whole p is considered as true positive' uses p both as a count and as a set of instances. Please rephrase with distinct symbols, such as 'let S be the set of anomalous instances in a segment and let s=|S|...'.
Circularity Check
No significant circularity: evaluation is on disjoint data and the tuning procedure, though flawed, is not fitted to the test labels.
full rationale
The reported F1 values are not equivalent by construction to the hyperparameter-tuning input. The Keras Tuner optimisation (Section IV-B) is run on a separate unlabeled 5M dataset that does not overlap with the 2d1a and 10d2a evaluation periods, so the hyperparameters in Table VI are not fitted to the test data. The AARE and threshold equations (Eqs. 1-2) and the event-level TP rule (Section IV-C) are inherited from the authors' prior UoCAD paper [5], but they are adopted explicitly as the baseline method rather than smuggled in via citation; no uniqueness theorem or ansatz is imported to force the current result. The event-level TP rule is disclosed as an assumption and affects the metric level, but it is an evaluation convention, not a derivation step that makes the model output identical to its input. The Max Trials=1 setting in Table IV is a serious threat to the claim that the hyperparameters were actually optimised, and the absence of a UoCAD comparison means the 'impact' of tuning is not demonstrated; however, those are experimental-validity concerns, not circularity. We therefore find no circular step and assign score 0.
Assumptions & free parameters
free parameters (9)
- Input units (Bi-LSTM layer size) =
160
- Learning rate =
0.0001
- Number of layers =
2
- Dropout rate =
0.2
- Activation function =
ReLU
- Optimizer =
Adam
- Window size =
6, 12, 24, 48, 72, 96, 120, 144
- Threshold multiplier in Eq. 2 =
3
- Tuner setup: batch size, epochs, max trials =
100, 50, 1
assumptions (6)
- domain assumption AARE reconstruction error of the Bi-LSTM captures contextual anomalies
- domain assumption Feature-wise errors are approximately normal, so mean plus 3 sigma is a valid threshold
- domain assumption An anomalous segment is fully detected if any instance in it is flagged
- domain assumption Hyperparameters tuned on the unlabeled 5M dataset transfer to the 2d1a and 10d2a periods
- domain assumption The 5M dataset represents normal smart home conditions
- ad hoc to paper Deep learning training is deterministic enough that a single trial suffices
Cite this review
Pith. "Pith review of Exploring the impact of Optimised Hyperparameters on Bi-LSTM-based Contextual Anomaly Detector." pith.science (2026). https://pith.science/paper/XOQIII6S
@misc{pith2026250115053,
author = {Pith},
title = {Pith review of: Exploring the impact of Optimised Hyperparameters on Bi-LSTM-based Contextual Anomaly Detector},
year = {2026},
howpublished = {\url{https://pith.science/paper/XOQIII6S}},
note = {Machine review of arXiv:2501.15053}
}
read the original abstract
The exponential growth in the usage of Internet of Things in daily life has caused immense increase in the generation of time series data. Smart homes is one such domain where bulk of data is being generated and anomaly detection is one of the many challenges addressed by researchers in recent years. Contextual anomaly is a kind of anomaly that may show deviation from the normal pattern like point or sequence anomalies, but it also requires prior knowledge about the data domain and the actions that caused the deviation. Recent studies based on Recurrent Neural Networks (RNN) have demonstrated strong performance in anomaly detection. This study explores the impact of automatically tuned hyperparamteres on Unsupervised Online Contextual Anomaly Detection (UoCAD) approach by proposing UoCAD with Optimised Hyperparamnters (UoCAD-OH). UoCAD-OH conducts hyperparameter optimisation on Bi-LSTM model in an offline phase and uses the fine-tuned hyperparameters to detect anomalies during the online phase. The experiments involve evaluating the proposed framework on two smart home air quality datasets containing contextual anomalies. The evaluation metrics used are Precision, Recall, and F1 score.
Figures
Reference graph
Works this paper leans on
-
[1]
Anomaly detection in smart environments: A comprehensive survey,
D. F ¨ahrmann, L. Mart´ın, L. S´anchez, and N. Damer, “Anomaly detection in smart environments: A comprehensive survey,” IEEE Access, 2024
work page 2024
-
[2]
A systematic literature review of iot time series anomaly detection solutions,
A. Sgueglia, A. Di Sorbo, C. A. Visaggio, and G. Canfora, “A systematic literature review of iot time series anomaly detection solutions,” Future Generation Computer Systems , vol. 134, pp. 170–186, 2022
work page 2022
-
[3]
Unsupervised anomaly detection in time-series: An exten- sive evaluation and analysis of state-of-the-art methods,
N. Mejri, L. Lopez-Fuentes, K. Roy, P. Chernakov, E. Ghorbel, and D. Aouada, “Unsupervised anomaly detection in time-series: An exten- sive evaluation and analysis of state-of-the-art methods,” Expert Systems with Applications, p. 124922, 2024
2024
-
[4]
M. A. Belay, S. S. Blakseth, A. Rasheed, and P. Salvo Rossi, “Unsuper- vised anomaly detection for iot-based multivariate time series: Existing solutions, performance analysis and future directions,” Sensors, vol. 23, no. 5, p. 2844, 2023
work page 2023
-
[5]
A. A. Toor, J.-C. Lin, E. G. Gran, and M.-C. Lee, “UoCAD: An unsu- pervised online contextual anomaly detection approach for multivariate time series from smart homes,” in IoTBDS 2024: Proceedings of the 9th International Conference on Internet of Things, Big Data and Security, April 28-30, 2024, Angers, France . SciTePress, 2024
work page 2024
-
[6]
Real-time deep anomaly detection framework for multivariate time-series data in industrial iot,
H. Nizam, S. Zafar, Z. Lv, F. Wang, and X. Hu, “Real-time deep anomaly detection framework for multivariate time-series data in industrial iot,” IEEE Sensors Journal , vol. 22, no. 23, pp. 22 836–22 849, 2022
work page 2022
-
[7]
T. Nguyen-Da, P. Nguyen-Thanh, and M.-Y . Cho, “Real-time aiot anomaly detection for industrial diesel generator based an efficient deep learning cnn-lstm in industry 4.0,” Internet of Things, vol. 27, p. 101280, 2024
work page 2024
-
[8]
X. Li, W. Niu, X. Zhang, R. Zhang, Z. Yu, and Z. Li, “Improving performance of log anomaly detection with semantic and time features based on bilstm-attention,” in 2021 2nd International Conference on Electronics, Communications and Information Technology (CECIT) . IEEE, 2021, pp. 661–666
work page 2021
Show all 30 references
-
[9]
Contextual anomaly detection for multivariate time series data,
H. Kim and H. Kim, “Contextual anomaly detection for multivariate time series data,” Quality Engineering, vol. 35, no. 4, pp. 686–695, 2023
2023
-
[10]
Time series forecasting and point anomaly detection of sensor signals using lstm neural network architectures,
A. Bhoomika, S. N. S. Chitta, K. Laxmisetti, and B. Sirisha, “Time series forecasting and point anomaly detection of sensor signals using lstm neural network architectures,” in 2023 10th International Conference on Computing for Sustainable Global Development (INDIACom) , 2023...
2023
-
[11]
A hybrid cnn-lstm based approach for anomaly detection systems in sdns,
M. Abdallah, N. An Le Khac, H. Jahromi, and A. Delia Jurcut, “A hybrid cnn-lstm based approach for anomaly detection systems in sdns,” in Proceedings of the 16th International Conference on Availability, Reliability and Security , 2021, pp. 1–7
2021
-
[12]
Intelligent bilstm-attention-ibpnn method for anomaly detec- tion in financial auditing,
S. Wang, “Intelligent bilstm-attention-ibpnn method for anomaly detec- tion in financial auditing,” IEEE Access, 2024
2024
-
[13]
Deep learning for anomaly detection in multivariate time series: Approaches, applications, and challenges,
G. Li and J. J. Jung, “Deep learning for anomaly detection in multivariate time series: Approaches, applications, and challenges,” Information Fusion, vol. 91, pp. 93–102, 2023
2023
-
[14]
Performance anomaly detection in web services: an rnn-based approach using dy- namic quality of service features,
M. Hasnain, S. R. Jeong, M. F. Pasha, and I. Ghani, “Performance anomaly detection in web services: an rnn-based approach using dy- namic quality of service features,” Computers, Materials and Continua , vol. 64, no. 2, pp. 729–752, 2020
2020
-
[15]
Multi-head cnn– rnn for multi-time series anomaly detection: An industrial case study,
M. Canizo, I. Triguero, A. Conde, and E. Onieva, “Multi-head cnn– rnn for multi-time series anomaly detection: An industrial case study,” Neurocomputing, vol. 363, pp. 246–260, 2019
2019
-
[16]
Contextual anomaly detection on time series: A case study of metro ridership analysis,
K. Pasini, M. Khouadjia, A. Sam ´e, M. Tr ´epanier, and L. Oukhellou, “Contextual anomaly detection on time series: A case study of metro ridership analysis,” Neural Computing and Applications, pp. 1–25, 2022
2022
-
[17]
Salad: Self-adaptive lightweight anomaly detection for real-time recurrent time series,
M.-C. Lee, J.-C. Lin, and E. G. Gran, “Salad: Self-adaptive lightweight anomaly detection for real-time recurrent time series,” in2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMP- SAC). IEEE, 2021, pp. 344–349
2021
-
[18]
Repad2: Real-time, lightweight, and adap- tive anomaly detection for open-ended time series,
M.-C. Lee and J.-C. Lin, “Repad2: Real-time, lightweight, and adap- tive anomaly detection for open-ended time series,” arXiv preprint arXiv:2303.00409, 2023
2023 arXiv
-
[19]
Rola: A real-time online lightweight anomaly detection system for multivariate time series,
——, “Rola: A real-time online lightweight anomaly detection system for multivariate time series,” arXiv preprint arXiv:2305.16509 , 2023
2023 arXiv
-
[20]
Impact of recurrent neural networks and deep learning frameworks on real-time lightweight time series anomaly detection,
M.-C. Lee, J.-C. Lin, and S. Katsikas, “Impact of recurrent neural networks and deep learning frameworks on real-time lightweight time series anomaly detection,” arXiv preprint arXiv:2407.18439 , 2024
2024 arXiv
-
[21]
Contextual anomaly detection framework for big sensor data,
M. A. Hayes and M. A. Capretz, “Contextual anomaly detection framework for big sensor data,” Journal of Big Data , vol. 2, no. 1, pp. 1–22, 2015
2015
-
[22]
Time series contextual anomaly detection for detecting market manipulation in stock market,
K. Golmohammadi and O. R. Zaiane, “Time series contextual anomaly detection for detecting market manipulation in stock market,” in 2015 IEEE international conference on data science and advanced analytics (DSAA). IEEE, 2015, pp. 1–10
2015
-
[23]
Contextual anomaly detection for cyber-physical security in smart grids based on an artificial neural network model,
A. M. Kosek, “Contextual anomaly detection for cyber-physical security in smart grids based on an artificial neural network model,” in 2016 Joint Workshop on Cyber-Physical Security and Resilience in Smart Grids (CPSR-SG). IEEE, 2016, pp. 1–6
2016
-
[24]
Early anomaly detection in smart home: A causal association rule-based approach,
S. Hela, B. Amel, and R. Badran, “Early anomaly detection in smart home: A causal association rule-based approach,” Artificial intelligence in medicine, vol. 91, pp. 57–71, 2018
2018
-
[25]
An adaptive method based on contextual anomaly detection in internet of things through wireless sensor networks,
X. Yu, H. Lu, X. Yang, Y . Chen, H. Song, J. Li, and W. Shi, “An adaptive method based on contextual anomaly detection in internet of things through wireless sensor networks,” International Journal of Distributed Sensor Networks, vol. 16, no. 5, p. 1550147720920478, 2020
2020
-
[26]
Wisdom of the contexts: active ensemble learning for contextual anomaly detection,
E. Calikus, S. Nowaczyk, M.-R. Bouguelia, and O. Dikmen, “Wisdom of the contexts: active ensemble learning for contextual anomaly detection,” Data Mining and Knowledge Discovery , vol. 36, no. 6, pp. 2410–2458, 2022
2022
-
[27]
View plus - smart indoor air quality monitor,
AirThings, “View plus - smart indoor air quality monitor,” 2024. [On- line]. Available: view-source:https://www.airthings.com/en/view-plus
2024
-
[28]
Keras Tuner,
T. O’Malley, E. Bursztein, J. Long, F. Chollet, H. Jin, L. Invernizzi et al., “Keras Tuner,” https://github.com/keras-team/keras-tuner, 2019
2019
-
[29]
Tensorflow: Large-scale machine learning on heterogeneous distributed systems,
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin et al. , “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,” arXiv preprint arXiv:1603.04467, 2016
2016 arXiv
-
[30]
Keras: deep learning library for theano and tensorflow. 2015,
F. Chollet et al., “Keras: deep learning library for theano and tensorflow. 2015,” 2015
2015
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.