{"id":"53e172c3-7503-4ac2-bf74-6805635b9f5f","arxiv_id":"2501.15053","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"UoCAD-OH, a Bi-LSTM anomaly detector with Keras Tuner hyperparameters, reports F1 scores up to 0.97 on smart home air quality data, but lacks a baseline comparison to the original UoCAD.","lead":"This paper adds automatic hyperparameter tuning to UoCAD, an existing Bi-LSTM anomaly detector for smart home air quality data, and reports detection results on two datasets. It does not compare against the original UoCAD, so the claimed impact of tuning is not demonstrated.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"With Max Trials set to 1 (Table IV), the Keras Tuner Hyperband search evaluates at most one hyperparameter configuration, so the reported 'optimised' hyperparameters are not actually optimised; the central claim is unsupported.","rationale":"The reader's weakest_assumption is the event-level true-positive counting rule, which is indeed a valid concern: it makes Recall=1.0 nearly trivial for anomaly spans of 24-28 instances and inflates the headline F1. My stress-test, however, identifies a more fundamental problem: the hyperparameter optimisation procedure itself is configured with max_trials=1, meaning Keras Tuner evaluates exactly one configuration. A single configuration drawn from a large search space is not 'optimised' in any meaningful sense, so the paper's central independent variable is never established. The missing UoCAD baseline further prevents any statement about the 'impact' of tuning, since no comparison is made to the original fixed-hyperparameter detector. These issues compound: even if the event-level metric were replaced with per-instance scoring, the paper would still not demonstrate that optimised hyperparameters improve detection, because there is no evidence that optimisation occurred and no baseline against which to measure improvement. The reader's REJECT verdict remains appropriate.","tokens_in":9595,"tokens_out":5231,"duration_ms":50900,"concrete_test":"Instrument the Keras Tuner Hyperband search on the 5M dataset to log the total number of trials and each trial's validation loss, then re-run with max_trials set to e.g. 50. If the original run logged only one trial, or if the reported hyperparameters do not outperform a random sample from the same search space, the claim of optimised hyperparameters is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition for the title claim is that UoCAD-OH's hyperparameters are the result of an optimisation process. Section IV-B and Table IV set 'Max Trials' to 1 and 'Execution per Trial' to 1. In Keras Tuner's Hyperband, max_trials is the total number of configurations evaluated; with max_trials=1, only one configuration is trained, so the reported settings (160 units, ReLU, Adam, etc.) are a single draw from the search space, not a tuned optimum. The text even acknowledges that 'running multiple trials can sometimes provide more optimal results' but the setup prevents this. Additionally, no UoCAD baseline results are reported in Table VII or elsewhere, so even if the hyperparameters were tuned, the 'impact' of tuning is not demonstrated. The event-level TP rule in Section IV-C further inflates the headline F1, but the Max Trials issue is more fundamental: the independent variable of the study may never have been manipulated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes UoCAD-OH, an extension of the earlier UoCAD contextual anomaly detector for smart-home time series. The claimed contribution is an offline hyperparameter-optimization phase that tunes a Bi-LSTM model on a large unlabeled 5-month dataset using Keras Tuner's Hyperband, after which the tuned hyperparameters are used in an online detection phase on two smaller labeled datasets (2d1a and 10d2a). Detection performance is reported as Precision, Recall, and F1 for eight sliding-window sizes under 'individual' and 'majority' feature criteria. The headline reported result is an F1 of 0.97 for the 2d1a dataset under the individual criterion with window size 24.","tokens_in":9770,"tokens_out":3157,"duration_ms":28985,"significance":"If the claims were fully supported, the paper would demonstrate a practical recipe for automatically tuning a Bi-LSTM-based online anomaly detector on unlabeled data, with clear implications for deployment in smart-home monitoring. The use of real-world datasets with naturally embedded contextual anomalies is a strength, as is the transparent reporting of the search space and tuner configuration. However, the central empirical claim is not currently supported: Table IV configures the tuner with Max Trials=1, so no meaningful optimization occurs, and Table VII reports no UoCAD baseline, so the 'impact' of optimization is not demonstrated. The event-level true-positive counting rule further inflates recall in a way that is not clearly disclosed in the headline numbers.","major_comments":[{"comment":"The tuner is configured with Max Trials=1 and Execution per Trial=1. In Keras Tuner's Hyperband, max_trials is the total number of hyperparameter configurations evaluated; with max_trials=1, only a single configuration is drawn and trained. The 'optimised hyperparameters' in Table VI are therefore not the result of an optimization process but a single sample from the search space. The text in Section IV-B even acknowledges that 'running multiple trials can sometimes provide more optimal results,' which makes the configuration choice internally inconsistent with the paper's stated goal. This is load-bearing: the independent variable of the study—'optimised hyperparameters'—may not actually have been manipulated.","section":"IV-B, Table IV"},{"comment":"The title and introduction claim the paper explores the impact of optimised hyperparameters on UoCAD, yet Table VII reports results only for UoCAD-OH. No Precision, Recall, or F1 values for the original UoCAD method are given for either dataset or any window size, even though the experimental setup section says the study uses the same eight window sizes 'to ensure consistency with UoCAD's experiments and allow for a fair evaluation.' Without the UoCAD baseline under identical evaluation rules, the reader cannot assess whether the tuned hyperparameters improved, degraded, or left unchanged the detection performance. This missing comparison is central to the paper's main claim.","section":"IV-E, Table VII"},{"comment":"The event-level true-positive rule—'if p is the total number of anomalous instances and any instance i belongs to p, the whole p is considered as true positive'—converts a single flagged instance into a fully detected anomalous segment. Since each anomalous segment spans 24 to 28 instances, this rule makes Recall=1.0 achievable with very sparse detections and is therefore load-bearing for the reported high F1 values. The paper should report per-instance precision and recall alongside the segment-level numbers, or at minimum state clearly in the abstract and results that all reported recall values use this event-level definition.","section":"IV-C, Evaluation Metrics"},{"comment":"The discussion of window-size effects is inconsistent with Table VII. The text states that 'Ind-120 and Ind-144 are the only combinations for the 2d1a dataset that were least successful' and later concludes that 'the upper limit of window size for the 2d1a dataset is 96.' However, Table VII shows Ind-144 with Precision 0.81, Recall 1.0, and F1 0.89—a substantially better result than Ind-72 (F1 0.59) and comparable to Ind-48. Only Ind-120 yields all-zero scores. The conclusion that window sizes are effective only up to 96 is therefore not supported by the table, and the non-monotonic pattern of F1 across window sizes (0.94, 0.96, 0.97, 0.89, 0.59, 0.61, 0.00, 0.89 for Ind-6 through Ind-144) is not addressed. This undermines the claim made again in the conclusions that 'UoCAD-OH is capable of detecting anomalies for window sizes between 6 and 96.'","section":"IV-E, Results and Discussion"}],"minor_comments":[{"comment":"There are repeated spelling errors in key terms: 'hyperparamteres' appears in the abstract, and 'Hyperparamnters' appears in the title of the abstract and in Section I. These should be corrected to 'hyperparameters' throughout.","section":"Abstract"},{"comment":"The tuner setup text says the option 'Max retires per Trial' and later 'Max retries per Trial' is described as enabling 'the tuner to retry the trial in case it crashes or produces invalid results.' The first phrase is a typo, and the explanation is vague; please specify what invalid results mean and how retries affect the objective.","section":"IV-B"},{"comment":"In the sentence 'Hence, we can deduce that the upper limit of window size for the 2d1a dataset is 96 for UoCAD=OH,' the 'UoCAD=OH' is a typo and should be 'UoCAD-OH.'","section":"IV-E"},{"comment":"Table II uses a comma as the decimal separator in the Humidity standard deviation ('9,50') while all other tables and rows use a period. Please standardize the decimal notation.","section":"Table II"},{"comment":"Reference [27] gives the URL as 'view-source:https://www.airthings.com/en/view-plus,' which is not a proper citation URL. Please provide the stable page URL or a DOI if available.","section":"References"},{"comment":"Figures 1 and 2 are referenced in the text but their content is not described in the caption or body. Please clarify what is shown (e.g., predicted anomaly windows over time vs. ground truth) so the reader can interpret the visualizations.","section":"IV-E, Figures 1 and 2"},{"comment":"The notation in the event-level TP rule is confusing: 'if p is the total number of anomalous instances and any instance i belongs to p, the whole p is considered as true positive' uses p both as a count and as a set of instances. Please rephrase with distinct symbols, such as 'let S be the set of anomalous instances in a segment and let s=|S|...'.","section":"IV-C"}],"recommendation":"reject","confidential_remarks":"The Max Trials=1 configuration in Table IV is the kind of issue that could be a simple copy-paste mistake, but in its current form it invalidates the core contribution: there is no evidence that any optimization actually happened. Even if that were corrected, the missing UoCAD baseline would require adding a full set of comparative experiments, and the event-level evaluation rule needs to be reported clearly and separately. These are substantial changes rather than local revisions. The paper may be resubmitted as a new study if the experiments are redone with a real search budget and a UoCAD comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper fails on its own premise. The title promises 'optimised hyperparameters,' but Table IV sets Max Trials to 1 in Keras Tuner's Hyperband. That means the tuner evaluated exactly one hyperparameter configuration. The reported settings (160 units, ReLU, Adam) are a single draw from the search space, not a tuned optimum. The authors even acknowledge that 'running multiple trials can sometimes provide more optimal results,' yet they disabled it. So the independent variable of the study was never actually manipulated. The stress-test note hits the mark.\n\nWhat is genuinely new: the 10d2a dataset, two labeled contextual anomalies (heating malfunction and unintended cooking) collected from an AirThings device, and the window-size scan under individual and majority criteria. The paper is clearly written about the UoCAD extension, and the threshold equations are inherited transparently. If the numbers are accurate under the stated protocol, they show that event-level F1 scores vary dramatically with window size, which is a useful empirical observation.\n\nThe soft spots are serious. No UoCAD baseline results appear in Table VII, so even if tuning had happened, the 'impact' of optimised hyperparameters would be unmeasurable. The event-level true-positive rule (Section IV-C) treats an entire anomalous segment as detected if any instance is flagged, which inflates recall and makes F1 scores incomparable to per-instance metrics. No repeated runs or confidence intervals are reported, despite the acknowledged variance of deep learning. Code and data are not released, so reproducibility is low.\n\nThe paper is not a waste: the dataset is a small but real artifact, and the analysis of window-size effects is suggestive. But the central claim is unsupported by the experimental setup. This is a desk-reject level issue, not a revision-level one. I would send it back with a note that the tuner must be run with a real number of trials, the UoCAD baseline must be reported under the same protocol, and per-instance metrics should accompany the event-level ones.\n\nFor a reading group, it might be worth ten minutes to discuss why max_trials=1 is so easy to overlook, but I would not cite it or build on it. Recommendation: reject.","headline":"Max Trials = 1 means the tuner never tuned: the paper's central claim is unsupported, though the 10d2a dataset is a small useful artifact.","tokens_in":10365,"tokens_out":2227,"would_cite":false,"duration_ms":20416,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"UoCAD-OH claims that automatically tuned hyperparameters let an unsupervised Bi-LSTM sliding-window detector reach 0.97 F1 on smart-home contextual anomalies.","keywords":["contextual anomaly detection","Bi-LSTM","hyperparameter optimisation","sliding window","smart home time series","unsupervised learning","online anomaly detection","air quality data"],"falsifier":"Re-run the evaluation on the 2d1a and 10d2a datasets scoring each instance independently instead of counting a whole anomalous segment as a true positive when one of its instances is flagged; if per-instance Recall falls well below 1.0, the headline F1 scores depend on the aggregation rule.","tokens_in":9315,"feed_emoji":"🏠","tokens_out":6092,"duration_ms":52620,"temperature":0.7,"pith_summary":"UoCAD-OH is an attempt to show that automatic hyperparameter tuning improves the unsupervised Bi-LSTM (bidirectional long short-term memory) based contextual anomaly detector UoCAD. The paper claims that replacing fixed hyperparameters with values found by an offline Hyperband search on five months of unlabeled smart-home air-quality data preserves and in places improves detection, reaching best Precision 0.95, Recall 1.0, and F1 0.97 on a two-day dataset with one unintended-cooking anomaly, and detecting both anomalies in a ten-day dataset when window sizes 24 and 48 are used. This matters because existing RNN anomaly detectors usually fix hyperparameters by hand and pay little attention to sliding-window size, and the paper targets real-world contextual anomalies rather than synthetic point outliers. If correct, the result says a modest offline tuning step makes an online Bi-LSTM detector practical for smart-home air-quality monitoring.","feed_headline":"Hyperparameter-tuned Bi-LSTM reaches 0.97 F1 on smart-home anomalies","feed_subtitle":"Automatically tuned window sizes 24 and 48 detect cooking and heating anomalies in home air-quality time series.","key_machinery":"The central mechanism is a sliding-window Bi-LSTM: each window is read in both temporal directions, the model predicts the next instance, and the prediction error is converted into a feature-wise Average Absolute Relative Error (AARE). A dynamic threshold of three standard deviations above the running mean of historical AARE values decides whether an instance is anomalous. UoCAD-OH adds a Hyperband-based automatic hyperparameter search over input units, activation, learning rate, optimiser, number of layers, and dropout rate, run offline on a larger unlabeled dataset; the winning configuration (160 units, ReLU, learning rate 1e-4, Adam, 2 layers, dropout 0.2) then runs online with window sizes 6 to 144 and individual or majority voting across nine features. The event-level scoring rule, where one flagged instance labels the whole anomalous segment as detected, is also part of the evaluation machinery.","core_discovery":"The paper's central claim is that UoCAD-OH, which adds offline automatic hyperparameter optimisation to the UoCAD pipeline, can detect contextual anomalies in multivariate smart-home time series with high event-level precision and recall. Using eight window sizes (6 to 144) and two voting criteria (individual and majority), the tuned Bi-LSTM achieved its best result on the 2d1a dataset under the individual criterion: Precision 0.95, Recall 1.0, and F1 0.97 at window size 24. On the 10d2a dataset, window sizes 24 and 48 were the only ones that detected both anomalies; the majority criterion failed for 2d1a and detected at most one anomaly on 10d2a. The author's interpretation is that window sizes 24 and 48 are most effective, that the upper useful window size for 2d1a is around 96, and that individual-feature voting outperforms majority voting.","pith_inferences":["Editorial inference: the event-level scoring rule makes the reported recall optimistic relative to per-instance expectations; per-instance precision-recall would be a stricter test and the paper does not report it.","Editorial inference: because the tuning set and test sets come from the same room and sensor, the tuned hyperparameters may be environment-specific; testing on another home's air-quality series would show how much of the gain transfers.","Editorial inference: the best window sizes (24 and 48) are close to the anomaly durations (24 to 28 instances), which suggests window size should be matched to expected anomaly length or made adaptive, a connection the paper does not draw."],"forward_implications":["If UoCAD-OH is right, automatic hyperparameter tuning can substitute for manual fixing in UoCAD while keeping the same sliding-window Bi-LSTM pipeline.","Window sizes 24 and 48 are the most effective for these smart-home anomalies, and sizes above 96 stop working on the two-day dataset.","The majority criterion (five of nine features agreeing) is generally ineffective, so individual-feature voting is the better operating point for this method.","The same unintended-cooking anomaly is detected at similar quality in both datasets, suggesting the tuned model transfers across time periods in the same environment."],"supporting_citations":[{"why":"Supplies the UoCAD baseline: the sliding-window Bi-LSTM pipeline, the AARE and dynamic-threshold mechanism, and the 2d1a dataset.","marker":"[5]"},{"why":"Supplies the automatic hyperparameter optimisation procedure that produces the tuned Bi-LSTM configuration.","marker":"[28]"},{"why":"Supplies the smart-home air-quality sensor data for all three datasets used in tuning and evaluation.","marker":"[27]"},{"why":"Supplies the deep-learning implementation platform used to build and train the Bi-LSTM model.","marker":"[29]"},{"why":"Supplies the neural-network API used to define the Bi-LSTM layers.","marker":"[30]"}],"fun_headline_variants":["Auto-tuned Bi-LSTM nails smart-home anomalies with F1 0.97","Tuned Bi-LSTM spots home air anomalies: F1 0.97","Window size 24 best for tuned Bi-LSTM smart-home anomaly detection","Auto-optimized Bi-LSTM detects both home anomalies with F1 0.97","Tuned Bi-LSTM hits 0.97 F1 on home air quality anomaly detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported scores count an anomalous segment as detected if any single instance in it is flagged, so Recall 1.0 reflects event-level detection, not per-instance classification.","fun_headline_variants_meta":{"raw":{"variants":["Auto-tuned Bi-LSTM nails smart-home anomalies with F1 0.97","Tuned Bi-LSTM spots home air anomalies: F1 0.97","Window size 24 best for tuned Bi-LSTM smart-home anomaly detection","Auto-optimized Bi-LSTM detects both home anomalies with F1 0.97","Tuned Bi-LSTM hits 0.97 F1 on home air quality anomaly detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00122,"raw_usage":{"total_tokens":5022,"prompt_tokens":952,"completion_tokens":4070,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":3961}},"tokens_in":568,"tokens_out":4070,"duration_ms":24571,"temperature":1.0,"reasoning_tokens":3961,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:40:00.638859+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation on the 2d1a and 10d2a datasets scoring each instance independently instead of counting a whole anomalous segment as a true positive when one of its instances is flagged; if per-instance Recall falls well below 1.0, the headline F1 scores depend on the aggregation rule.","supporting_citations":[{"cited_title":"UoCAD: An unsu- pervised online contextual anomaly detection approach for multivariate time series from smart homes,","cited_arxiv_id":null,"evidence_quote":"Supplies the UoCAD baseline: the sliding-window Bi-LSTM pipeline, the AARE and dynamic-threshold mechanism, and the 2d1a dataset."},{"cited_title":"Keras Tuner,","cited_arxiv_id":null,"evidence_quote":"Supplies the automatic hyperparameter optimisation procedure that produces the tuned Bi-LSTM configuration."},{"cited_title":"View plus - smart indoor air quality monitor,","cited_arxiv_id":null,"evidence_quote":"Supplies the smart-home air-quality sensor data for all three datasets used in tuning and evaluation."},{"cited_title":"Keras: deep learning library for theano and tensorflow. 2015,","cited_arxiv_id":null,"evidence_quote":"Supplies the neural-network API used to define the Bi-LSTM layers."}],"review_version":1}