{"id":"ba3d14bb-b03b-4f8d-a0df-0aa31aab19c9","arxiv_id":"2506.12183","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Sliding-window cross-validation yields higher and more stable AUC-PR than walk-forward for subsequence anomaly detection on two small CAN fault datasets.","lead":"This study compares two time-series cross-validation strategies, walk-forward and sliding window, for evaluating fault-detection classifiers on automotive CAN bus data. It finds that sliding-window evaluation gives higher and more stable precision-recall scores, and argues that evaluation design materially changes reported anomaly detection performance.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SW advantage may be an artifact of unequal single-class fold exclusion: §4.3 claims SW has more such folds, yet Definitions 2-3 give identical test windows, so the implementation must be checked before trusting the pooled comparison.","rationale":"The reader's weakest assumption is the fairest target: the pooled comparison of AUC-PR between WF and SW is only meaningful if excluded single-class folds do not selectively remove harder or easier evaluations from one strategy. The paper itself flags this in §4.3 and §6.4. My stress-test adds a sharper observation: under the formal definitions in §3.3.1, the test subsequences for WF and SW are literally identical for the same k, ω, and δ. If so, the number of single-class test folds and the class-ratio profile cannot differ between strategies, contradicting §4.3's claim. This puts the burden on the implementation: either the code follows the definitions (making the §4.3 claim false, but the comparison fair) or it deviates (making the comparison potentially biased). In both cases, the current manuscript is not internally consistent enough to support the abstract's unqualified 'SW consistently yields higher' claim. I also note the variance claim is contradicted by Fig. 3's text ('SW ... exhibits higher interquartile range'), so the abstract's 'reduced fold-to-fold performance variance' is at best unsubstantiated. The proposed check directly resolves the more fundamental question—whether the fold-exclusion bias actually lands—by inspecting the code and re-running the comparison on matched folds. Given the paper's reproducible code and data release, this is a feasible and decisive test. My verdict remains CONDITIONAL, matching the reader's; no change in recommendation.","tokens_in":25771,"tokens_out":9616,"duration_ms":112320,"concrete_test":"Inspect the released GitHub code (github.com/TovNephesh/MTS_CAN) and extract, for each dataset (power injector, spark plug), each K in {3,...,9}, and each fold k, the test time indices for WF and SW. First, verify whether S^k_test(WF) equals S^k_test(SW) for all k. Then count how many folds are excluded (0% or 100% positive) per strategy. If the excluded-fold counts differ, re-run the pooled AUC-PR comparison and the Mann-Whitney test using only folds that have identical test windows (or are matched on test-positive ratio) and are non-degenerate. If the SW advantage disappears or reverses in this matched analysis, the headline claim is a fold-selection artifact; if it persists, the exclusion-bias concern is refuted for this dataset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Comparison of WF and SW AUC-PR scores in §5.1 is the empirical core of the paper. The pooled median and Mann-Whitney one-sided tests are reported after dropping folds whose test sets contain only one class, because AUC-PR is undefined there. The paper states (§4.3) that such single-class test folds are 'prevalent under the sliding window configuration, especially at higher fold counts' and (§6.4) that class imbalance is different for WF and SW, limiting comparability. However, Definition 2 (WF) and Definition 3 (SW) define the test subsequence identically for fixed k, ω, δ: both use S^k_test = {x_{ω+(k−1)δ+1}, ..., x_{ω+kδ}}. Hence, under these definitions, the test windows, and therefore their positive-class ratios and the set of excluded single-class folds, cannot differ between strategies. The manuscript therefore contradicts its own formal setup. If the released code implements the definitions, the exclusion-bias worry disappears but §4.3 and §6.4 are inaccurate; if the code deviates (e.g., different window starts or offsets for SW), then the pooled comparison is computed over different fold populations, and the reported SW advantage could be an artifact of which folds were dropped. Either way, the central claim is not robustly supported until the actual fold construction is verified and the comparison is re-run on a matched, non-degenerate set of folds.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper conducts an empirical comparison of two temporal cross-validation strategies, walk-forward (WF) and sliding window (SW), for subsequence anomaly detection in multivariate time series, using two CAN-bus fault datasets. Eight classifiers (SVM, RF, XGBoost, ResNet, TCN, LSTM+FCN, InceptionTime, ROCKET) are evaluated with AUC-PR under K-folds 3 through 9. The central claim is that SW yields consistently higher median AUC-PR and lower fold-to-fold variance than WF, especially for deep architectures, and that the choice of TSCV strategy materially changes benchmark conclusions. The authors release code and data and report a series of Mann-Whitney tests in support of the claim, while also acknowledging limitations around class imbalance and fold comparability.","tokens_in":26035,"tokens_out":3115,"duration_ms":75489,"significance":"If the main claim is correct, the paper addresses a real gap: benchmark evaluations of multivariate time series anomaly detectors often ignore how the temporal validation scheme interacts with classifier performance and class imbalance. The study is one of the few to systematically vary TSCV strategy, fold count, and classifier family on a controlled fault-injection dataset, and it ships public code and datasets, which is a concrete strength for reproducibility. However, the central comparison currently rests on a contradiction in the formal definitions and on statistical analyses that pool non-independent fold-level scores; these issues must be resolved before the significance of the SW advantage can be assessed.","major_comments":[{"comment":"The manuscript contradicts its own formal setup. Definition 2 (WF) and Definition 3 (SW) specify identical test subsequences, S^k_test = {x_{ω+(k−1)δ+1}, ..., x_{ω+kδ}}, for fixed k, ω, δ. Consequently, the positive-class ratios of the test folds, and hence the set of single-class test folds excluded from AUC-PR evaluation, cannot differ between WF and SW under these definitions. Yet §4.3 states that single-class test folds are 'prevalent under the sliding window configuration, especially at higher fold counts,' and §6.4 says class imbalance is different for WF and SW, limiting comparability. Either the implementation deviates from the definitions (e.g., different ω or different window starts for SW), in which case the pooled comparison in §5.1 is over different fold populations and the reported SW advantage may be an artifact of which folds were dropped, or the implementation matches the definitions, in which case §4.3 and §6.4 are inaccurate. The authors must verify the fold construction in the released code and re-run the comparison on a matched, non-degenerate set of folds, reporting how many folds were excluded per strategy and K.","section":"Definitions 2–3; §4.3; §6.4"},{"comment":"The statistical evidence for the central claim is not robust. The Mann-Whitney U tests in Section 5.1 and Table 2 treat fold-level AUC-PR scores as independent observations, but folds from the same time series are strongly dependent, and SW windows are explicitly overlapping. This dependence can inflate apparent differences and produce p-values reported as 0.00. The manuscript should use a cluster-robust permutation test or a mixed-effect model with fold/sequence as a random effect, report confidence intervals for median differences, and address the multiple comparisons across K values and classifiers (e.g., with a false-discovery-rate correction). Without this, the 'consistently' in the abstract is not supported by the reported statistics.","section":"§5.1; §5.3; Table 2"},{"comment":"The comparison between WF and SW is confounded by the training window size. Section 5 says ω is 'fluctuating and dependant on TSCV strategy and K-fold number,' but the paper never reports the actual ω values used for each strategy and K. In Definition 2 the training window grows as ω+(k−1)δ, while in Definition 3 the training window has fixed length ω but starts at 1+(k−1)δ; if the ω assigned to SW differs from the initial ω assigned to WF, then the comparison in §5.1 is partly a comparison of training set sizes and recency, not of validation strategy per se. The authors should report the full schedule of ω and δ for both strategies and, if possible, include an analysis that matches training set sizes or conditions on them.","section":"§5; §4.1 (Definitions 2–3)"},{"comment":"The limitation section concedes that class imbalance 'limits comparability of the two TSCV methods as class imbalance is different for WF and SW.' This concession directly qualifies the paper's main empirical conclusion. The manuscript should either provide evidence that the differing imbalance profiles do not drive the SW advantage (e.g., by stratifying on test positive-class ratio or restricting to folds with comparable ratios) or soften the abstract's claim accordingly. As written, the conclusion in Section 7 that 'SW consistently outperformed WF' goes beyond what the acknowledged limitations and the reported analysis can support.","section":"§6.4"}],"minor_comments":[{"comment":"The final sentence is grammatically incomplete: 'TSCV design in benchmarking anomaly detection models on streaming time series and provide guidance' should be rephrased, e.g., 'This study demonstrates that TSCV design matters in benchmarking anomaly detection models on streaming time series and provides guidance for selecting evaluation strategies.'","section":"Abstract"},{"comment":"The notation 'K∈{3,9}' is ambiguous; the experiments and Table 3 clearly include K = 3,4,5,6,7,8,9. Please write K ∈ {3,4,...,9}.","section":"§5"},{"comment":"The RF p-value is reported as 0.35 in the text but as 0.18 in Table 2; these should be reconciled, and exact p-values should be reported rather than rounded to 0.00.","section":"Table 2; §5.3"},{"comment":"There are several typos, including 'medidan' for 'median' and 'subsequnce' for 'subsequence', which should be corrected throughout.","section":"§5.5"},{"comment":"The statement 'Extreme cases such as test folds with only a single class were prevalent under the sliding window configuration' conflicts with Definitions 2–3, as noted in the major comments; once the fold-construction issue is resolved, this sentence should be revised to match the actual implementation.","section":"§4.3"},{"comment":"The subsection begins 'This section we contextualize our work', which is ungrammatical; it should be 'This section contextualizes our work'.","section":"§2.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The stress-test note lands. Read the definitions carefully: Definition 2 (WF) and Definition 3 (SW) give exactly the same test subsequence for fixed k, ω, and δ. The training windows differ, but the test windows do not. So single-class test folds cannot be more prevalent under SW, and the class-imbalance profiles cannot differ. Section 4.3 claims they do, and Section 6.4 concedes the same. If the code implements the definitions, those passages are wrong. If the code deviates, then the pooled comparison in Section 5.1 is computed over different fold populations, and the reported SW advantage could be an artifact of which folds were dropped. Either way, the central claim isn't robustly supported until someone checks the actual fold construction.\n\nWhat the paper does well: it ships public code and data, which is real evidence, and the limitations section is honest about dataset size and class imbalance. The empirical comparison across eight classifiers and K = 3–9 is systematic, and the authors explicitly position their finding against Cerqueira et al. (2020), which found growing windows better for forecasting. The specific claim that SW yields higher and more stable AUC-PR for subsequence anomaly detection on CAN fault data is new and worth testing.\n\nSoft spots in proportion: the internal contradiction is load-bearing, not cosmetic. Beyond that, the Mann-Whitney tests are run on pooled, non-independent fold scores without cluster-robust or mixed-effect analysis, and there is no multiple-comparison correction across K and classifiers. The paper also has internal p-value inconsistencies — RF is 0.18 in Table 2 but 0.35 in the text, XGBoost is 0.00 in the table but 0.01 in the text. The abstract's final sentence is grammatically broken. None of this is fatal on its own, but together it means the empirical core needs a careful revision.\n\nWho this is for: practitioners benchmarking fault-detection classifiers on streaming multivariate data, and methodologists working on temporal validation. It deserves a serious referee — the question is important and the paper is honest — but the referee should demand verification of the fold construction, a matched-fold re-analysis, and a statistically cleaner comparison before the claim is accepted.","headline":"Useful but internally inconsistent: the reported sliding-window advantage may be an artifact of uneven fold exclusion, since the paper's own Definitions 2 and 3 define identical test windows for WF and SW.","tokens_in":26580,"tokens_out":1989,"would_cite":false,"duration_ms":24996,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"How you slice time in cross-validation changes anomaly detection benchmark results: sliding-window validation yields higher median AUC-PR and lower variance than walk-forward on multivariate CAN fault data, especially for deep classifiers.","keywords":["temporal cross-validation","multivariate time series","subsequence anomaly detection","walk-forward validation","sliding window validation","AUC-PR","fault detection","controller area network"],"falsifier":"Re-run the same eight classifiers and $K$ values on the two CAN datasets while keeping single-class folds (e.g., scoring them with a metric defined for degenerate test sets) or after matching WF and SW folds to have identical positive-class-ratio distributions; if the sliding-window median AUC-PR advantage disappears or reverses under either correction, the paper's central claim is refuted.","tokens_in":25551,"feed_emoji":"📈","tokens_out":7056,"duration_ms":85226,"temperature":0.7,"pith_summary":"The paper sets out to show that the choice of temporal cross-validation (TSCV) scheme is a decision that changes the measured quality of subsequence anomaly detectors, not a neutral procedural detail. Using multivariate controller area network (CAN) fault data, it trains eight classifiers under walk-forward (WF) validation, where the training window grows, and sliding-window (SW) validation, where a fixed-size training window slides forward. The central finding is that SW produces higher median AUC-PR scores and lower fold-to-fold variance, with the advantage concentrated in deep classifiers that are sensitive to local temporal continuity, while random forest is essentially unaffected. The authors argue that overlapping windows preserve fault signatures better, especially at low fold counts, so benchmarking studies should report and justify the temporal validation strategy they use.","feed_headline":"Sliding-window cross-validation lifts anomaly-detection benchmark scores","feed_subtitle":"Choosing a temporal validation scheme changes median AUC-PR, with deep models gaining most from overlapping windows.","key_machinery":"The machinery is the temporal partition itself. Walk-forward defines fold $k$'s training subsequence as $S^k_{\\text{train}}=\\{x_1,\\dots,x_{\\omega+(k-1)\\delta}\\}$ and test subsequence as $S^k_{\\text{test}}=\\{x_{\\omega+(k-1)\\delta+1},\\dots,x_{\\omega+k\\delta}\\}$, so the training window keeps growing and never revisits old data. Sliding window keeps a fixed-length training window $S^k_{\\text{train}}=\\{x_{1+(k-1)\\delta},\\dots,x_{\\omega+(k-1)\\delta}\\}$ with the same test-set definition, so consecutive training windows overlap in time. That overlap is the operative mechanism: it gives the classifier repeated, locally contiguous views of intermittent fault signatures and keeps recent context, which the paper credits for higher AUC-PR and lower variance, while the expanding WF window loses recency and creates test-time discontinuity.","core_discovery":"The paper's central claim is that temporal validation design materially changes benchmark conclusions for subsequence anomaly detection in multivariate time series. On two CAN fault datasets (power injector and spark plug), all eight classifiers—SVM, random forest, XGBoost, ResNet, TCN, LSTM+FCN, InceptionTime, and ROCKET—achieve higher median AUC-PR under SW than under WF except random forest, whose small drop from 0.80 to 0.77 is not statistically significant. The differences are significant at $\\alpha=0.05$ for the other seven classifiers, with the largest shift from 0.50 to 0.76 for SVM. The paper also finds that the SW advantage is clearest at $K=3$ through $K=6$, weakens at $K=7$, and re-emerges at $K=8$ and $K=9$. It concludes that overlap-based sliding-window evaluation, rather than growing-window walk-forward evaluation, should be preferred for benchmarking fault-like subsequence anomalies in streaming multivariate data.","pith_inferences":["The class-imbalance asymmetry between WF and SW means pooled AUC-PR comparisons may overstate the SW advantage; a fairer evaluation would weight folds by positive-class ratio or use a metric robust to degenerate test folds.","If the SW advantage is driven by overlap, then window length $\\omega$ and offset $\\delta$ should modulate the effect; sweeping those parameters, which the paper deliberately fixed, is a direct testable extension.","Because the datasets are two small CAN recordings, the conclusion that SW is generally better for MTS subsequence anomaly detection is a hypothesis about other domains; applying the same protocol to larger public MTS fault datasets would show whether it transfers.","Walk-forward's failure despite more training data suggests that recency of the training distribution, not volume, matters for intermittent faults; adding an explicit forgetting mechanism to WF could close that gap."],"forward_implications":["Benchmark studies of subsequence anomaly detectors should report the temporal cross-validation scheme and fold count, because the choice alone moves median AUC-PR substantially (e.g., SVM from 0.50 to 0.76).","At lower fold counts ($K=3$ through $K=6$), sliding-window validation gives a statistically significant advantage at $\\alpha=0.05$; at $K=7$ the advantage loses significance but returns at $K=8$ and $K=9$, so conclusions drawn from a single fold count should not be generalized.","Deep classifiers (ResNet, TCN, LSTM+FCN, InceptionTime, ROCKET) all improve significantly under SW, while random forest stays flat, meaning the validation scheme can alter which architecture appears best in a benchmark.","The overlapping windows of SW preserve fault signatures more effectively at low fold counts, so evaluations meant to mimic streaming fault detection should prefer fixed-window overlap at modest $K$.","Because the two strategies produce different class-imbalance profiles across folds, any comparison of validation schemes should state how degenerate single-class folds were handled."],"supporting_citations":[{"why":"Supplies the taxonomy that defines point, subsequence, and time-series anomalies, framing the detection task at the center of the paper.","marker":"[7]"},{"why":"Provides the prior empirical conclusion that growing windows (akin to WF) are best for univariate non-stationary data, which the paper argues against for imbalanced multivariate subsequence detection.","marker":"[12]"},{"why":"Supplies the walk-forward and prequential validation definitions that the paper formalizes as the WF strategy.","marker":"[39]"},{"why":"Defines standard k-fold cross-validation, the baseline that temporal cross-validation extends and replaces in time-ordered settings.","marker":"[46]"},{"why":"Defines sample-based sliding windows over data streams, which the paper adopts as the SW strategy.","marker":"[59]"},{"why":"Provides the vehicle fault-injection experimental setup and ensemble fault-detection context that the data collection follows.","marker":"[72]"},{"why":"Supplies the CAN-D pipeline used to reverse engineer binary CAN payloads into the multivariate time series datasets.","marker":"[74]"}],"fun_headline_variants":["Sliding-window beats walk-forward for fault-like anomaly benchmarks","Cross-validation choice alters AUC-PR in multivariate anomaly detection","For subsequence anomalies, sliding-window CV yields higher scores","Temporal validation scheme changes anomaly detection rankings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that dropping test folds that contain only one class produces an unbiased comparison between WF and SW; in the CAN data those folds are more common under SW at higher fold counts, and the two strategies produce different class-imbalance profiles, so the reported SW advantage could be an artifact of which folds were excluded rather than a property of the validation method.","fun_headline_variants_meta":{"raw":{"variants":["Sliding-window beats walk-forward for fault-like anomaly benchmarks","Cross-validation choice alters AUC-PR in multivariate anomaly detection","For subsequence anomalies, sliding-window CV yields higher scores","Temporal validation scheme changes anomaly detection rankings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000938,"raw_usage":{"total_tokens":4029,"prompt_tokens":984,"completion_tokens":3045,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":2979}},"tokens_in":600,"tokens_out":3045,"duration_ms":27020,"temperature":1.0,"reasoning_tokens":2979,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:56:36.367580+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same eight classifiers and $K$ values on the two CAN datasets while keeping single-class folds (e.g., scoring them with a metric defined for degenerate test sets) or after matching WF and SW folds to have identical positive-class-ratio distributions; if the sliding-window median AUC-PR advantage disappears or reverses under either correction, the paper's central claim is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the walk-forward and prequential validation definitions that the paper formalizes as the WF strategy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines standard k-fold cross-validation, the baseline that temporal cross-validation extends and replaces in time-ordered settings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines sample-based sliding windows over data streams, which the paper adopts as the SW strategy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the vehicle fault-injection experimental setup and ensemble fault-detection context that the data collection follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CAN-D pipeline used to reverse engineer binary CAN payloads into the multivariate time series datasets."}],"review_version":1}