{"id":"e84e3500-2f61-43fb-b7b4-707e4717cc8a","arxiv_id":"2501.17976","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"KoopAGRU, a GRU-based Koopman model with FFT time-variant/invariant decomposition, reports an average F1 of 90.88% on five anomaly detection benchmarks, exceeding cited baselines.","lead":"A new deep learning model, KoopAGRU, combines Fourier decomposition, Koopman operator theory, and GRU encoders to detect anomalies in time series. The authors report the best average F1-score on five standard benchmarks, but the evaluation relies on per-dataset tuning and lacks released code or error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA F1 claim rests on per-dataset validation-tuned anomaly ratio; fixing r=1 alone drops the paper's own average to 85.09, below the best baseline.","rationale":"Agree with the reader that the weakest point is the evaluation protocol. I identify the same load-bearing concern: the per-dataset, validation-set selection of the anomaly ratio r (and alpha, beta) that baselines were not given. This is the single decisive issue for the central SOTA claim. The paper's own ablation provides a ready falsification: from Table IX, averaging the F1 scores at r=1 across the five datasets gives 85.09, below ModernTCN's 86.62. Thus the reported 90.88% is the maximum over a small grid of r for each dataset, not a representative performance under a fixed threshold convention. Unless the baselines were allowed the same per-dataset validation-threshold search, the comparison is apples-to-oranges. The reader's CONDITIONAL verdict is appropriate; code release and a fixed or preregistered threshold protocol are necessary before the claim can be assessed. I also note secondary inconsistencies (intro vs. abstract F1, PSM text vs. table, alpha value typos) that do not independently sink the claim but reinforce the need for careful re-evaluation. The architecture itself has a plausible rationale (FFT decomposition plus Koopman operators and GRU encoders), and the Table IV arithmetic checks out, so I am not inclined to move the verdict to REJECT; the path to verification is a controlled experimental re-run.","tokens_in":21607,"tokens_out":9282,"duration_ms":85816,"concrete_test":"Recompute the average F1 from the paper's own Table IX with a fixed r=1 for all five datasets (alpha and beta kept at the reported best values). If, as the table implies, the average is 85.09 (< 86.62), the 90.88% SOTA claim is an artifact of per-dataset r selection, and the authors should either adopt a fixed threshold protocol or benchmark baselines under the same per-dataset validation tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract, Table IV) is a 90.88% average F1 that beats 20 baselines. Section IV.B (Eq. 22) sets the anomaly threshold as the r-th percentile of validation prediction errors, and Section IV.C.b gives dataset-specific r: 0.5 for SMD, 1 for MSL/PSM, 4 for SWaT/SMAP. Section IV.F.4 (Table IX) confirms r was selected per dataset to maximize validation F1. The paper's own numbers show this selection is decisive: fixing r=1 for all datasets yields F1s of 97.16, 90.61, 85.31, 67.20, and 85.18 (PSM, SWaT, SMD, SMAP, MSL), an average of 85.09—below the 86.62 of the best baseline (ModernTCN). Thus the reported 90.88% is the per-dataset argmax over an r grid, not a robust protocol-based result. The 20 baselines in Table IV were not given this validation-label-informed threshold search, and the paper does not state their threshold protocol. Without a matched evaluation protocol, the SOTA claim is not established; supporting evaluation sloppiness (abstract 90.88% vs. intro 90.37%, PSM text 97.03 vs. table 97.16, alpha comment listing 'SMAP and SMAP') reinforces the need for a controlled re-evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes KoopAGRU, a forecasting-based unsupervised time-series anomaly detector that combines FFT-based decomposition into time-variant and time-invariant components, GRU encoders that learn Koopman observables, and fixed-size Koopman operators, trained with a single loss. The evaluation on SMD, PSM, MSL, SMAP, and SWaT reports an average F1 of 90.88% against 20 baselines, together with resource-efficiency comparisons. The central claim is that KoopAGRU achieves a new state of the art in unsupervised time-series anomaly detection with fast inference and modest parameter counts.","tokens_in":21872,"tokens_out":6985,"duration_ms":69324,"significance":"If the state-of-the-art result were obtained under an evaluation protocol matched to the baselines, this would be a useful contribution: it is one of the few Koopman/DeepDMD formulations applied to unsupervised anomaly detection, it uses measurement-inclusive observables to avoid a decoder, and the training objective in Section III.G is clean and simple to implement. The paper also states its limitations explicitly in Section IV.G, which is good practice. However, the empirical claim is currently conditional on label-informed per-dataset threshold tuning, and the manuscript contains several numerical inconsistencies. No code release is mentioned, so the efficiency numbers are not independently checkable. I assess the significance as high if the protocol issue can be resolved, but the evidence in the present version does not yet support the headline claim.","major_comments":[{"comment":"The threshold ratio r is a per-dataset free parameter selected on validation labels: Eq. (22) sets delta as the r-th percentile of validation prediction errors, Section IV.C.a fixes r=0.5/1/1/4/4 per dataset, and Section IV.F.4 with Table IX confirms that r was chosen to maximize validation F1. This is label-informed threshold selection, and the paper's own Table IX shows it is decisive: fixing r=1 for all datasets gives F1 values of 97.16, 90.61, 85.31, 67.20, and 85.18 on PSM, SWaT, SMD, SMAP, and MSL, an average of 85.09, which is below the 86.62 average of ModernTCN in Table IV. The reported 90.88% average is therefore an argmax over a per-dataset r grid, and the manuscript does not state what threshold protocol was used for the 20 baselines. The central state-of-the-art claim is not established without a matched, pre-specified threshold protocol.","section":"IV.B, IV.C.a, IV.F.4 (Eq. 22; Table IX)"},{"comment":"The numerical record is internally inconsistent. The abstract reports 90.88% average F1 while the introduction reports 90.37%; Section IV.E.c reports a PSM F1 of 97.03 while Table IV and Table IX report 97.16; Table VII pairs PSM with (alpha=0.1, beta=0.8) even though Tables V and VI identify the optimal PSM configuration as (alpha=0, beta=0.5); and Section IV.C.b gives alpha=0.1 for SMD while Table V's SMD column reaches its maximum at alpha=0.5. These inconsistencies must be corrected before the evaluation can be reproduced.","section":"Abstract, Introduction, IV.C.b, IV.E.c, Table IV, Table V, Table VII"},{"comment":"No error bars, multiple-seed results, or statistical tests are reported. Section IV.C.c describes a single training setup, and Table IV reports one F1 value per dataset. Because the headline advantage over ModernTCN is about 4.26 points and the threshold protocol is sensitive, single-run numbers are insufficient to support a state-of-the-art claim; please report mean and standard deviation over at least five seeds and, ideally, a significance test against the strongest baselines.","section":"IV.C.c, Table IV"},{"comment":"Section IV.G honestly lists longer training time and limited dataset coverage as limitations, but Table X shows that KoopAGRU's training time on SWaT is 3039 seconds versus 1477 seconds for TimesNet and 862 seconds for Autoformer. If efficiency is part of the contribution, it should be framed as inference-time and parameter efficiency rather than overall efficiency, because the training-time picture is unfavorable on at least one dataset.","section":"IV.G, Table X"}],"minor_comments":[{"comment":"Section IV.D states that 20 baselines are compared, but the enumeration contains 19 named models unless TimesNet (I) and TimesNet (R) are counted separately; please clarify the count.","section":"IV.D"},{"comment":"Section IV.E.b contains the typo 'SWaR' for 'SWaT', and Section IV.C.b says 'SMAP and SMAP' where the second dataset name appears to be missing.","section":"IV.E.b, IV.C.b"},{"comment":"Equation (18) uses Phi X_t before it is defined in Eq. (20), and Phi X_{t+1} is defined later in Eq. (16); reorder the definitions or add forward references.","section":"III.G, Eq. (18)"},{"comment":"The table captions use the informal headers 'Alpha' and 'Beta' and the abbreviation 'SW AT'; please use consistent dataset names and explain the parenthetical hyperparameter values in each caption.","section":"Tables V, VI, VII"},{"comment":"References [65] and [26] are the same paper; citing it twice in different positions is confusing.","section":"References"},{"comment":"No code release is mentioned; providing code or a configuration file would materially improve reproducibility, especially because many hyperparameters are dataset-specific.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The per-dataset r tuning issue is the load-bearing problem. I would ask the editor to require a fixed-threshold or fully pre-specified protocol analysis, with the same protocol applied to all baselines, before considering the paper for acceptance. The method itself appears sound and the central idea is publishable if the empirical claim can be made robust. The numerical inconsistencies in Tables IV through IX should also be audited carefully."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"KoopAGRU is a legitimate new architecture: it takes the FFT-based time-variant/invariant decomposition from Koopa and replaces the linear layers with GRU encoders that learn state-inclusive Koopman observables, with a fixed-size Koopman matrix and no decoder. That combination is new for anomaly detection, the loss is coherent, and the resource-efficiency tables show a small model with fast inference. The ablation study is unusually thorough: α, β, λ, r, and normalization configs are each varied, and the best settings are reported per dataset. That transparency is real and useful.\n\nThe problem is that the headline number — 90.88% average F1, beating 20 baselines — is not earned by a fair protocol. The threshold δ is the r-th percentile of validation errors, and r is chosen per dataset to maximize validation F1 (Eq. 22, Section IV.C.b, Table IX). The paper's own ablation shows that fixing r=1 for all datasets gives an average of 85.09, below the best baseline ModernTCN at 86.62. So the SOTA claim is essentially the per-dataset argmax over an r grid. The 20 baselines were not given the same validation-label-informed threshold search, and the paper doesn't state their threshold protocol. That alone makes the comparison unfair.\n\nThere are also smaller inconsistencies that don't help: abstract says 90.88% but intro says 90.37%; the PSM discussion says 97.03 while Table IV says 97.16; Table V's caption lists 'SMAP and SMAP' for α; Table VII's PSM settings (α=0.1, β=0.8) conflict with Table V's PSM best (α=0, β=0.5). No code, no seeds, no error bars, so we can't check variance.\n\nNone of this kills the architecture. The method is plausible and the loss is well-defined. What needs fixing is the evaluation: use a fixed threshold protocol (e.g., a single r for all datasets, or a threshold selected without validation-label feedback), give baselines the same threshold-tuning budget, report seed variance, and correct the text/table conflicts. If the authors do that, the model may still be competitive — the resource efficiency and inference speed are genuinely attractive.\n\nRecommendation: send it to peer review, but with the clear expectation that the evaluation protocol be overhauled before acceptance. This is not a desk-reject case; the architecture and analysis are worth a serious referee. I'd cite the method as related work, but not the SOTA claim as-is.","headline":"Legitimate new architecture; the 90.88% SOTA claim is an artifact of per-dataset threshold tuning and needs a fair re-evaluation.","tokens_in":22463,"tokens_out":3755,"would_cite":false,"duration_ms":33018,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"KoopAGRU claims 90.88% average F1, beating 20 baselines on five benchmarks.","keywords":["time-series anomaly detection","Koopman operator theory","Deep Dynamic Mode Decomposition","Gated Recurrent Units","Fast Fourier Transform","unsupervised learning","prediction-error thresholding"],"falsifier":"Apply the paper's own validation rule to every baseline: for each dataset, choose $r$ (and any equivalent threshold or anomaly-ratio parameters) to maximize F1 on the validation split, then compute test F1. If KoopAGRU's average F1 lead over ModernTCN, TimesNet, and FEDformer drops to about one point or less, the claimed state-of-the-art result is explained by tuning protocol, not by the Koopman/GRU architecture.","tokens_in":21340,"feed_emoji":"🚨","tokens_out":9683,"duration_ms":89766,"temperature":0.7,"pith_summary":"The paper is trying to establish that Koopman-operator ideas, normally used for forecasting nonlinear dynamics, can be turned into a high-performing unsupervised time-series anomaly detector. It claims that KoopAGRU—an FFT split of each window into stable and changing parts, GRU encoders that learn Koopman observables, and fixed-size Koopman operators—can predict the next step and flag anomalies when prediction error exceeds a dataset-specific percentile threshold. The reported result is a new average F1-score of 90.88% on SMD, MSL, SMAP, SWaT, and PSM, above all 20 baselines (the introduction states 90.37%). The design would matter if true because removing the decoder and inverse mapping, while keeping the Koopman matrix small, gives faster inference and modest resource use. The paper itself notes in its limitations section that training time is longer than alternatives and that only five datasets have been tested.","feed_headline":"Koopman model reports 90.88% average F1 on five anomaly datasets","feed_subtitle":"FFT plus GRU-learned Koopman observables beats 20 baselines on SMD, MSL, SMAP, SWaT, PSM.","key_machinery":"The central object is a finite-dimensional approximation of the Koopman operator: a fixed-size linear matrix $K$ that advances lifted observables one step in time. KoopAGRU builds the observable vector as $\\Psi(X_t) = [X_t; \\psi(X_t)]$, with $\\psi$ learned by GRU encoders, so the raw measurements appear inside the lifted state and can be recovered without an inverse mapping. The FFT filter splits the input into dominant time-invariant frequencies $G_\\alpha$ and the remaining time-variant frequencies, making the operator block-structured and letting $\\beta$ weight the invariant branch. This machinery converts the nonlinear forecasting problem into a linear one, and the single forecasting loss doubles as the anomaly score: points whose prediction errors exceed the $r$-th percentile of validation errors are flagged.","core_discovery":"KoopAGRU claims that a nonlinear, multi-sensor time series can be treated as approximately linear by lifting each window into an observable space and applying a fixed-size Koopman operator. The FFT split separates each look-back window into a time-invariant part (dominant frequencies shared across all windows) and a time-variant part (the remainder). Two GRU encoders learn Koopman observables for the two parts; because the observable vector includes the raw measurements themselves, the next-step prediction is read off directly and no decoder or inverse map is needed. Training minimizes the Frobenius distance between the lifted future window and the operator-applied lifted present window, regularized by the norms of the Koopman matrices. On SMD, MSL, SMAP, SWaT, and PSM, the paper reports an average F1-score of 90.88%, above all 20 baselines.","pith_inferences":["Editorial extension: a direct stress test not reported in the paper is to freeze a single ($\\alpha$, $\\beta$, $r$) configuration across all five datasets and measure the average F1 loss; this separates architecture from per-dataset calibration.","Editorial extension: the global FFT split (one $\\alpha$ per dataset) could be replaced by per-window or per-channel frequency masks; datasets like PSM, whose anomalies are spikes rather than periodicity breaks, might then show larger gains than the paper reports.","Editorial extension: since $K_{\\mathrm{var}}$ is a learned linear operator, its eigenvalues can be inspected after training; a testable prediction is that anomaly onsets on SWaT and SMAP align with growing magnitude of specific Koopman modes, which would turn the anomaly score into a spectrally interpretable signal."],"forward_implications":["If the reported numbers hold, KoopAGRU would be the leading unsupervised detector on the SMD, MSL, SMAP, SWaT, and PSM benchmark, ahead of transformer and TCN baselines.","The measurement-inclusive observable design means future values are read directly from the lifted state, so inference needs no decoder and can run in one forward pass.","The fixed-size Koopman operator keeps parameter counts between roughly 0.7 million and 1.6 million across datasets, with lower GPU and RAM footprints than the transformer baselines in the reported runs.","The dataset-specific $\\beta$ values suggest that weighting the time-invariant component is the main tuning lever; SWaT's F1 moves from 83.60 to 94.83 as $\\beta$ goes from 0 to 0.8.","The model is strongest on datasets whose anomalies break periodic patterns and weakest on PSM's irregular, spike-like anomalies, consistent with the frequency-split design."],"supporting_citations":[{"why":"Defines the Koopman operator that the model approximates with DeepDMD.","marker":"[1]"},{"why":"Named as the source of DeepDMD, the procedure for learning Koopman observables from data.","marker":"[3]"},{"why":"Supplies the normalization scheme applied to the time-variant component to handle non-stationary data.","marker":"[8]"},{"why":"Provides the FFT-based split into time-invariant and time-variant components that KoopAGRU adopts.","marker":"[56]"},{"why":"Is the forecasting-with-threshold anomaly-detection approach adapted by KoopAGRU.","marker":"[65]"},{"why":"Supplies the non-overlapping sliding-window preprocessing used on all datasets.","marker":"[66]"},{"why":"Provides the SMD dataset used for evaluation.","marker":"[9]"},{"why":"Provides the MSL and SMAP spacecraft datasets used for evaluation.","marker":"[10]"},{"why":"Provides the SWaT water-treatment dataset used for evaluation.","marker":"[11]"},{"why":"Provides the PSM pooled-server-metric dataset used for evaluation.","marker":"[12]"}],"fun_headline_variants":["Koopman GRU model beats 20 baselines with 90.88% F1 on five datasets","Koopman observables from FFT+GRU boost anomaly F1 to 90.88%","Taming nonlinear dynamics: Koopman GRU hits 90.88% F1","No decoder, pure operator: KoopAGRU reads anomalies straight from lifted state","Koopman theory + GRUs: F1 90.88% on five anomaly benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 90.88% average depends on choosing $r$, $\\alpha$, and $\\beta$ separately per dataset using validation data; if the baselines were not given the same per-dataset calibration, the reported lead is a tuning artifact rather than a model advantage.","fun_headline_variants_meta":{"raw":{"variants":["Koopman GRU model beats 20 baselines with 90.88% F1 on five datasets","Koopman observables from FFT+GRU boost anomaly F1 to 90.88%","Taming nonlinear dynamics: Koopman GRU hits 90.88% F1","No decoder, pure operator: KoopAGRU reads anomalies straight from lifted state","Koopman theory + GRUs: F1 90.88% on five anomaly benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000892,"raw_usage":{"total_tokens":3842,"prompt_tokens":933,"completion_tokens":2909,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":2787}},"tokens_in":549,"tokens_out":2909,"duration_ms":21054,"temperature":1.0,"reasoning_tokens":2787,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:27:40.268568+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the paper's own validation rule to every baseline: for each dataset, choose $r$ (and any equivalent threshold or anomaly-ratio parameters) to maximize F1 on the validation split, then compute test F1. If KoopAGRU's average F1 lead over ModernTCN, TimesNet, and FEDformer drops to about one point or less, the claimed state-of-the-art result is explained by tuning protocol, not by the Koopman/GRU architecture.","supporting_citations":[{"cited_title":"Koopa: Learning non-stationary time series dynamics with koopman predictors,","cited_arxiv_id":null,"evidence_quote":"Provides the FFT-based split into time-invariant and time-variant components that KoopAGRU adopts."},{"cited_title":"Fed-anids: Federated learning for anomaly- based network intrusion detection systems,","cited_arxiv_id":null,"evidence_quote":"Is the forecasting-with-threshold anomaly-detection approach adapted by KoopAGRU."},{"cited_title":"Characteristics of Interplanetary Discontinuities in the Inner Heliosphere Revealed by Parker Solar Probe","cited_arxiv_id":"2106.00189","evidence_quote":"Supplies the non-overlapping sliding-window preprocessing used on all datasets."}],"review_version":1}