{"id":"d25a2148-9730-4354-808e-5afc71ca39ec","arxiv_id":"2508.19456","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"ReLATE+ detects adversarial attacks in time-series data, classifies attack family, and selects a resilient model via dataset similarity, reporting near-Oracle accuracy with roughly 78% lower overhead.","lead":"This paper presents ReLATE+, a pipeline that checks time-series data for adversarial attacks, classifies the attack type, and reuses a pre-trained model from the most similar known dataset instead of retraining all models. The authors report a 77.68% reduction in computational overhead while staying within about 2% of the best possible model's performance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ReLATE+ selects its deployed model using test-set labels within the top-3 candidates, so the claimed 2.02%-of-Oracle and 13.28%-over-random results are not established for a deployment-time system.","rationale":"The reader's formal weakest_assumption was the transferability of dataset-level embedding similarity, but their rationale correctly identified the more decisive flaw: the evaluation protocol selects the deployed model using test-set labels. I agree with that rationale and elevate it to the primary load-bearing concern. The transferability assumption matters only if the model selection were otherwise valid; the test-label selection is a first-order invalidation of the reported numbers. The manuscript also contains an unfinished placeholder for the average overhead reduction ('X%', Section V.B) and does not release code or data, but these are secondary. A validation-based reselection is a direct, minimal test that would settle whether the headline claims survive without the leak. Because the current protocol compares an oracle-over-top-3 against a label-free random baseline and a full oracle, the central claim of near-Oracle performance with 77.68% overhead reduction is not supported as written. The framework idea has merit, but the evidence presented does not establish the claimed results.","tokens_in":18502,"tokens_out":4278,"duration_ms":40083,"concrete_test":"Re-run Module 3 (Table VII and Figures 4-5) with the deployed model chosen strictly from validation data: for Case 1 select the top-3 model with highest validation accuracy; for Cases 2 and 3 select the top-3 model with lowest validation ASR. Keep all other components identical, including the similarity-based top-3 selection. If the resulting accuracy gap to Oracle exceeds 2.02% or the ASR gap exceeds the reported ~1.9% (Case 2) and ~2.6% (Case 3), the claimed near-Oracle performance is an artifact of test-label selection. Also report the mean and worst-case ASR/accuracy across the top-3 candidates to quantify the oracle-over-top-3 effect.","verdict_should_be":"REJECT","load_bearing_attack":"The headline numerical claims rest on an evaluation protocol that peeks at test labels. In Module 3 Step 2 (Section III.D) and Algorithm 1 (lines 9, 17, 36), ReLATE+ evaluates the three transferred models on the test portion of the incoming dataset and keeps the best performer. The reported 'ReLATE+ performance' is therefore an oracle over the top-3 candidate set, not a model chosen from data available at deployment time (training and validation only). This makes the comparison asymmetric: the random baseline draws a single model uniformly without test labels and averages over 1,000 draws, while ReLATE+ is allowed to select the best of three candidates using test labels. Oracle is also defined as the best model on the test data over all 14 models. Thus the reported 2.02% average gap to Oracle and 13.28% improvement over random selection are upper bounds of what a genuinely deployable ReLATE+ would achieve; a version forced to choose one model before seeing test labels would likely land closer to the median of the top-3, not the max. The transferability assumption (Section III.D Step 1) is also under-tested, but the test-label selection alone invalidates the central claim as stated. A secondary artifact: Section V.B contains an unresolved placeholder ('X%') for the overall overhead reduction, which further undermines the 77.68% figure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ReLATE+ is a framework for multivariate time-series classification under adversarial attacks that combines three stages: (1) detection of clean, fully attacked, or partially attacked inputs using Fourier and Wavelet transforms; (2) classification of detected attacks into two groups via an XGBoost classifier trained on the performance benchmark database; and (3) selection of the most resilient model by matching the incoming dataset's CNN-based embeddings to the most similar dataset in the benchmark repository and reusing that dataset's top three models. The paper reports average computational overhead reduction of 77.68%, accuracy within 2.02% of the Oracle, and a 13.28% improvement over random model selection across six UEA datasets and seven adversarial attacks.","tokens_in":18812,"tokens_out":4652,"duration_ms":43746,"significance":"If the reported performance held under a deployment-relevant protocol, the detect-classify-select pipeline would be a useful practical contribution: it addresses the real cost of exhaustive model training and testing in time-series classification under adversarial threat. The paper has some commendable components: an explicit pseudo-code algorithm, an epsilon ablation, a comparison of three similarity metrics, and a clear separation of clean, fully attacked, and partially attacked cases. However, the central quantitative claims are not established because the final model is selected using test-set labels. In addition, the overhead section contains an unresolved placeholder, and no code or data availability statement is provided, so the experimental results cannot be independently audited. The framework's core idea is plausible, but the evidence as presented is insufficient to support the headline numbers.","major_comments":[{"comment":"Step 2 of Module 3 and Algorithm 1 (lines 8, 17, and 36) select the deployed model by evaluating the three transferred candidates on the test portion of the incoming dataset (testin) and keeping the best performer. This is test-label leakage: the random baseline averages over 1,000 uniformly chosen models without seeing test labels, Oracle is the best of all 14 models on the test set, and ReLATE+ is effectively the best of the top-3 candidates on the test set. The reported 2.02% gap to Oracle and 13.28% improvement over random selection are therefore upper bounds for any deployment-time system. The authors should either select the model using only the validation portion and report test performance of that single selected model, or report the distribution (e.g., min, median, max) of the top-3 candidates instead of only the best.","section":"Section V.A.3 / Algorithm 1"},{"comment":"The overhead analysis contains an unresolved placeholder: 'Even when including ReLATE+’s internal framework costs, the overall reduction remains high at X% on average.' The abstract and conclusion state a 77.68% average overhead reduction, but the only per-case reductions reported in Section V.B are 81.07%, 73.21%, and 75.73%, which average to about 76.67%, and the overall figure is never computed. As written, the 77.68% claim is not reproducible from the paper and must be either corrected with the actual computation or removed.","section":"Section V.B"},{"comment":"The detection threshold T = 13% is tuned on validation sets 'within the PBD' and then evaluated in a leave-one-out rotation over the same six datasets in Section V.A. It is not stated whether the validation sets used for threshold selection overlap with the validation sets of datasets that later play the role of the incoming dataset; if they do, the detection rates in Tables II-IV are optimistically biased. Additionally, Algorithm 1 loop over all D in B does not explicitly exclude the incoming dataset's own precomputed PBD entry from the similarity and top-3 ranking; the text says the remaining datasets form the drive, but the algorithm and the description should state this exclusion explicitly, since otherwise self-similarity would trivially select the incoming dataset's own best models.","section":"Section III.E.2 / Section V.A"}],"minor_comments":[{"comment":"There are typos and naming inconsistencies: Figure 2 uses 'ReLATE' in the legend while the framework is 'ReLATE+', and Figure 4 labels the y-axis 'Attack Sucess Rate' instead of 'Attack Success Rate'.","section":"Figures 2 and 4"},{"comment":"The last column header is formatted as 'A WR' and should read 'AWR' for ArticularyWordRecognition; this typo also appears in the table's row entries.","section":"Table III"},{"comment":"The statement that Module 2 'requires only 3% of the Oracle’s cost' is presented without any measurement basis; the paper should specify how this percentage is computed or cite the corresponding timing measurement.","section":"Section V.B"},{"comment":"No code, data, or reproducibility statement is provided, which makes it impossible to verify the reported tables and figures; a public artifact or detailed per-dataset results would be needed for a serious journal.","section":"Overall"}],"recommendation":"reject","confidential_remarks":"The test-label leakage in Algorithm 1 is the decisive issue: the headline numbers compare an oracle over the top-3 candidates against a genuinely blind random baseline. This is not a local presentation flaw, and correcting it requires rerunning the entire experimental evaluation. The unresolved 'X%' placeholder in Section V.B and the lack of a reproducibility statement reinforce the impression that the manuscript is not ready for publication in its current form. The authors could potentially resubmit a substantially revised version that selects models from validation data only and reports all top-3 outcomes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the detection and attack-group classification parts are a decent engineering contribution, but the model-selection evaluation is compromised because the protocol picks the deployed model using test-set labels. The headline numbers—2.02% from Oracle, 13.28% over random—are therefore upper bounds, not verified deployment performance.\n\nWhat's actually new: ReLATE+ extends the authors' prior ReLATE with a three-way clean/partial/full attack detection, an attack-group classifier (XGBoost on two broad groups), and a similarity-based top-3 model transfer that avoids testing all 14 models. The Fourier/Wavelet fusion is a sensible idea, and the detection results in Tables II–IV show the two transforms do complement each other, e.g., BasicMotions where Fourier fails and Wavelet succeeds, and UWave DeepFool where it's the reverse. The overhead-reduction concept—only evaluate three models from the most similar dataset—is practical and worth exploring.\n\nThe soft spot is not minor. In Module 3 Step 2 and Algorithm 1 (lines 9, 17, 36), ReLATE+ evaluates the three transferred models on the test portion of the incoming dataset and selects the best. The random baseline draws a single model uniformly without test labels. So the comparison is asymmetric: ReLATE+ is an oracle over a top-3 set, random is a single draw. A deployable version that chooses one model before seeing test labels would land closer to the median of the top-3, and the 2.02%-gap claim collapses. This is a load-bearing flaw; it doesn't invalidate the detection modules, but it does invalidate the central model-selection claim as stated.\n\nSecondary issues: the overhead section contains a literal 'X%' placeholder, so the 77.68% average reduction is not actually computed in the text. No code or data is released, and the repository is only six UEA datasets, mostly HAR and motion, so the cross-domain generalization claims are thin. The 13% detection threshold is tuned on validation sets from the same repository, which is acceptable but worth noting as the only tuned quantity.\n\nWho's this for? Readers working on adversarial robustness or efficient model selection for time series can learn from the detection design and the practical framing. But they should not rely on the near-Oracle result. The paper needs a serious referee because the flaw is fixable and the framework has merit, but as written the evidence does not support the headline numbers.","headline":"The attack detection and group classification are worth reading, but the headline near-Oracle model-selection results are unverified because the evaluation protocol selects the deployed model using test-set labels, and the overhead section is unfinished.","tokens_in":19306,"tokens_out":4258,"would_cite":false,"duration_ms":36325,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReLATE+ claims that a detect-classify-select pipeline can choose near-Oracle time-series classifiers while cutting computation by 77.68%.","keywords":["time-series classification","adversarial attacks","model selection","dataset similarity","Fourier and wavelet detection","computational overhead reduction","multivariate time series","resilient machine learning"],"falsifier":"Run the framework on a held-out dataset that is embedding-similar to one repository dataset but is known to favor a different model family; if the transferred top-three models underperform random selection, the similarity-transfer premise fails. A direct check is to compare the framework's top-three ranking with the true ranking obtained from exhaustive training on many held-out datasets and measure how often the true best model is absent from the chosen three.","tokens_in":18348,"feed_emoji":"🛡️","tokens_out":6842,"duration_ms":57926,"temperature":0.7,"pith_summary":"ReLATE+ is a framework for multivariate time-series classification that tries to make deep-learning model selection cheap and attack-aware. The central claim is that a three-stage pipeline—detect whether incoming data is clean, partially attacked, or fully attacked; classify any detected attack into one of two structural families; then select the best-performing models from the benchmark dataset whose CNN embeddings are most similar—can match the Oracle, the best model found by exhaustive training, almost exactly. Across six multivariate time-series datasets, the paper reports an average 77.68% reduction in computational overhead while staying within 2.02% of Oracle performance and beating random model selection by 13.28%. If this holds, real-time and resource-constrained systems can deploy resilient classifiers without retraining a large model zoo on every new data stream.","feed_headline":"Near-Oracle model picks with 77.68% less compute","feed_subtitle":"Reusing models from the most similar dataset keeps accuracy within 2.02% of the exhaustive best.","key_machinery":"The load-bearing mechanism is dataset-level embedding similarity: a small one-dimensional CNN trained on the training split of each dataset yields feature embeddings, which are L2-normalized and compared with cosine similarity. The most similar benchmark dataset's top three models—ranked by accuracy for clean data and by attack success rate for attacked data—are transferred to the new dataset. Two auxiliary mechanisms support this transfer: a max-of-two Fourier and Wavelet detector with a 13% threshold that splits data into clean, partial, or full attack, and an XGBoost attack-group classifier that maps attacks into two coarse families so the similarity search can be restricted to plausible attack conditions.","core_discovery":"The core discovery is that model performance on an unseen time-series dataset can be predicted well enough by the performance of models on the most embedding-similar dataset in a precomputed benchmark. ReLATE+ formalizes this as a detect-classify-select loop: Fourier and Wavelet anomaly scores are combined by taking their maximum to label data as clean, partially attacked, or fully attacked; an XGBoost classifier trained on the benchmark assigns detected attacks to either an iterative-gradient family or an optimization-and-decision family; and a lightweight CNN trained on each dataset produces L2-normalized embeddings compared by cosine similarity to find the nearest benchmark dataset. The top three models from that dataset are then evaluated, and the best among them is deployed. In leave-one-out experiments on six multivariate time-series datasets, the paper reports that this procedure lands within 2.02% of the Oracle, improves on random model selection by 13.28% on average, and reduces total computational overhead by 77.68%.","pith_inferences":["The reported savings should grow with repository size, because ReLATE+ evaluates a fixed top three models while Oracle cost scales with the number of candidate models; this scaling property is not tested in the paper.","A natural stress test is a truly out-of-distribution dataset with no close embedding match: the framework should degrade toward random selection rather than toward Oracle, and the paper does not report this regime.","The max-of-two Fourier and Wavelet detector implies that any attack leaving no detectable trace in either spectral or time-frequency domain would evade Module 1; the epsilon-0.01 ablation already hints at this boundary.","The attack-group abstraction suggests a testable design principle for defenses: perturbation geometry, not the named attack, is what determines which model family survives, which could simplify future robustness benchmarks."],"forward_implications":["Practitioners can avoid training all candidate models on each new dataset; only the top three models from the most similar known dataset need to be evaluated.","The detection and attack-group stages are lightweight enough that the end-to-end overhead saving of 77.68% survives in streaming settings where retraining cycles are impractical.","Exact attack identity is not required: grouping attacks into iterative-gradient versus optimization-and-decision families is enough to guide resilient model selection.","The similarity transfer appears to work across domains, since the six benchmark datasets cover different application areas and the reported gap to Oracle stays near 2%.","The framework turns a model repository into a reusable asset: each new dataset enriches the benchmark and can improve future selections without retraining earlier entries."],"supporting_citations":[{"why":"Supplies the taxonomy and implementations of the seven adversarial attacks used to populate the performance benchmark.","marker":"[18]"},{"why":"Supplies the MLP, FCN, and ResNet architectures that anchor the 14-model repository.","marker":"[40]"},{"why":"Supplies the LSTM, GRU, and LSTM-FCN baselines whose performance is recorded in the benchmark.","marker":"[41]"},{"why":"Supplies InceptionTime, one of the candidate deep models compared in the model-selection study.","marker":"[45]"},{"why":"Supplies OmniScaleCNN, another candidate model whose robustness is ranked in the repository.","marker":"[47]"},{"why":"Provides the XGBoost classifier used in Module 2 to assign detected attacks to one of two groups.","marker":"[51]"},{"why":"Provides the six multivariate time-series datasets that form the benchmark database and the leave-one-out evaluation setup.","marker":"[52]"},{"why":"Dynamic time warping serves as one alternative similarity metric against which CNN-plus-cosine is compared.","marker":"[54]"},{"why":"Wasserstein distance serves as the other alternative similarity metric in the comparison study.","marker":"[55]"}],"fun_headline_variants":["ReLATE+ cuts compute 77.68% while staying within 2% of Oracle","Attack-aware reuse: 77.68% less compute, 2.02% from Oracle","Detect, classify, reuse: ReLATE+ saves 77.68% compute, near-Oracle","Similarity-based model selection slashes compute 77.68%","ReLATE+ reuses models from similar datasets, cutting compute 77.68%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that when two datasets produce similar feature embeddings, the models that worked best on the old dataset will also work best on the new one.","fun_headline_variants_meta":{"raw":{"variants":["ReLATE+ cuts compute 77.68% while staying within 2% of Oracle","Attack-aware reuse: 77.68% less compute, 2.02% from Oracle","Detect, classify, reuse: ReLATE+ saves 77.68% compute, near-Oracle","Similarity-based model selection slashes compute 77.68%","ReLATE+ reuses models from similar datasets, cutting compute 77.68%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001057,"raw_usage":{"total_tokens":4445,"prompt_tokens":965,"completion_tokens":3480,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":3363}},"tokens_in":581,"tokens_out":3480,"duration_ms":24694,"temperature":1.0,"reasoning_tokens":3363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:52:46.939896+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the framework on a held-out dataset that is embedding-similar to one repository dataset but is known to favor a different model family; if the transferred top-three models underperform random selection, the similarity-transfer premise fails. A direct check is to compare the framework's top-three ranking with the true ranking obtained from exhaustive training on many held-out datasets and measure how often the true best model is absent from the chosen three.","supporting_citations":[{"cited_title":"How deep learning sees the world: A survey on adversarial attacks & defenses,","cited_arxiv_id":null,"evidence_quote":"Supplies the taxonomy and implementations of the seven adversarial attacks used to populate the performance benchmark."},{"cited_title":"Time series classification from scratch with deep neural networks: A strong baseline,","cited_arxiv_id":null,"evidence_quote":"Supplies the MLP, FCN, and ResNet architectures that anchor the 14-model repository."},{"cited_title":"Lstm fully convolutional networks for time series classification,","cited_arxiv_id":null,"evidence_quote":"Supplies the LSTM, GRU, and LSTM-FCN baselines whose performance is recorded in the benchmark."},{"cited_title":"Inceptiontime: Finding alexnet for time series classification,","cited_arxiv_id":null,"evidence_quote":"Supplies InceptionTime, one of the candidate deep models compared in the model-selection study."},{"cited_title":"Xgboost: A scalable tree boosting system,","cited_arxiv_id":null,"evidence_quote":"Provides the XGBoost classifier used in Module 2 to assign detected attacks to one of two groups."},{"cited_title":"Uea multivariate time series classification repository,","cited_arxiv_id":null,"evidence_quote":"Provides the six multivariate time-series datasets that form the benchmark database and the leave-one-out evaluation setup."},{"cited_title":"Wasserstein discriminant analysis,","cited_arxiv_id":null,"evidence_quote":"Wasserstein distance serves as the other alternative similarity metric in the comparison study."}],"review_version":1}