{"id":"9b79a7e5-6a67-4976-8a29-6a377acc39c0","arxiv_id":"2506.01450","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ShaTS computes Shapley attributions directly on semantic groups of time-series features, improving sensor- and process-level anomaly explanations over post hoc SHAP on the SWaT dataset.","lead":"A new explainability module computes Shapley values over predefined groups of time-series features, such as time instants, sensors, or processes, instead of summing feature-level attributions after the fact. Tests on the SWaT water-treatment dataset suggest the grouped approach gives more focused anomaly explanations than post hoc SHAP and runs faster.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim of outperforming SHAP rests on a top-1 accuracy metric whose counting rule is inconsistent with Table 4 and lacks a baseline, so the quantitative superiority claim is not yet supported.","rationale":"The reader's weakest_assumption and my analysis converge on the same load-bearing concern: the top-1 metric used to compare ShaTS with SHAP is heuristic, inconsistently applied, and without a baseline. The reader already recommends CONDITIONAL, and my recomputation of Table 4 strengthens that recommendation rather than changing it. I considered other possible concerns: the theory sections are standard Shapley material with correct equations; the approximation method follows Maleki et al.; the model is a reasonably evaluated bidirectional LSTM; the resource comparison is informative even if budget-unfair. The main genuine soft spot is the evaluation of explanation quality. The paper self-reports 14 sensor-level and 19 process-level hits for ShaTS, but Table 4 shows attacks where the attacked sensor is not top-ranked (e.g., Attack 1: true MV101, top LIT101; Attack 4: true MV504, top PIT502; Attack 21: true MV101/LIT101, top AIT503; Attack 26: true P101/LIT301, top P102; Attack 30: true LIT101/P101/MV201, top P102; Attack 34: true P101, top P101 is ranked second behind MV101). Claims in Section 5.5.1 that ShaTS 'correctly identified the affected sensor or actuator in 14 cases' are thus not reproducible from the table under the stated rule. A strict top-1 read yields about 12 sensor hits; even if one counts any attacked sensor in the top-3, ShaTS still lags on several single-target attacks. The absence of a random baseline and significance testing is the more fundamental issue: with 44 sensors and 6 processes, expected chance top-1 hits are 21/44 ≈ 0.5 sensors and 3.5 processes, so the 19/21 process result is strong, but the 14/21 sensor result is the key claimed advantage over SHAP's 6/21 and needs a permutation test. The resource comparison also mixes subset counts, but this is secondary because the qualitative scalability result for window size is plausible. Overall, the method is sound and the paper is valuable as an engineering contribution, but the headline superiority claim should be reworded to acknowledge the heuristic nature of the evaluation until a baseline and significance test are added.","tokens_in":22612,"tokens_out":4185,"duration_ms":34151,"concrete_test":"Recompute Table 4 with a strict top-1 rule: for each of the 21 detected attacks, count ShaTS and SHAP as correct only if the single top-ranked sensor/actuator is among the attack points listed in the 'Attack Point' column, and similarly for processes. Then run a permutation test: shuffle the group labels within each attack (or permute the top-1 rank ordering) 10,000 times and compare the observed 14-vs-6 sensor margin to the chance distribution, reporting a p-value. If the strict count drops to 12 or below and the margin is not significant against the permutation baseline, the claim that ShaTS outperforms SHAP in explainability should be downgraded to a qualitative/demonstrative claim pending operator validation or a stronger metric.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that ShaTS provides more precise explanations and outperforms SHAP (abstract, Section 5.5.1, Section 6). The evidence is a top-1 comparison: for each of 21 detected attacks, the method with the highest average normalized attribution at the attacked sensor/process is said to be correct. The reader flags this as the weakest assumption, and it is indeed load-bearing. First, the counting rule is not consistently applied: for multi-target attacks (e.g., Attack 23: P602, DPIT301, MV302; Attack 26: P101, LIT301; Attack 27: P302, LIT401; Attack 30: LIT101, P101, MV201; Attack 37: P501, FIT502; Attack 39: FIT401, AIT502), ShaTS is scored correct only when it ranks one of the attacked sensors first, yet for single-target attacks like Attack 4 (MV504, an actuator not listed in Table 2) and Attack 21 (MV101/LIT101), the paper still reports a top-1 detection even though no attacked sensor is ranked first (Attack 4: PIT502; Attack 21: AIT503). Recomputing Table 4 under the paper's own top-1 rule gives roughly 12 of 21 for ShaTS and about 5 of 21 for SHAP, not the quoted 14 and 6. Second, the metric has no baseline: a random or frequency-based sensor baseline could plausibly achieve 3-6 top-1 hits because P1, P3, and P5 sensors appear often as targets. Third, the top-1 average normalized attribution is sensitive to the normalization choice (dividing by the sum of all group values in a window can amplify small differences) and no variance or significance testing is reported over windows or background samples. The resource comparison also uses unequal budgets: m=3428 subsets for KernelSHAP vs m=880 for ShaTS sensor grouping, so the efficiency gap partly reflects different total coalition evaluations, not just grouping. These are addressable, but the abstract-level claim of outperforming SHAP is currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ShaTS, a model-agnostic Shapley-based explanation method for time-series models. ShaTS computes Shapley values on a priori defined feature groups (time instants, individual sensors/actuators, or industrial processes) rather than on raw features, with an exact formula and a stratified-sampling approximation. The method is integrated into an LSTM-based anomaly-detection pipeline and evaluated on the SWaT benchmark against KernelSHAP with post hoc grouping. The authors report that ShaTS better identifies attacked sensors/actuators and processes, produces temporally consistent explanations, and is substantially more resource-efficient. The resource-efficiency experiments are detailed; the explainability comparison, however, rests on a top-1 hit-rate metric whose counting rule is inconsistent with the reported table and which lacks a chance baseline or significance testing.","tokens_in":22938,"tokens_out":14023,"duration_ms":139208,"significance":"If the evaluation issues are resolved, the contribution is a useful and principled extension of Shapley values to a priori feature groups for time-series models. The core Shapley equations are standard and correctly presented, the sampling estimator is a published approach, and no parameters are fitted to the benchmark. The open-source release and detailed resource measurements are strengths. The claimed superiority over SHAP in explainability, however, is not yet supported: the sensor-level counts in Section 5.5.1 and Table 4 are not reproducible under the paper's own top-1 rule, and the absence of a random or frequency baseline makes it difficult to interpret even the corrected hit rates. The temporal-consistency claim is supported only by visual inspection.","major_comments":[{"comment":"The headline sensor-level counts (ShaTS 14/21, SHAP 6/21) cannot be reproduced from Table 4 under the stated rule that the top-ranked component is the one with the highest average contribution. Applying that rule literally to the table's top entries yields 13 ShaTS hits (attacks 2, 6, 7, 8, 10, 19, 23, 27, 31, 32, 34, 36, 39) and 5 SHAP hits (attacks 2, 6, 8, 23, 39). If the intended rule instead counts any attacked element appearing anywhere in the printed top-3, the counts are 15 and 8. Neither matches 14/6. The authors must state the exact counting convention, correct the counts accordingly, and provide per-attack decision lists in a machine-readable form. This is load-bearing because the claimed explainability advantage over SHAP rests entirely on this comparison.","section":"Section 5.5.1 / Table 4"},{"comment":"The top-1 hit-rate comparison has no chance baseline and no uncertainty quantification. With 21 attacks and 44 possible sensors, a uniform random top-1 guess would be expected to hit about 0.5 attacks; a frequency baseline that always selects the most frequently attacked sensor would also score several hits by construction, and the reported SHAP count of 6 is close to such a baseline. The authors should report a random baseline, a frequency baseline, and confidence intervals or a permutation test over attack labels. The same concern applies to the process-level comparison, where random expectation is about 3.5 of 21. Without these, the statement that ShaTS 'outperforms SHAP' at the sensor/actuator level is not statistically supported.","section":"Section 5.5.1 (Experiment 2)"},{"comment":"The metric used for Table 4 is the average of 'normalized' contribution per group, where normalization divides by the sum of all group values in a window. Since Shapley values can be negative, dividing by their signed sum is not well defined without specifying whether absolute values are used, and small differences can be amplified by this normalization. The manuscript should specify the exact normalization (signed, absolute, or min-max) and demonstrate that the top-1 conclusions are stable under reasonable alternative normalizations.","section":"Section 5.5.1 (normalization)"},{"comment":"The temporal-consistency analysis is entirely qualitative. The claim that ShaTS 'preserves temporal dependencies' and 'revealed a clear shift in influence' is based on visual inspection of Figure 4, with no quantitative metric for either method. For example, the authors could measure the fraction of windows in which the top-ranked time instant falls in the anomalous portion of the window, or compute a rank correlation between the attribution profile and the attack boundary. Without such a metric, the temporal-superiority claim is not supported.","section":"Section 5.5.1 (Experiment 1)"}],"minor_comments":[{"comment":"Attack 4 is listed as targeting MV504, but MV504 does not appear in Table 2's process mapping, in Figure 5's sensor list, or in the 44 active sensors/actuators described in Section 5.2.3. If MV504 was removed during preprocessing or is not part of the modeled feature set, the sensor-level evaluation of Attack 4 is ill-defined and should be clarified or excluded.","section":"Table 4 / Table 2 / Figure 5"},{"comment":"The background-dataset size labels are rendered as one concatenated string (\"32016080402010\"); please use separate tick labels for readability.","section":"Figures 8 and 9"},{"comment":"The grid search lists 'Number of Epochs' as a hyperparameter but only tests the single value 5; please either report the actual range searched or remove this entry from the table.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The core method is sound and the resource-efficiency evidence is credible; the main weakness is the evaluation used to support the explainability superiority claim. I do not see grounds for rejection, but the top-1 metric must be corrected, baselined, and tested for significance before the central claim is acceptable. The paper is within the scope of the journal and the open-source release is a positive feature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nI read the ShaTS paper. The core idea is straightforward and legitimate: instead of computing Shapley values per feature and aggregating afterwards, group the features a priori — by time instant, sensor, or process — and compute Shapley values directly on those groups. This is a real difference from post hoc aggregation, it preserves the temporal structure of the input window, and it also cuts the number of Shapley values you need, because you work with 44 groups instead of 690 features. The authors implement it cleanly, the Shapley equations and the Maleki sampling approximation are standard and correctly described, and the code is on GitHub. The resource scaling experiments, where they vary coalition count and background size, are genuinely informative: ShaTS stays within GPU memory and under one second per window in configurations where KernelSHAP exceeds the limit.\n\nThe soft spot is the evaluation of explainability. The paper claims ShaTS identifies the attacked sensor in 14 of 21 attacks and SHAP in 6. I counted from Table 4 using their own top-1 rule and got 13 and 5. The discrepancy matters because the entire 'outperforms SHAP' claim rests on this top-1 heuristic. There is also no baseline: a random or frequency-based sensor might get a few hits by chance, and no significance test is reported. The resource comparison in the main test is not apples-to-apples either: KernelSHAP is run with 3428 subsets while ShaTS uses 880 for the sensor grouping, so part of the efficiency gain is just a smaller budget — although the sweep in Experiment 3 shows the trend holds across budgets. The temporal consistency analysis is visual only, which is fine as a demonstration but not a quantitative result.\n\nThese are all fixable. The method is plausible, the writing is clear, and the authors are not overclaiming in the body — the abstract and conclusion do say 'outperforms' without qualification. A good referee would ask for a corrected table, a random or permutation baseline, and confidence intervals on the top-1 accuracy. That is a manageable revision.\n\nI would bring this to a reading group if you are doing time-series xAI, and I would cite it for the a priori grouping idea. It deserves peer review, not desk rejection.","headline":"ShaTS is a clean engineering idea with an open-source implementation, but the paper's central claim of outperforming SHAP is undermined by a top-1 counting error and a missing baseline.","tokens_in":23587,"tokens_out":4149,"would_cite":true,"duration_ms":38851,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Computing Shapley values on a priori feature groups preserves temporal dependencies and identifies the attacked process first in 19 of 21 SWaT attacks, against 8 for post hoc SHAP.","keywords":["Shapley values","time series explainability","anomaly detection","Industrial Internet of Things","feature grouping","SHAP","SWaT dataset","model-agnostic explanations"],"falsifier":"Count the top-1 hits in Table 4 under the paper's own rule: ShaTS must name the attacked sensor first in 14 of the 21 detected attacks (SHAP in 6) and the attacked process first in 19 (SHAP in 8); a different count would falsify the claim.","tokens_in":22438,"feed_emoji":"🔍","tokens_out":8927,"duration_ms":94261,"temperature":0.7,"pith_summary":"ShaTS is a model-agnostic explainability method that computes Shapley values on feature groups defined before the attribution step, rather than on raw features. The paper argues that this a priori grouping preserves temporal dependencies in time-series data, so explanations point directly at the time instants, sensors/actuators, or processes responsible for an anomaly instead of fragmenting importance across individual features. On the SWaT water-treatment benchmark, the method is claimed to identify the attacked sensor first in 14 of 21 detected attacks and the attacked process first in 19 of 21, against 6 and 8 for the post hoc SHAP baseline, while using less GPU memory and staying closer to the one-second real-time bound. A sympathetic reader would care because industrial operators need to know which physical component to inspect, and the paper offers a way to get that answer without sacrificing real-time feasibility.","feed_headline":"Grouped Shapley values beat SHAP on time-series anomaly attacks","feed_subtitle":"On SWaT, ShaTS names the attacked sensor first in 14 of 21 attacks (SHAP: 6) while using less memory.","key_machinery":"The load-bearing device is a priori feature grouping: partitioning the model input into groups (time instants, sensor/actuator identities, or industrial processes) before any coalition is evaluated, and then treating each group as a single Shapley player. This makes each attribution semantically interpretable and shrinks the player set from 690 raw feature values to 10, 44, or 6 players, so the approximate Shapley computation with $m = 20\\cdot|G|$ sampled coalitions stays tractable and preserves temporal structure. The approximation follows the stratified sampling scheme of reference [15], with the coalition value estimated by averaging model predictions over a background dataset.","core_discovery":"The central discovery is that where you put the grouping in Shapley computation changes both the quality and the cost of the explanation. ShaTS partitions each time window into a small set of semantically meaningful players, either the 10 instants, the 44 sensors/actuators, or the 6 SWaT processes, and then runs the Shapley formula over those groups using a sampled-coalition approximation with a 500-instance background dataset. The reported result is that this yields sharper attributions than computing KernelSHAP on all features and aggregating post hoc: the attacked sensor is the top-ranked group in 14 of 21 attacks with ShaTS versus 6 with SHAP, and the attacked process in 19 of 21 versus 8. Resource measurements show ShaTS staying within the 16 GB GPU limit and near-constant per-window execution time as window size grows, whereas the SHAP baseline often spills to CPU and exceeds the one-second real-time threshold.","pith_inferences":["I infer that the grouping principle should transfer to other multivariate time-series settings with natural groups, such as ECG leads, network flows, or weather stations; rerunning the same protocol on a second labeled dataset would test this.","A natural next evaluation, not reported in the paper, would have operators rank explanations or would compare ShaTS against a random/permutation baseline; that would test whether the top-1 metric is truly measuring explanation quality.","The resource advantage is largely structural, since grouping reduces the number of Shapley players; other Shapley estimators that accept grouped players would likely show similar gains, independent of the specific sampling scheme used here."],"forward_implications":["If the paper's results hold, an anomaly detector can be paired with an explanation service that tells the operator which process and which sensor/actuator to inspect while the attack is still ongoing.","At the sensor and process level, explanation cost stops growing with the window length, because the number of players is fixed by the grouping; this makes longer context windows practical for real-time monitoring.","The method is model-agnostic, so the same a priori grouping device can be added to LSTM, CNN, or GRU anomaly detectors without retraining them.","The comparison suggests that post hoc aggregation of per-feature Shapley values, the common practice in the reviewed literature, systematically dilutes the attribution to the true attacked component."],"supporting_citations":[{"why":"Defines KernelSHAP, the standard Shapley approximation used as the post hoc comparison baseline throughout the experiments.","marker":"[10]"},{"why":"Supplies the stratified sampling-based Shapley approximation that ShaTS uses to bound the number of evaluated coalitions.","marker":"[15]"},{"why":"Provides the SWaT water treatment dataset and its attack labels, the empirical ground for all reported explanation and resource results.","marker":"[29]"},{"why":"Prior work on a priori feature grouping and causal inference in LSTM anomaly detection that ShaTS extends and compares against conceptually.","marker":"[28]"}],"fun_headline_variants":["ShaTS top-ranks attacked sensor 14/21 vs SHAP's 6 on SWaT","ShaTS groups time steps to sharpen Shapley, cut memory use","SWaT: ShaTS finds attacked process 19/21 times, SHAP 8","ShaTS keeps Shapley real-time: near-constant GPU time per window","ShaTS's temporal grouping improves Shapley accuracy and efficiency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the SWaT attack labels are the ground truth for a good explanation and measures 'correct' by whether the attacked sensor or process is the top-ranked attribution; if operators or a random/permutation baseline were used instead, the reported advantage over SHAP could shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["ShaTS top-ranks attacked sensor 14/21 vs SHAP's 6 on SWaT","ShaTS groups time steps to sharpen Shapley, cut memory use","SWaT: ShaTS finds attacked process 19/21 times, SHAP 8","ShaTS keeps Shapley real-time: near-constant GPU time per window","ShaTS's temporal grouping improves Shapley accuracy and efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000716,"raw_usage":{"total_tokens":3224,"prompt_tokens":954,"completion_tokens":2270,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":2161}},"tokens_in":570,"tokens_out":2270,"duration_ms":14666,"temperature":1.0,"reasoning_tokens":2161,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:43:08.581870+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count the top-1 hits in Table 4 under the paper's own rule: ShaTS must name the attacked sensor first in 14 of the 21 detected attacks (SHAP in 6) and the attacked process first in 19 (SHAP in 8); a different count would falsify the claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SWaT water treatment dataset and its attack labels, the empirical ground for all reported explanation and resource results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work on a priori feature grouping and causal inference in LSTM anomaly detection that ShaTS extends and compares against conceptually."}],"review_version":1}