{"id":"b055c19c-163c-4318-ac7f-5328600de4a7","arxiv_id":"2412.04532","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"WinTSR multiplies feature ablation scores by a time relevance score obtained by masking all features at each lag, improving faithfulness metrics across five models and three datasets.","lead":"WinTSR is a new way to explain time series predictions by rescaling feature importance with a time-level relevance score, computed by masking all features at each past time step. It is tested on three real datasets and five modern forecasting models, and claims to beat ten existing interpretation methods on faithfulness metrics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"WinTSR's top ranking may be an artifact of evaluation: its importance scores are defined by the same masking/perturbation operation used in the comprehensiveness/sufficiency metrics, and no synthetic ground-truth experiment supports the 'significantly outperforms' claim.","rationale":"I agree with the reader's identified weakest assumption: the comprehensiveness/sufficiency protocol is the load-bearing premise. I sharpen it to a specific alignment: the same masking operation defines both WinTSR's scores and the evaluation metric, so the benchmark partly measures self-consistency. This is a correctness risk, not just a consensus disagreement. The paper deserves credit for the open-source framework, the breadth of models/datasets, and the transparency of the method; those contributions are independent of the headline ranking claim. However, the claim 'significantly outperforms' needs a ground-truth anchor. The proposed synthetic experiment is cheap and decisive: it directly tests whether the method recovers known important features. If it fails, the authors should soften the claim to 'achieves the best perturbation-based faithfulness scores under the masking proxy' and add significance testing. My recommendation is to keep the reader's CONDITIONAL verdict but make the condition explicit: acceptance requires synthetic ground-truth validation or an alternative unbiased evaluation. Thus verdict_should_be = UNCHANGED (the reader already reached CONDITIONAL).","tokens_in":15841,"tokens_out":9176,"duration_ms":97800,"concrete_test":"Create a synthetic forecasting task with known ground-truth feature-time importance: y_t = 2·x_{1,t-1} + x_{2,t-24} + ε, with x drawn as standard normal and ε noise, such that only feature 1 at lag 1 and feature 2 at lag 24 are relevant. Train the five models (DLinear, MICN, SegRNN, iTransformer, CALF) on this data, then run WinTSR and all ten baseline methods on a test set. Compute top-5% and top-10% precision/recall against the ground-truth (feature, lag) pairs, and also report the paper's AOPC/AOPCR metrics. If WinTSR does not show significantly better ground-truth recovery than Feature Ablation or Integrated Gradients, the benchmark advantage in Table 4 is an artifact of metric-method alignment rather than explanatory fidelity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that WinTSR 'significantly outperforms other local interpretation methods' (Abstract). The evidence is Table 4, where WinTSR has the best average rank on all three datasets. This evidence is not persuasive because the faithfulness metrics are aligned with the method's construction. In Algorithm 1, the importance score φ_{j,l} is defined as the product of the output change when feature (j,l) is masked and the output change when the entire time step l is masked. The evaluation in Section 4.4 then masks the top-k% of these scores and measures the output change. Thus WinTSR is explicitly built from the same perturbation operation that the benchmark rewards; methods such as Integrated Gradients, GradientSHAP, and DynaMask optimize different objectives and are systematically disadvantaged. Moreover, the method is not guaranteed to maximize the perturbation metric even among perturbation methods: in a linear model with weights w_{j,l}, the product reweights each feature by the aggregate time-step effect, so a feature with small weight at a strongly-perturbed time step can outrank a feature with larger weight at a quiet time step, contradicting the model's linear truth. No synthetic ground-truth experiment is included to validate that the perturbation proxy tracks true importance; the paper criticizes prior work for using synthetic datasets but does not provide a ground-truth check itself. Finally, 'significantly' is not supported statistically: no significance tests or error bars on AOPC/AOPCR values are reported, only rank means and standard deviations. The combination of metric alignment, a heuristic product, and absence of ground-truth validation makes the headline claim conditionally supported at best.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes WinTSR, a local post-hoc, model-agnostic interpretation method for multivariate time series models. WinTSR computes a time-relevance score for each look-back step by masking all features at that step and measuring the output change, then computes per-feature relevance by masking each feature individually and multiplies the two quantities to form the final importance score. The method is benchmarked on three real-world datasets (Electricity, Traffic, MIMIC-III) across five model architectures (DLinear, MICN, SegRNN, iTransformer, CALF) and compared with ten existing interpretation methods, using comprehensiveness and sufficiency derived from masking the top-k% features. The authors report average ranks showing WinTSR first on all three datasets and claim that it 'significantly outperforms' other local interpretation methods, while also releasing an open-source framework.","tokens_in":16092,"tokens_out":4273,"duration_ms":42469,"significance":"If the central claim is validated, WinTSR would be a useful, cheap, and model-agnostic interpretation tool for modern time series forecasting and classification models, and the companion open-source framework is a clear contribution to reproducibility and ease of benchmarking. The experimental scope, covering five architectures including a foundation model, is substantially wider than most prior work in this area. However, the headline claim rests on an evaluation protocol that is structurally aligned with the method's own definition, and the absence of statistical significance testing and of a ground-truth synthetic benchmark leaves the central claim unsupported. The paper's value as a framework and a careful empirical survey is real, but the evidence for 'significantly outperforms' is not yet convincing.","major_comments":[{"comment":"The abstract and Section 5.1 claim that WinTSR 'significantly outperforms' other methods, but no statistical significance testing is provided. Table 4 reports AOPCR/AOPC values and average ranks with standard deviations, but there are no confidence intervals on the AOPC/AOPCR values, no error bars over the three iterations mentioned in Table 8, and no paired test (e.g., Wilcoxon signed-rank) across models or datasets. The reported standard deviations on ranks reflect variation across the five models, not the sampling uncertainty of the comparison. Please add significance tests and report variance across seeds or bootstrap resamples.","section":"5.1, Table 4"},{"comment":"The evaluation metrics in Section 4.4 (comprehensiveness and sufficiency) are computed by masking the top-k% features and measuring the change in model output. WinTSR's importance scores in Algorithm 1 are defined by the same masking-and-output-change operation: the feature relevance is |f(Xt) − f(Xt \\ x_{j,l})|. Thus, high comprehensiveness/sufficiency scores may partly reflect self-consistency between the method's definition and the evaluation metric, rather than external explanatory fidelity. Gradient-based and learned-mask methods are not constructed from this exact masking operation and are systematically disadvantaged. To support the claim, the authors should include a synthetic experiment with known ground-truth importance (e.g., a linear model or a controlled nonlinear generative process) and show that WinTSR recovers the true importance better than the baselines.","section":"3.2, Algorithm 1 vs 4.4"},{"comment":"The product form of the importance score distorts feature ranking even in simple models. For a linear model f = Σ_{j,l} w_{j,l} x_{j,l}, the time-relevance score is Δtime_l = |Σ_j w_{j,l}(x_{j,l} − b_{j,l})|, so the final score φ_{j,l} = |w_{j,l}(x_{j,l} − b_{j,l})| × Δtime_l. This double-counts the aggregate effect of a time step, so a feature with small weight at a strongly perturbed time step can outrank a feature with larger weight at a quiet time step, contradicting the model's actual linear dependence. The paper does not include a synthetic linear benchmark to test whether this rescaling improves or distorts the recovered importance ranking. Please provide such a controlled experiment.","section":"3.2, Eq. (2) and Algorithm 1"},{"comment":"The ranking procedure is not fully specified. The text says methods are ranked 'for each dataset and model in terms of overall comprehensiveness and Sufficiency' and then ranks are averaged, but it does not state whether comprehensiveness rank and sufficiency rank are averaged equally, how ties are broken, or whether the rank is computed per model before averaging. The conclusion is sensitive to this aggregation: for MIMIC-III, TSR has much higher comprehensiveness (0.80–0.90 vs WinTSR's 0.50–0.56) yet ranks 5th because of poor sufficiency. Please define the ranking rule explicitly and also report per-metric ranks separately.","section":"5.1, Table 4"}],"minor_comments":[{"comment":"The model type labels in Table 5 are inconsistent with Section 4.2: MICN is the CNN-based model and SegRNN is the RNN-based model, but the table lists 'MICN (RNN)' and 'SegRNN (CNN)'. Please correct the labels.","section":"Table 5"},{"comment":"The 'featuregenerator()' baseline is not described precisely. The text says random values are drawn from a normal distribution, but the distribution parameters and whether the baseline is generated per instance or fixed per run should be stated. A sensitivity analysis of the baseline distribution would also strengthen the method's robustness claim.","section":"Algorithm 1, Section 3.2"},{"comment":"The MIMIC-III row in Table 6 is labeled 'Lookback = 36hr' while the column headers say 24hr and 48hr, and the text states that the lookback was varied to 24 and 36 hours for MIMIC-III. Please align the table header with the text.","section":"5.4, Table 6"},{"comment":"The Electricity and Traffic datasets are listed as having five features, but these are one target series plus four time-encoded features (month, day, hour, day of week). This should be clarified in the main text, since the regression benchmark thus has very low feature dimensionality, which limits the generality of the conclusions for high-dimensional multivariate inputs.","section":"Table 2 and Section A"},{"comment":"The training parameters report 'iterations 3' but the paper does not state how the three iterations are used (e.g., multiple random seeds, multiple runs, or multiple train/test splits). Please clarify and report the resulting variance in Table 4.","section":"Appendix B, Table 8"}],"recommendation":"major_revision","confidential_remarks":"The paper's main weakness is that the headline claim of 'significantly outperforms' is not supported by the evidence: the evaluation protocol is structurally aligned with the method's construction, no statistical tests are performed, and no synthetic ground-truth experiment is included. This is fixable within the scope of the manuscript by adding a synthetic benchmark, significance testing, and a clearer ranking definition. The open-source framework and breadth of model coverage are notable strengths that justify further review after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"WinTSR is a simple, fast, model-agnostic perturbation method that rescales per-feature ablation importance by a per-time-step relevance score. Its real value is the package: a clean public framework that benchmarks 10+ interpretation methods against 5 modern architectures, including a foundation model, on 3 real datasets. That is useful work. But the paper's headline claim that WinTSR 'significantly outperforms' the alternatives is not backed by the evidence as written.\n\nThe novelty is a genuine simplification of TSR. Instead of running a second interpretation method and comparing importance matrices, WinTSR computes a time relevance score by masking all features at a given time step and multiplies it into the feature-level ablation score. That removes TSR's main computational burden; the runtime table shows 32–367x speedups over TSR.\n\nThe load-bearing problem is the evaluation. Comprehensiveness and sufficiency are measured by masking the top-k% of the importance scores and observing output change. WinTSR's scores are defined by the same masking operation, so the ranking partly measures self-consistency rather than independent explanatory power. This is a known hazard for all perturbation metrics; it doesn't sink the paper, but it means 'significantly outperforms' is really 'outperforms on this perturbation proxy.' Second, there are no significance tests or confidence intervals on the AOPC/AOPCR values, only rank means and standard deviations. Third, the product heuristic has an actual distortion: in a linear model, a feature with a small weight at a strongly perturbed time step can outrank a larger-weight feature at a quiet step. A synthetic ground-truth experiment—which the paper criticizes others for lacking—would have settled this cheaply. Fourth, the regression benchmarks use only 5 features (one target plus four time encodings), so that part of the evaluation is thin.\n\nFor readers building or evaluating time series interpretation methods, the framework and the benchmark are worth having. I would cite it. But the central claim needs revision: either add statistical rigor and a ground-truth sanity check, or state the result as consistent with prior work on perturbation metrics. The paper deserves a serious referee, with the expectation of major revision rather than acceptance as-is.","headline":"A genuinely useful framework and a simple, fast method, but the 'significantly outperforms' claim outruns the evidence: the faithfulness proxy is aligned with the method's own masking, and there are no significance tests.","tokens_in":16672,"tokens_out":3420,"would_cite":true,"duration_ms":34581,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WinTSR presents a post-hoc interpretation method that rescales each feature's saliency by a time-relevance score, and reports that it outperforms ten baselines across five model architectures and three real-world datasets.","keywords":["time series interpretability","local explanation","temporal saliency rescaling","perturbation-based attribution","model-agnostic","forecasting","transformer foundation model","comprehensiveness and sufficiency"],"falsifier":"Generate synthetic data with a known causal lag structure—say output depends only on feature 1 at lag 3 and feature 2 at lag 24, with all other entries pure noise—train any of the five models, apply WinTSR, and check whether the two true (feature, lag) positions receive the top importance scores across many seeds; if an exhaustive single-feature ablation recovers the known positions while WinTSR does not, the temporal rescaling step is not doing the explanatory work claimed.","tokens_in":15621,"feed_emoji":"📈","tokens_out":8925,"duration_ms":78062,"temperature":0.7,"pith_summary":"The paper sets out to establish that a simple two-stage masking rule improves local explanations of multivariate time series models: instead of scoring each feature in isolation, it first asks how much each past time step matters to the model and then rescales every feature score by that time relevance. WinTSR masks all features at a given lag, measures the change in model output, normalizes those changes across the look-back window, and multiplies each individual feature's masking score by the resulting time weight. This makes temporal dependencies explicit in the attribution, such as the fact that the last observed hour or the same hour one day earlier often carries most of the predictive signal. The paper evaluates on three real-world datasets (electricity, traffic, and ICU mortality), five architectures including a transformer and an LLM-based foundation model, and ten comparison methods, reporting the best average rank for WinTSR in every dataset. If correct, the method gives practitioners a cheap, model-agnostic way to read which sensors and which historical moments drive a forecast or a clinical prediction, without training a surrogate model.","feed_headline":"Time-aware rescaling wins the time series explanation benchmark","feed_subtitle":"It multiplies feature saliency by a time-relevance score, beating ten baselines on forecasting and mortality tasks.","key_machinery":"The central object is the importance matrix $\\phi_t = \\{\\phi_{j,l,t}\\}$, where $\\phi_{j,l,t} = \\Delta^{\\mathrm{feature}}_{j,l,t} \\times \\Delta^{\\mathrm{time}}_{l,t}$. Here $\\Delta^{\\mathrm{time}}_{l,t}$ is the normalized magnitude of the change in model output when all features at lag $l$ are replaced by baseline values, and $\\Delta^{\\mathrm{feature}}_{j,l,t}$ is the change when only feature $j$ at lag $l$ is replaced. Both deltas are $\\ell^1$ distances between original and perturbed predictions, and the baseline values are drawn from a normal distribution. The multiplication is the rescaling mechanism: it says a feature is credited only when both the feature and the time step it occupies move the model. The paper contrasts this with TSR, which rescales using a second attribution method's output, and with WinIT, which perturbs whole sliding windows and subtracts; WinTSR's product form is what carries the claimed gains.","core_discovery":"WinTSR claims that local explanations of multivariate time series are more faithful when feature importance is multiplied by temporal importance. The algorithm first masks every feature at each lag $l$ in the look-back window, records the $\\ell^1$ change in model output, and normalizes these changes across time to obtain a time-relevance score $\\Delta^{\\mathrm{time}}_{l,t}$. It then masks each individual feature $x_{j,l,t}$, records a feature-relevance $\\Delta^{\\mathrm{feature}}_{j,l,t}$, and outputs $\\phi_{j,l,t} = \\Delta^{\\mathrm{feature}}_{j,l,t} \\times \\Delta^{\\mathrm{time}}_{l,t}$. The claimed discovery is that this rescaling captures delayed impacts between input and output better than either feature-only or time-only attributions, with runtime close to plain feature ablation and 32 to 367 times faster than TSR. The reported evidence is the best average rank in every dataset: $1.4 \\pm 0.5$ on Electricity, $1.4 \\pm 0.5$ on Traffic, and $2.4 \\pm 1.5$ on MIMIC-III, on the comprehensiveness and sufficiency metrics.","pith_inferences":["Editorial extension: the benchmark may reward methods whose scoring operation matches the masking used in evaluation, so a synthetic ground-truth test with known lags would be the cleanest way to confirm the claimed faithfulness.","Editorial extension: the rescaling step is separable from the underlying saliency estimator; applying the same time-weight product to gradient-based scores like integrated gradients could make those methods temporally aware without the cost of full TSR.","Editorial extension: the method returns a full (output, feature, lag) importance tensor, so it could support downstream uses the paper does not develop, such as grouping patients by which temporal signatures drive mortality or monitoring when a model's time-relevance curve shifts."],"forward_implications":["Practitioners can obtain faithful local explanations for modern architectures, including transformer and LLM-based forecasters, without training a surrogate model.","The reported runtime makes temporal rescaling practical where TSR was not run at all, such as on the CALF foundation model.","Because the method is model-agnostic and output-based, the same code path works for multi-horizon regression and for binary clinical classification.","The open-source framework allows side-by-side benchmarking of 20+ time series models and 10+ interpretation methods, which the paper argues earlier evaluations lacked by relying on older LSTM/GRU baselines.","The time-relevance weights themselves are inspectable and show interpretable structure, with the highest weights at the most recent hour and the same hour one day earlier in electricity and traffic."],"supporting_citations":[{"why":"Defines TSR, the temporal-saliency-rescaling idea that WinTSR simplifies and benchmarks against.","marker":"Ismail et al. 2020"},{"why":"Defines WinIT, the windowed temporal-importance competitor, and the MIMIC-III preprocessing WinTSR follows.","marker":"Leung et al. 2023"},{"why":"Supplies the comprehensiveness and sufficiency definitions used to evaluate faithfulness.","marker":"DeYoung et al. 2019"},{"why":"Defines AOPCR, the regression perturbation-curve metric used in evaluation.","marker":"Ozyegen, Ilic, and Cevik 2022"},{"why":"Provides the ground-truth-free interpretation evaluation protocol the paper adopts.","marker":"Turbé et al. 2023"},{"why":"Provides the Time-Series-Library implementations of the five evaluated models.","marker":"Wu et al. 2023"},{"why":"Defines Feature Ablation, the strongest perturbation baseline and second-ranked method.","marker":"Suresh et al. 2017"},{"why":"Supplies the MIMIC-III critical care dataset used for the classification evaluation.","marker":"Johnson et al. 2016"},{"why":"Supplies the UCI Electricity dataset used for the regression evaluation.","marker":"Trindade 2015"}],"fun_headline_variants":["Time-aware rescaling bests 10 explainers on 3 real datasets","WinTSR multiplies saliency by time importance to top benchmark","Feature times time importance lifts faithfulness in time series","Rescaling by temporal relevance outperforms 10 baselines","Windowed rescaling beats others on forecasting and mortality tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that faithfulness is correctly measured by masking the top-ranked features (comprehensiveness) or keeping only them (sufficiency) and observing the output change; because WinTSR's scores are produced by exactly the same masking operation used in that evaluation, part of its reported advantage may reflect self-consistency rather than an independently verified explanation.","fun_headline_variants_meta":{"raw":{"variants":["Time-aware rescaling bests 10 explainers on 3 real datasets","WinTSR multiplies saliency by time importance to top benchmark","Feature times time importance lifts faithfulness in time series","Rescaling by temporal relevance outperforms 10 baselines","Windowed rescaling beats others on forecasting and mortality tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1274,"prompt_tokens":970,"completion_tokens":304,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":218}},"tokens_in":586,"tokens_out":304,"duration_ms":4174,"temperature":1.0,"reasoning_tokens":218,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:30:29.916540+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate synthetic data with a known causal lag structure—say output depends only on feature 1 at lag 3 and feature 2 at lag 24, with all other entries pure noise—train any of the five models, apply WinTSR, and check whether the two true (feature, lag) positions receive the top importance scores across many seeds; if an exhaustive single-feature ablation recovers the known positions while WinTSR does not, the temporal rescaling step is not doing the explanatory work claimed.","supporting_citations":[{"cited_title":"(2020) proposed to separate the temporal dimension when calculating feature importance and rescaling it","cited_arxiv_id":null,"evidence_quote":"Defines TSR, the temporal-saliency-rescaling idea that WinTSR simplifies and benchmarks against."},{"cited_title":"(2023) explicitly accounted for the temporal dependence among observations of the same feature by summarizing its importance over a lookback window","cited_arxiv_id":null,"evidence_quote":"Defines WinIT, the windowed temporal-importance competitor, and the MIMIC-III preprocessing WinTSR follows."}],"review_version":1}