{"id":"27c90314-eb2a-499e-b526-7ba5c318379f","arxiv_id":"2505.08768","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SPAT, a sensitivity-based method that prunes whole attention modules using a dispersion score called SEND, cuts FLOPs by roughly 35% on average while slightly improving MSE/MAE on several forecasting benchmarks.","lead":"SPAT prunes entire attention modules from time series transformers based on a gradient dispersion score, cutting some FLOPs while improving forecast error. It is a practical compression recipe, but its reported gains partly depend on pruning ratios chosen after seeing test results.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gains do not isolate SEND: for iTransformer the best configuration (α=0.9, N=3) removes all attention layers, bypassing the SEND ranking, and no random/fixed-layer pruning baseline is provided for PatchTST.","rationale":"The reader's weakest assumption was that the SEND score is a valid measure of attention-module effectiveness, and the reader's rationale explicitly noted the missing random pruning baseline and the iTransformer configuration that removes all attention modules. My read agrees with that assessment and sharpens it: the iTransformer case is not merely a missing baseline but an internal inconsistency with the paper's own claim of selective pruning and retaining only the most effective attention mechanisms. Since the final pruned iTransformer uses α=0.9 with N=3, all MHA layers are removed; no SEND-based ranking is involved. The PatchTST result, where SEND does choose one layer, is the only evidence for the selection mechanism, and it lacks the minimal random/fixed-layer control needed to rule out that any one-layer removal plus fine-tuning produces the same small gain. The paper does provide code, reproducible FLOPs/parameter counts, and a coherent experimental setup, so this is not an unfounded claim; it is an empirically testable gap. The appropriate verdict remains CONDITIONAL: the authors need to add a random/fixed-layer pruning baseline, select alpha on validation data, and explicitly reconcile the iTransformer all-attention-removal configuration with the 'selective' claim. My analysis does not change the reader's verdict, so no verdict adjustment is needed.","tokens_in":16938,"tokens_out":6178,"duration_ms":57686,"concrete_test":"On PatchTST with the ETTh1 or ILI setting (α=0.3, lookback 336), run the full SPAT fine-tuning pipeline for at least five seeds under four layer-removal policies: (a) the SEND-lowest layer, (b) the SEND-highest layer, (c) a randomly chosen layer, and (d) the first layer. Select α on the validation split, not the test split. If the SEND-lowest policy is not significantly better (paired test across seeds) than random and fixed removal, the claim that SEND identifies which attention layers to prune is unsupported, and the headline result reduces to 'removing (some) attention layers plus fine-tuning helps.' For completeness, repeat the same comparison for iTransformer with α=0.3; if all-attention removal remains best, then SEND is irrelevant for that architecture.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim — that SPAT selectively removes redundant MHA modules — is not exercised in the configuration that produces most of the reported gains. For iTransformer, Table 4 shows the best pruning ratio is α=0.9 on every dataset where it is reported, and since iTransformer has N=3 attention layers (Table 6), Eq. 14 gives K=⌈0.9·3⌉=3, i.e., all MHA modules are removed. The SEND ranking from Eq. 13 is therefore never used for iTransformer; the pruned iTransformer in Tables 1–3 is simply the original model with every attention layer deleted and fine-tuned. The average MSE improvement for iTransformer (4.418%) is more than two-thirds of the combined 2.842% headline reduction when averaged with PatchTST's 1.266%, so the largest source of the headline result bypasses the proposed selection mechanism. For PatchTST, only one of three MHA layers is pruned (α=0.3), so SEND does choose a layer, but no experiment compares this choice against a random layer, the first/last layer, or the highest-SEND layer under the same fine-tuning protocol. Without such a baseline, the observed gains cannot be attributed to SEND; they could result simply from reducing model capacity and fine-tuning. Additionally, α is selected from test performance in Table 4, which makes the reported improvements over the original model optimistic. The load-bearing assumption is therefore that the SEND ordering, rather than the act of removing attention layers plus fine-tuning, drives the improvements — and that assumption is untested, and contradicted for iTransformer where no ordering is used.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SPAT, a structured pruning method that removes entire multi-head attention (MHA) modules from transformer-based time series forecasting models. The importance of each MHA module is measured during a pretraining phase by SEND (Sensitivity Enhanced Normalized Dispersion), computed as the average row-wise standard deviation of softmax-normalized sensitivity gradients. Layers with the lowest SEND scores are removed according to a pruning ratio α, and the model is fine-tuned. Experiments on eight multivariate datasets with PatchTST and iTransformer report average reductions of 2.842% in MSE, 1.996% in MAE, 35.274% in FLOPs, and 28.191% in parameters, as well as claims that the pruned models outperform lightweight, Mamba-based, and LLM-based baselines in both standard and zero-shot inference. The paper also includes ablation studies on the pruning ratio, lookback window length, model efficiency, and a limitations section.","tokens_in":17223,"tokens_out":3832,"duration_ms":37357,"significance":"If the results hold, the paper's central idea is valuable: pruning entire attention modules, rather than individual weights or heads, could make transformer forecasters smaller and faster without specialized hardware, and the reported accuracy gains over the original models are surprising and interesting. The SEND metric itself is simple, contains no fitted constants, and the authors provide code and detailed per-horizon results in the appendix, which strengthens reproducibility. The paper also honestly lists limitations in Appendix D. However, the empirical support for the specific claim that SEND-based selection, rather than the act of removing attention layers plus fine-tuning, drives the improvements is incomplete. The current experiments do not isolate the selection mechanism, and the largest contributor to the headline gains uses a configuration in which the SEND ranking is never exercised. Because the paper's central contribution is the selective pruning criterion, this gap is load-bearing.","major_comments":[{"comment":"The pruning ratio α is selected from the test-set results reported in Table 4 and then used to produce the headline improvements in Tables 1–3. This is test-set-based model selection, which makes the reported gains optimistic and undercuts the claim that SPAT improves over the original models in a well-defined way. The manuscript should specify whether α was chosen on a validation split, or should provide a separate evaluation protocol that does not use test data for selecting α.","section":"§4.2, Table 4, Eq. (14)"},{"comment":"For iTransformer, the best configuration in Table 4 is α=0.9 on all reported datasets, and Table 6 shows that iTransformer has N=2, 3, or 4 attention layers. With K=ceil(αN), α=0.9 removes all MHA modules (K=N) for every iTransformer configuration. Therefore the SEND ranking in Eq. (13) is never used for iTransformer; the pruned iTransformer results in Tables 1–3 are simply the original model with every attention layer removed and then fine-tuned. Since iTransformer contributes an average MSE improvement of 4.418%, more than two-thirds of the combined 2.842% headline reduction comes from a configuration that bypasses the proposed selection mechanism. The authors should either report results for a configuration where SEND actually selects a subset of layers, or explicitly separate the claim 'removing all attention layers helps iTransformer' from the claim 'SEND selects which layers to remove.'","section":"§3.3, §4.2, Table 4, Table 6, Eq. (14)"},{"comment":"No experiment isolates the contribution of the SEND ranking. For PatchTST, α=0.3 removes one of three attention layers, so SEND does choose a layer, but the ablation only varies α and never compares the SEND-selected layer against a random layer, the first/last layer, or the layer with the lowest/highest SEND under the same fine-tuning protocol. Without such a baseline, the observed improvements could result from capacity reduction and fine-tuning rather than from the proposed importance metric. The authors should add a fixed-layer or random-layer pruning baseline and report the identity of the pruned layer selected by SEND.","section":"§4.4, Table 4"},{"comment":"The abstract and contributions state that SPAT-pruned models outperform lightweight, Mamba-based, and LLM-based methods in both standard and zero-shot inference, but Table 3 does not support this for pruned iTransformer. For example, in ETTh1→ETTh2 pruned iTransformer has MSE 0.363 versus TSLANet 0.359; in ETTh2→ETTh1 pruned iTransformer has MSE 0.596 versus the best 0.496; and pruned iTransformer is also worse than several baselines in ETTm1→ETTh2, ETTm2→ETTh1, and ETTm2→ETTh2. The zero-shot claim should be restricted to pruned PatchTST or otherwise qualified according to the actual table results.","section":"§4.2, Table 3, Abstract"}],"minor_comments":[{"comment":"There are formatting errors in Table 1, such as \"0 .43\" in the ETTh1 row and irregular spacing in several cells; these should be cleaned up for readability.","section":"Table 1"},{"comment":"Table 4 omits the ETTh2 and ETTm2 datasets that appear in the other main tables. The authors should either include them or explain why they are excluded from the pruning-ratio ablation.","section":"Table 4"},{"comment":"References [7] (Approximate attention with MLP) and [15] (Bi-Mamba+) both list the same arXiv identifier 2404.15772, which appears to be a duplicate or a citation error. Please correct the reference for Bi-Mamba+.","section":"References [7] and [15]"},{"comment":"The table uses the symbols ⋆ and † in the caption but the corresponding footnotes are not present in the text; please either add the footnotes or remove the symbols.","section":"Table 3"},{"comment":"Algorithm 1 introduces a state set S that is not further explained in the text; the notation should be defined or removed.","section":"Algorithm 1"},{"comment":"The sentence \"For PatchTST, α = 0.9, which removes all MHA modules, leads to a performance drop\" is correct only because PatchTST has N=3 layers and ceil(0.9*3)=3; the same logic makes α=0.9 vacuous for iTransformer with N≤4, which is directly relevant to the major comment above and should at least be acknowledged in the ablation discussion.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an interesting and timely problem, and the SEND metric is clean and easy to adopt. The main concern is that the evaluation does not yet demonstrate that the SEND ranking, rather than the removal of layers plus fine-tuning, is responsible for the reported gains. The iTransformer configuration that produces most of the headline improvement removes all attention layers, making the proposed selection mechanism vacuous in exactly the setting where the improvement is largest. I would want to see either new experiments with a random-layer/fixed-layer baseline and a validation-based selection of α, or a substantial rewriting of the claims to separate the two findings. The zero-shot claim also needs to be softened to match Table 3. These are fixable within the scope of a major revision, so I do not recommend rejection unless the authors are unwilling to add the necessary experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible, practical pruning recipe with real efficiency wins, but the experiments do not actually test the paper's central claim. The SEND ranking is supposed to choose which attention layers to remove, yet the configuration that produces most of the reported accuracy gains—iTransformer at α=0.9—removes every attention layer in every dataset shown in Table 4. For N=2,3,4, K=⌈αN⌉ equals N, so the ranking in Eq. 13 is never used. For PatchTST, exactly one of three layers is pruned and SEND does make a choice, but there is no comparison to random or fixed-layer pruning under the same fine-tuning protocol. Without that baseline, the observed improvements could just come from capacity reduction plus fine-tuning.\n\nWhat the paper does well: SPAT is clean and easy to implement. Compute sensitivity of the attention matrices, normalize row-wise, rank by dispersion, prune the bottom-ranked MHA modules, and fine-tune. Removing whole attention modules is a sensible structured-pruning choice that gives real speedups without special hardware. The reported FLOPs and parameter reductions are credible: over 50% for iTransformer, around 14% for PatchTST. Pruned models generally match or beat the originals, and the zero-shot results are interesting even if not fully explained. Code is public, which helps reproducibility.\n\nThe soft spots are real and load-bearing. Alpha is selected from test-set performance in Table 4 and then the same test set is used to report the headline gains—that is optimistic by construction. No validation split is used. As noted, the iTransformer result bypasses SEND entirely. The claim that higher SEND dispersion means higher effectiveness is asserted in Section 3.2 but never validated against any alternative selection rule. Baseline tables in Table 2 lack error bars, making it hard to tell whether the small MSE differences matter. These are fixable with an extra experiment: pick alpha on validation, then compare SEND against random pruning, first/last-layer pruning, and a no-pruning fine-tune control.\n\nWho this is for: people working on efficient time-series forecasting or structured pruning of transformers. The paper deserves a serious referee—not a desk rejection—because the recipe is useful and the central empirical pattern (many MHA modules are redundant) is worth documenting. But as it stands, it supports the weaker claim that removing attention modules plus fine-tuning helps, not the stronger claim that SEND is the reason. My recommendation: send it to peer review with a request for major revision, specifically the validation-based alpha selection and the random/fixed-layer ablation.","headline":"A useful pruning recipe with real speedups, but the headline mechanism—SEND—is not actually tested in the configuration that drives most of the gains.","tokens_in":17809,"tokens_out":2582,"would_cite":false,"duration_ms":28494,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SPAT removes redundant multi-head attention modules from transformer forecasters, cutting FLOPs and improving forecast accuracy.","keywords":["attention pruning","structured pruning","time series forecasting","multi-head attention","sensitivity analysis","model compression","zero-shot transfer","transformer"],"falsifier":"A reader could take a transformer forecaster, compute SEND rankings, and compare fine-tuned performance after removing the lowest-ranked layer against removing a randomly chosen layer over several seeds; if random removal matches SEND removal on average, the metric is not doing the ranking work.","tokens_in":16678,"feed_emoji":"✂️","tokens_out":8601,"duration_ms":74609,"temperature":0.7,"pith_summary":"SPAT is a structured pruning method for multivariate time series forecasting. Its central claim is that many whole multi-head attention (MHA) modules inside transformer forecasters are redundant or harmful, and that deleting them—not pruning individual weights or heads—makes the model smaller and faster while slightly improving forecast accuracy. The paper proposes a dynamic sensitivity metric, SEND, which ranks attention layers by the dispersion of their gradient-based importance scores, removes the lowest-ranked layers, and fine-tunes the surviving network. Across PatchTST and iTransformer on eight datasets, the pruned models achieve average reductions of 2.842% in MSE, 1.996% in MAE, and 35.274% in FLOPs, and they beat lightweight, Mamba-based, and LLM-based baselines in both standard and zero-shot inference. The point of interest is that expensive attention machinery can be cut wholesale, without specialized hardware, and still preserve or improve generalization.","feed_headline":"Deleting whole attention layers shrinks forecasters, lifts accuracy","feed_subtitle":"SPAT cuts up to 35% of FLOPs from transformer forecasters while improving forecast error.","key_machinery":"The central object is the SEND (Sensitivity Enhanced Normalized Dispersion) score, computed per attention layer. For each layer, SPAT builds a sensitivity matrix as the gradient of the average training loss with respect to a binary connection mask over the attention score matrix, takes absolute values, normalizes each row with a softmax, averages across attention heads, and then averages the row-wise standard deviations of the resulting matrix. That scalar ranks the layer; the bottom-K layers are removed and the model is fine-tuned. The load-bearing mechanism is the assumed link between dispersion of normalized gradient importance and the usefulness of an attention layer, as opposed to degenerate attention that behaves like a scaled identity mapping.","core_discovery":"The paper claims that transformer-based forecasters are over-provisioned with attention: many MHA layers contribute little or actively hurt performance, and removing whole layers chosen by a sensitivity-based importance score improves efficiency and accuracy. The discovery is that a single dispersion statistic, SEND, can identify those layers during pretraining. SEND is the average row-wise standard deviation of a softmax-normalized sensitivity matrix, where sensitivity is the gradient of the loss with respect to a binary connection mask on the attention scores. On the paper's experiments, pruned PatchTST improves average MSE by 1.266% and MAE by 0.824% with 13.990% fewer FLOPs, while pruned iTransformer improves MSE by 4.418% and MAE by 3.167% with over 50% fewer FLOPs. The pruned models also retain zero-shot forecasting ability across datasets, outperforming the compared lightweight, Mamba-based, and LLM-based baselines.","pith_inferences":["Because SEND uses only attention scores and gradients, the same ranking could be tested on attention layers in vision transformers or language models, where whole-MHA pruning has been explored with other criteria.","A direct way to isolate the metric's contribution is to compare SEND-ranked removal against random layer removal with the same fine-tuning budget; if random matches SEND, the gains come from fine-tuning and architecture, not from the score.","iTransformer's attention operates over channels rather than time patches, and it benefits more from pruning; this suggests channel-dimension attention is more redundant, so SPAT's gains may grow with the number of series channels.","The pretraining-and-finetuning loop leaves room to stack other compression techniques, such as head pruning or quantization on surviving layers, so the reported FLOP reductions are not necessarily the ceiling."],"forward_implications":["Transformer forecasters contain whole attention layers that can be removed without hurting accuracy, and on most datasets removal improves forecast error.","Module-level pruning gives speedups on ordinary GPUs because the surviving model is a smaller transformer, unlike unstructured weight pruning that needs specialized hardware.","Keeping the most effective attention layers is what sustains zero-shot forecasting across datasets; pruning too aggressively ($\\alpha=0.9$ on PatchTST) degrades performance.","The optimal number of layers to remove depends on the architecture: PatchTST's best result comes from removing one layer, while iTransformer improves even after removing several.","SPAT sits on top of existing transformer forecasters unchanged, so it can be combined with patching, masking, or other efficiency techniques."],"supporting_citations":[{"why":"It shows that degenerate MHA modules behave like scaled identity mappings, motivating the removal of entire attention layers.","marker":"[7]"},{"why":"It introduces single-shot connection sensitivity, the basis for computing importance from gradients of a loss with respect to a mask.","marker":"[12]"},{"why":"It provides the influence-function view that sensitivity is the loss change from removing a connection, which SPAT approximates via gradients.","marker":"[11]"},{"why":"It presents filter-level sensitivity pruning, the direct predecessor for ranking and removing whole modules by sensitivity.","marker":"[6]"},{"why":"It defines PatchTST and the benchmark protocol, including lookback, horizons, and channel independence, that SPAT uses for evaluation.","marker":"[23]"},{"why":"It supplies the zero-shot transfer setting and serves as the LLM-based baseline that pruned models are compared against.","marker":"[9]"},{"why":"It supplies TSLANet, a strong lightweight baseline used to show pruned transformers remain competitive.","marker":"[5]"},{"why":"It represents the Mamba-based baseline family used in the comparison.","marker":"[15]"},{"why":"It provides DLinear, the lightweight linear baseline that motivates showing attention modules still matter after pruning.","marker":"[34]"}],"fun_headline_variants":["Sensitivity metric finds useless attention layers in forecasters","Cutting entire attention modules slashes FLOPs, curbs error","Whole-layer attention pruning trims forecasters, fixes overfitting","Pruning whole attention layers boosts forecaster accuracy","SPAT: drop redundant attention layers, improve time series forecasts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a low SEND score—low dispersion in the normalized sensitivity gradients—truly flags an attention layer as redundant, so removing the lowest-scoring layers is better than removing random layers.","fun_headline_variants_meta":{"raw":{"variants":["Sensitivity metric finds useless attention layers in forecasters","Cutting entire attention modules slashes FLOPs, curbs error","Whole-layer attention pruning trims forecasters, fixes overfitting","Pruning whole attention layers boosts forecaster accuracy","SPAT: drop redundant attention layers, improve time series forecasts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1293,"prompt_tokens":984,"completion_tokens":309,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":226}},"tokens_in":600,"tokens_out":309,"duration_ms":3621,"temperature":1.0,"reasoning_tokens":226,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:46:23.140818+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could take a transformer forecaster, compute SEND rankings, and compare fine-tuned performance after removing the lowest-ranked layer against removing a randomly chosen layer over several seeds; if random removal matches SEND removal on average, the metric is not doing the ranking work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It introduces single-shot connection sensitivity, the basis for computing importance from gradients of a loss with respect to a mask."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the influence-function view that sensitivity is the loss change from removing a connection, which SPAT approximates via gradients."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It presents filter-level sensitivity pruning, the direct predecessor for ranking and removing whole modules by sensitivity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the zero-shot transfer setting and serves as the LLM-based baseline that pruned models are compared against."},{"cited_title":"Eldele, M","cited_arxiv_id":null,"evidence_quote":"It supplies TSLANet, a strong lightweight baseline used to show pruned transformers remain competitive."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides DLinear, the lightweight linear baseline that motivates showing attention modules still matter after pruning."}],"review_version":1}