Pith. sign in

REVIEW 4 major objections 5 minor 35 references

Sensorformer: Cross-patch attention with global-patch compression is effective for high-dimensional multivariate time series forecasting

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Sensorformer compresses global patch information into Sensor vectors and then extracts both cross-time and cross-variable dependencies, outperforming existing Transformer-based forecasting methods on nine benchmarks.

desk verdict A genuinely new two-stage attention compression scheme that is plausibly useful, but the paper overclaims causality and the empirical comparison needs to be redone under a uniform protocol before the SOTA claim can be trusted. read the letter →

arxiv 2501.03284 v1 pith:CMF43GIB submitted 2025-01-06 cs.LG

Liyang Qin , Xiaoli Wang , Chunhua Yang , Huaiwen Zou , Haochuan Zhang This is my paper · ORCID
classification cs.LG
keywords Transformermultivariatetimeseriesforecastingcross-patchattentionpatch-basedmodelsdynamiccausallagcompressionhigh-dimensional
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that existing Transformer-based multivariate time series forecasting methods fail to fuse cross-time and cross-variable dependencies because they force causally unrelated data into the same token when there are dynamic time lags between variables. Sensorformer addresses this with a two-stage pure cross-patch attention block: first it compresses the global patch information of every variable into one Sensor vector per variable, using the last patch as the query; then it uses these vectors as keys and values so that every patch can simultaneously attend across time and across variables. The paper claims this architecture captures the correct correlations even with dynamic lags, reduces the asymptotic cost of cross-patch self-attention from quadratic in patch count to linear, and achieves the best or second-best error on more benchmark settings than any of nine compared methods.

What carries the argument

The Sensor Attention Block is the core object: a pure cross-patch attention block with two stages. Input patches are the rows of $E_{\mathrm{Patches}} \in \mathbb{R}^{D \times N \times d\_model}$. Stage one sets the query to the last patch of each variable ($E_{\mathrm{Patches}}[:, -1, :]$) and the keys and values to all patches; after multi-head attention, layer normalization, and an MLP, it outputs $D$ Sensor vectors, one per variable, intended as a compressed summary of the causal history. Stage two sets the queries to all patches and the keys and values to the Sensor vectors, allowing each patch to gather global cross-time and cross-variable information. The mechanism is what allows the complexity reduction and what the paper credits for capturing dynamic time lags.

What would settle it

Train Sensorformer on a synthetic dataset where the causal lag between variables is engineered to be longer than the last patch or where the predictive information for a target variable resides in an early patch rather than the final one. If a variant that uses the first patch or an average patch as the first-stage query matches or beats the last-patch version, the claim that the last patch accumulates the relevant causal history would be refuted.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that a two-stage cross-patch attention mechanism can effectively fuse cross-time and cross-variable dependencies for multivariate time series forecasting. In the first stage, the last patch of each variable's input sequence acts as the query against all patches, producing a compressed representation the authors call the Sensor. In the second stage, every patch queries the Sensor, so attention weights can link any time period of any variable through a compressed global summary. This design avoids merging entire sequences or single time steps that may be misaligned under dynamic causal lags, and it cuts complexity from $O(D^2 \cdot N^2 \cdot d\_model)$ to $O(D^2 \cdot N \cdot d\_model)$. The experiments on nine real-world datasets show Sensorformer achieving the lowest or second-lowest error 64 times across all settings, ahead of iTransformer, PatchTST, and other baselines.

Load-bearing premise

The method assumes that the last patch of each variable's input sequence, after patching and embedding, has accumulated all historical causal information relevant to the future, so using it as the sole query in the first attention stage loses no essential information; this premise is asserted rather than derived.

Editorial extensions

If this is right

  • If Sensorformer's compression is effective, the cost of cross-patch attention can be reduced by a factor of the patch count without the accuracy loss that earlier channel-dependent transformers showed.
  • The method provides a way to fuse cross-variable and cross-time dependencies in one attention block, avoiding the forced fusion of misaligned time points under dynamic causal lags.
  • On the nine tested benchmarks, Sensorformer attains the lowest or second-lowest MSE/MAE 64 times, more than iTransformer and PatchTST.
  • Ablation results indicate both stages contribute; removing the second-stage attention hurts more than removing the first stage, while pure cross-patch self-attention is slower and uses more memory.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same two-stage compression could be applied to other token-based sequence models where cross-sequence attention is expensive, such as multipanel images or recommender systems.
  • A direct test of the causal-lag mechanism would be to generate synthetic series with known, shifting lags and measure whether Sensorformer recovers the correct lag structure in its attention weights.
  • The paper's time-lag analysis measures correlation lag, not causation; a causal benchmark would be needed to confirm the authors' interpretation.
  • The complexity reduction suggests that for very high-dimensional systems (thousands of variables), Sensorformer may scale better than channel-dependent transformers, but the experiments only go up to 862 dimensions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Sensorformer, a transformer architecture for multivariate time series forecasting that replaces pure cross-patch self-attention with a two-stage 'Sensor Attention Block'. In the first stage, the last patch of each variable serves as the query and all patches serve as keys/values, producing D compressed 'Sensor' vectors; in the second stage, all patches attend to these Sensor vectors. The authors claim this reduces attention complexity from O(D^2 * N^2 * d_model) to O(D^2 * N * d_model), better handles dynamic causal lags between variables, and achieves state-of-the-art results on nine benchmarks. The manuscript reports experiments on MSE/MAE, an analysis of time lags via Pearson correlation, efficiency and memory comparisons, ablations of both attention stages, and a robustness study across random seeds.

Significance. If the empirical claims are upheld, the proposed two-stage compression offers a simple and principled way to reduce the cost of cross-patch attention while retaining both cross-time and cross-variable dependency extraction, which would be a useful contribution for high-dimensional forecasting. The complexity derivation is explicit and correct, the code and reproduction scripts are promised publicly, and the ablation study separates the effects of the two stages. However, the significance is currently conditional on the visibility and fairness of the main experimental table and on evidence supporting the last-patch-as-query premise, which is the architectural cornerstone of the method.

major comments (4)
  1. [§4.1, Table 1] The central SOTA claim is based on Table 1, but the table content is missing from the manuscript text: only the caption and the aggregate count of 64 lowest/second-lowest errors are given. Without the per-dataset, per-horizon MSE/MAE entries, the reader cannot verify the claimed superiority of Sensorformer over iTransformer, PatchTST, and the other baselines. The full table must be included, and ideally accompanied by standard deviations or significance tests.
  2. [Appendix A.2, §4.1] The comparison protocol is not consistent across methods. The text states that iTransformer, PatchTST, and Crossformer were reproduced with hyperparameters matched to Sensorformer (10 epochs, batch size 32), while 'for the results of other baseline methods ... we referred to the results reported in [13]'. Results taken from [13] were obtained under their own training schedules, which typically use far more than 10 epochs, so they do not provide a head-to-head comparison under the fixed 10-epoch protocol. Either rerun all baselines under the exact same protocol or clearly report both reproduced and cited numbers with labels stating which protocol each number came from.
  3. [§3.1, Algorithm 1, Fig. 5] The load-bearing premise of the first-stage attention — that 'the last patch of each variable can be regarded as the result of accumulating all history causal relationships' — is asserted without derivation or validation. A patch contains only the final P time steps of the lookback window, and using a single query vector per variable per head collapses all historical patch information into one weighted average, which may lose multi-lag causal information. The ablation in Fig. 5 only removes the first stage entirely; it does not vary the query choice (e.g., first patch, middle patch, average patch, or a learned query). Please add an ablation that varies the query selection and reports how much information from earlier patches is retained, to substantiate the architectural motivation.
  4. [§4.2, Table 2, Abstract, §1, §5] The manuscript repeatedly uses causal terminology ('causal time lag', 'causal relationships', 'ability to capture correct inter-variable correlations and causal relationships') based on a Pearson correlation analysis that only measures the lag of the maximum-correlation patch. This evidence supports correlation lag, not causation. The authors themselves acknowledge in §4.2 that the results 'do not fully prove the existence of time lag in the causal relationship', but the abstract and conclusions state the causal claim more strongly. Either soften the causal language throughout or provide a synthetic causal experiment (e.g., known lagged causal generative processes) to directly test whether the architecture recovers the correct causal lags.
minor comments (5)
  1. [§3.3, Algorithms 1 and 2] Both algorithms contain two consecutive 'Step 3' lines; the second one should be labeled 'Step 4'.
  2. [§4.4] There is a typo in the paragraph describing ablations: 'attentio ssn' should be 'attention'.
  3. [Table 2, Appendix B.1] The phrase 'casual time lag' is used in multiple places (e.g., Table 2 title, Table 4, Fig. 6) and should be corrected to 'causal time lag'.
  4. [Fig. 4] The bar charts for training time and CUDA memory usage have small numerical labels; adding the exact values as a table alongside the figure would improve readability.
  5. [Appendix A.2] The sentence 'the training epochs was fixed to 10' is repeated and uses incorrect subject-verb agreement; please revise to 'the number of training epochs was fixed to 10' in both occurrences.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the last-patch-query premise is an unvalidated design assumption, not a derived result.

full rationale

The paper's derivation chain contains no step that reduces to its own inputs. The core architectural choice, using the last patch of each variable as the sole query in first-stage attention, is introduced as an explicit modeling assumption in Section 3.1: 'the last patch of each variable in the input sequence can be regarded as the result of accumulating all history causal relationships.' This is a premise about what the last patch represents, not a conclusion derived from the attention equations, and it is not defined in terms of the predicted outputs. The attention computations in Algorithms 1 and 2 are standard multi-head attention with specified Q/K/V shapes; the claimed complexity reduction O(D^2*N^2*d_model) to O(D^2*N*d_model) follows arithmetically from those shapes and is not an input to the method. The benchmark results are obtained by training the model and evaluating on held-out test sets, with no fitted constant renamed as a prediction. The paper contains no load-bearing self-citation: references to iTransformer, PatchTST, Crossformer, and the Time-Series-Library are either external works or third-party implementations used for baselines. Section 4.2 even acknowledges that the causal-lag analysis 'does not fully prove the existence of time lag in the causal relationship,' which further confirms that the paper is not circularly assuming its own conclusion. The main weakness, that the last patch may not adequately summarize multi-lag historical information, is a correctness or validity concern, not a circularity concern.

Assumptions & free parameters 8 free parameters · 3 assumptions · 1 invented entities

The paper is empirical; no theory is derived. Free parameters are standard hyperparameters chosen via validation sweeps and then fixed. The axioms are stated beliefs about causal history and attention, and the only invented entity is the internal Sensor representation, which lacks independent evidence outside the model's performance.

free parameters (8)
  • patch length P = 32
    Chosen after sweep in Appendix B.2; fixed across all datasets.
  • patch stride S = 8
    Chosen for overlap between patches; sweep in Appendix B.2.
  • d_model = 256
    Representation dimension; chosen as a trade-off in Appendix B.2.
  • encoder layers = 2
    Fixed for Sensorformer and transformer baselines.
  • attention heads = 2
    Fixed for all models.
  • training epochs = 10
    Fixed for all methods without early stopping; this experimental choice may bias comparisons.
  • batch size = 32
    Fixed for all methods.
  • learning rate = 0.0001
    Fixed for all methods.
assumptions (3)
  • domain assumption The last patch of each variable can be regarded as the result of accumulating all historical causal relationships.
    Stated in Section 3.1; underpins the choice of query in the first attention stage.
  • domain assumption Dynamic time lags between variables cause existing channel-dependent transformers to fuse causally unrelated data, limiting performance.
    Motivates the architecture; supported only by a correlation heuristic in Section 4.2, not by causal analysis.
  • ad hoc to paper Fixing training epochs to 10 for all methods is a fair comparison protocol.
    Specific to this paper's experimental design; likely disadvantages baselines that benefit from longer training.
invented entities (1)
  • Sensor (S_sensor)
    purpose: Compressed global representation of all patches, used as key/value in second-stage attention.
    Internal learned vector; no externally falsifiable handle beyond the model's empirical performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sensorformer: Cross-patch attention with global-patch compression is effective for high-dimensional multivariate time series forecasting." pith.science (2026). https://pith.science/paper/CMF43GIB

@misc{pith2026250103284,
  author       = {Pith},
  title        = {Pith review of: Sensorformer: Cross-patch attention with global-patch compression is effective for high-dimensional multivariate time series forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CMF43GIB}},
  note         = {Machine review of arXiv:2501.03284}
}
abstract

Among the existing Transformer-based multivariate time series forecasting methods, iTransformer, which treats each variable sequence as a token and only explicitly extracts cross-variable dependencies, and PatchTST, which adopts a channel-independent strategy and only explicitly extracts cross-time dependencies, both significantly outperform most Channel-Dependent Transformer that simultaneously extract cross-time and cross-variable dependencies. This indicates that existing Transformer-based multivariate time series forecasting methods still struggle to effectively fuse these two types of information. We attribute this issue to the dynamic time lags in the causal relationships between different variables. Therefore, we propose a new multivariate time series forecasting Transformer, Sensorformer, which first compresses the global patch information and then simultaneously extracts cross-variable and cross-time dependencies from the compressed representations. Sensorformer can effectively capture the correct inter-variable correlations and causal relationships, even in the presence of dynamic causal lags between variables, while also reducing the computational complexity of pure cross-patch self-attention from $O(D^2 \cdot Patch\_num^2 \cdot d\_model)$ to $O(D^2 \cdot Patch\_num \cdot d\_model)$. Extensive comparative and ablation experiments on 9 mainstream real-world multivariate time series forecasting datasets demonstrate the superiority of Sensorformer. The implementation of Sensorformer, following the style of the Time-series-library and scripts for reproducing the main results, is publicly available at https://github.com/BigYellowTiger/Sensorformer

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 20 canonical work pages

  1. [13]

    iTransformer: Inverted transformers are effective for time series forecasting

    Liu, Yong, et al. " iTransformer: Inverted transformers are effective for time series forecasting." arXiv preprint arXiv:2310.06625 (2023)

    arXiv 2023

  2. [1]

    Attention is all you need

    Vaswani, A . "Attention is all you need." Advances in Neural Information Processing Systems (2017)

    work page 2017

  3. [2]

    The llama 3 herd of models

    Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024)

    arXiv 2024

  4. [3]

    ChatGLM: A Family of Large Language Models from GLM-130B to GLM- 4 All Tools

    GLM, Team, et al. "ChatGLM: A Family of Large Language Models from GLM-130B to GLM- 4 All Tools." arXiv preprint arXiv:2406.12793 (2024)

    arXiv 2024

  5. [4]

    Tokens -to-token vit: Training vision transformers from scratch on imagenet

    Yuan, Li, et al. "Tokens -to-token vit: Training vision transformers from scratch on imagenet." Proceedings of the IEEE/CVF international conference on computer vision. 2021

    work page 2021

  6. [5]

    Supervision exists everywhere: A data efficient contrastive language - image pre-training paradigm

    Li, Yangguang, et al. "Supervision exists everywhere: A data efficient contrastive language - image pre-training paradigm." arXiv preprint arXiv:2110.05208 (2021)

    arXiv 2021

  7. [6]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day

    Li, Chunyuan, et al. "Llava-med: Training a large language-and-vision assistant for biomedicine in one day." Advances in Neural Information Processing Systems 36 (2024)

    work page 2024

  8. [7]

    Are transformers effective for time series forecasting?

    Zeng, Ailing, et al. "Are transformers effective for time series forecasting?." Proceedings of the AAAI conference on artificial intelligence. V ol. 37. No. 9. 2023

    work page 2023

Show all 35 references
  1. [8]

    A time series is worth 64 words: Long -term forecasting with transformers

    Nie, Yuqi, et al. "A time series is worth 64 words: Long -term forecasting with transformers." arXiv preprint arXiv:2211.14730 (2022)

    2022 arXiv

  2. [9]

    Crossformer: Transformer utilizing cross -dimension dependency for multivariate time series forecasting

    Zhang, Yunhao, and Junchi Yan. "Crossformer: Transformer utilizing cross -dimension dependency for multivariate time series forecasting." The eleventh international conference on learning representations. 2023

    2023

  3. [10]

    Timexer: Empowering transformers for time series forecasting with exogenous variables

    Wang, Yuxuan, et al. "Timexer: Empowering transformers for time series forecasting with exogenous variables." arXiv preprint arXiv:2402.19072 (2024)

    2024 arXiv

  4. [11]

    Onenet: Enhancing time series forecasting models under concept drift by online ensembling

    Wen, Qingsong, et al. "Onenet: Enhancing time series forecasting models under concept drift by online ensembling." Advances in Neural Information Processing Systems 36 (2024)

    2024

  5. [12]

    The capacity and robustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting

    Han, Lu, Han-Jia Ye, and De-Chuan Zhan. "The capacity and robustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting." IEEE Transactions on Knowledge and Data Engineering (2024)

    2024

  6. [14]

    Scalable Transformer for High Dimensional Multivariate Time Series Forecasting

    Zhou, Xin, et al. "Scalable Transformer for High Dimensional Multivariate Time Series Forecasting." Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2024

    2024

  7. [15]

    DeepAR: Probabilistic forecasting with autoregressive recurrent networks

    Salinas, David, et al. "DeepAR: Probabilistic forecasting with autoregressive recurrent networks." International journal of forecasting 36.3 (2020): 1181-1191

    2020

  8. [16]

    Modeling long- and short -term temporal patterns with deep neural networks

    Lai, Guokun, et al. "Modeling long- and short -term temporal patterns with deep neural networks." The 41st international ACM SIGIR conference on research & development in information retrieval. 2018

    2018

  9. [17]

    Tensorized LSTM with adaptive shared memory for learning trends in multivariate time series

    Xu, Dongkuan, et al. "Tensorized LSTM with adaptive shared memory for learning trends in multivariate time series." Proceedings of the AAAI Conference on Artificial Intelligence. V ol. 34. No. 02. 2020

    2020

  10. [18]

    Timesnet: Temporal 2d-variation modeling for general time series analysis

    Wu, Haixu, et al. "Timesnet: Temporal 2d-variation modeling for general time series analysis." arXiv preprint arXiv:2210.02186 (2022)

    2022 arXiv

  11. [19]

    Scinet: Time series modeling and forecasting with sample convolution and interaction

    Liu, Minhao, et al. "Scinet: Time series modeling and forecasting with sample convolution and interaction." Advances in Neural Information Processing Systems 35 (2022): 5816-5828

    2022

  12. [20]

    TemDep: Temporal Dependency Priority for Multivariate Time Series Prediction

    Liu, Shu, et al. "TemDep: Temporal Dependency Priority for Multivariate Time Series Prediction." Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 2023

    2023

  13. [21]

    Multivariate time -series forecasting with temporal polynomial graph neural networks

    Liu, Yijing, et al. "Multivariate time -series forecasting with temporal polynomial graph neural networks." Advances in neural information processing systems 35 (2022): 19414-19426

    2022

  14. [22]

    Graph wavenet for deep spatial -temporal graph modeling

    Wu, Zonghan, et al. "Graph wavenet for deep spatial -temporal graph modeling." arXiv preprint arXiv:1906.00121 (2019)

    2019 arXiv

  15. [23]

    N -BEATS: Neural basis expansion analysis for interpretable time series forecasting

    Oreshkin, Boris N., et al. "N -BEATS: Neural basis expansion analysis for interpretable time series forecasting." arXiv preprint arXiv:1905.10437 (2019)

    2019 arXiv

  16. [24]

    A transformer -based framework for multivariate time series representation learning

    Zerveas, George, et al. "A transformer -based framework for multivariate time series representation learning." Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. 2021

    2021

  17. [25]

    Informer: Beyond efficient transformer for long sequence time -series forecasting

    Zhou, Haoyi, et al. "Informer: Beyond efficient transformer for long sequence time -series forecasting." Proceedings of the AAAI conference on artificial intelligence. V ol. 35. No. 12. 2021

    2021

  18. [26]

    Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting

    Wu, Haixu, et al. "Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting." Advances in neural information processing systems 34 (2021): 22419-22430

    2021

  19. [27]

    Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting

    Liu, Shizhan, et al. "Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting." International conference on learning representations. 2021

    2021

  20. [28]

    Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting

    Zhou, Tian, et al. "Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting." International conference on machine learning. PMLR, 2022

    2022

  21. [29]

    Rethinking Channel Dependence for Multivariate Time Series Forecasting: Learning from Leading Indicators

    Zhao, Lifan, and Yanyan Shen. "Rethinking Channel Dependence for Multivariate Time Series Forecasting: Learning from Leading Indicators." arXiv preprint arXiv:2401.17548 (2024)

    2024 arXiv

  22. [30]

    Deep time series models: A comprehensive survey and benchmark

    Wang, Yuxuan, et al. "Deep time series models: A comprehensive survey and benchmark." arXiv preprint arXiv:2407.13278 (2024)

    2024 arXiv

  23. [31]

    Layer normalization

    Ba, Jimmy Lei. "Layer normalization." arXiv preprint arXiv:1607.06450 (2016)

    2016 arXiv

  24. [32]

    Time-Series-Library: A Library for Advanced Deep Time Series Models

    THUML. "Time-Series-Library: A Library for Advanced Deep Time Series Models." GitHub repository, 2024. Available at: https://github.com/thuml/Time-Series-Library

    2024

  25. [33]

    Revisiting long -term time series forecasting: An investigation on linear mapping

    Li, Zhe, et al. "Revisiting long -term time series forecasting: An investigation on linear mapping." arXiv preprint arXiv:2305.10721 (2023)

    2023 arXiv

  26. [34]

    Long- term forecasting with tide: Time-series dense encoder

    Das, Abhimanyu, et al. "Long- term forecasting with tide: Time-series dense encoder." arXiv preprint arXiv:2304.08424 (2023)

    2023 arXiv

  27. [35]

    Non- stationary transformers: Exploring the stationarity in time series forecasting

    Liu, Yong, et al. "Non- stationary transformers: Exploring the stationarity in time series forecasting." Advances in Neural Information Processing Systems 35 (2022): 9881-9893. Appendix A. Experimental Details A.1 Datasets and Baselines Dataset: We conducted experiments on 9 m...

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.