SCFormer: Structured Channel-wise Transformer with Cumulative Historical State for Multivariate Time Series Forecasting

Shiwei Guo; Yi Wang; Yunfei Han; Yupeng Ma; Ziang Chen

REVIEW 3 major objections 5 minor 38 references

SCFormer: Structured Channel-wise Transformer with Cumulative Historical State for Multivariate Time Series Forecasting

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that forcing all linear layers in a channel-wise Transformer to respect temporal order and feeding in a HiPPO-compressed cumulative historical state reduces multivariate forecasting error by up to 16.9% over a strong…

desk verdict Worth a serious look as an empirical architecture, but the temporal-causality story is the weakest part and should not survive peer review as stated. read the letter →

arxiv 2505.02655 v1 pith:HOPUVFBO submitted 2025-05-05 cs.LG cs.AI

Shiwei Guo , Ziang Chen , Yupeng Ma , Yunfei Han , Yi Wang This is my paper · ORCID

classification cs.LGcs.AI

keywords multivariatetimeseriesforecastingchannel-wiseTransformertemporalcausalitystructuredlineartransformationtriangularmatrix1DconvolutionHiPPOcumulativehistoricalstate

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SCFormer sets out to establish that multivariate time series forecasting by channel-wise Transformers has two fixable defects: unconstrained linear transformations let future values influence past features, and the model sees only the look-back window, ignoring history before it. The paper proposes forcing every linear map in the Transformer to respect temporal order, via triangular matrices or 1D convolutions, and feeding the model a HiPPO-compressed cumulative historical state alongside the look-back. Reported results show the triangular variant reduces average MSE by 12.3% on ECL, 16.9% on Exchange, and 8.9% on Weather relative to the main channel-wise baseline, while roughly halving parameters. A reader would care because both changes are simple and model-agnostic: if true, causality and long memory can be added to a transformer without redesigning attention.

What carries the argument

Structured temporal-constraint matrices and HiPPO memory. The paper forces every linear map in the channel-wise Transformer to be upper triangular (or a 1D convolution, shown to be equivalent to a triangular Toeplitz matrix), so each time step's feature is built only from itself and earlier steps. HiPPO, High-order Polynomial Projection Operators, compresses the variable-length history before the look-back window into a fixed-size coefficient vector by projection onto orthogonal polynomial bases, updating it recursively; this state plays the role of the transition matrix in a Markov view, while the channel-wise Transformer acts as the emission model.

What would settle it

Run the same SCFormer architecture with the triangular or convolutional constraints removed but the number of trainable parameters held fixed, for example by using dense matrices with matched rank; if the unconstrained model matches or beats SCFormer on ECL, Weather, and Exchange, then the temporal-constraint mechanism is not what drives the reported gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that two additions to channel-wise attention improve multivariate time series forecasting: all linear transformations, including query, key, value, and feed-forward matrices, must be structured so that later time steps cannot influence earlier features, and the model should maintain a HiPPO projection of the entire history before the look-back window as a cumulative state, treating forecasting as emission from that memory rather than a Markov transition from a fixed window. The authors report that the triangular variant lowers average MSE by 12.3% on ECL, 16.9% on Exchange, and 8.9% on Weather compared with the strongest channel-wise baseline, and that replacing dense matrices with triangular or convolutional structure cuts parameters by about 50% or 90%, respectively. Ablations removing either component degrade performance, and attaching the HiPPO state to several other transformer variants improves most benchmarks.

Load-bearing premise

The paper's results rest on the premise that letting future values influence how past time steps are represented is harmful enough to cause overfitting, rather than merely being a harmless modeling choice.

Editorial extensions

If this is right

Channel-wise attention can be made causally safe by structuring only the linear maps, eliminating future-to-past leakage in Q/K/V and feed-forward layers without changing the attention score formula.
The HiPPO cumulative state is portable: the paper's Table 4 shows adding it to Reformer, Informer, Flowformer, and Flashformer improves most benchmarks, so other transformer families can adopt the memory module independently.
Parameter efficiency follows structurally: triangular weight matrices contain about half the degrees of freedom of dense matrices and 1D convolutions roughly a tenth, so the accuracy gains come with a smaller model.
Look-back and cumulative history are complementary, not redundant: ablations that remove either component hurt accuracy, and lengthening the look-back still helps even when the cumulative state is present.
The Markov-to-emission reformulation suggests long-horizon forecasting can be treated as reading out a latent state of the whole series rather than extrapolating a fixed window.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The temporal-constraint principle is likely acting partly as an inductive bias or regularizer rather than a strict causal necessity, since a model can still read the full look-back window; a fair test would match parameter counts between constrained and unconstrained variants.
Because the HiPPO state is separate from the look-back, SCFormer could be combined with channel-independent or patch-based backbones, not only channel-wise attention, to test whether the memory mechanism transfers beyond attention over channels.
The paper's framing of forecasting as emission from a transition matrix invites a direct comparison with state-space sequence models, where the HiPPO update is already a standard memory primitive; such hybrids are a natural next step.
The attention-score and future-correlation analysis implies channel attention could be evaluated by how well it anticipates true inter-channel correlation in the forecast horizon, which suggests a diagnostic or auxiliary loss not explored in the paper.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's what I'd want you to know before you read SCFormer: the empirical recipe is more credible than the story the authors tell about it. The architecture — HiPPO cumulative state feeding a channel-wise Transformer whose linear projections are triangular or 1D-convolutional — is new as a combination, and the reported gains (12% MSE on ECL, 17% on Exchange, 9% on Weather over iTransformer) are in a believable range. Code is public, ablations are thorough, and the HiPPO boost replicates across Reformer, Informer, Flowformer, and Flashformer. That is real work.

The soft spot is the temporal-causality justification. The paper claims unconstrained linear transforms are wrong because future elements shouldn't influence past ones. But the model reads a look-back window and outputs a future horizon; nothing in the task requires causal masking inside the window. The paper even says 'an upper or lower triangular matrix does not affect temporal constraints,' which contradicts its own definition: lower-triangular would let j>i influence a_i. So the stated principle is not just unproven; it's incoherent. The only ablation for the constraint (Table 3a) is on ETT, and it changes sparsity and parameter count at the same time, so the small gains could be regularization or capacity reduction. There is no parameter-matched unconstrained control, and no constraint ablation on ECL, Exchange, or Weather. The authors should either supply that control or drop the causal framing.

Other weaknesses are more standard: baselines are quoted from their papers rather than rerun; no error bars or significance tests; the model loses on Traffic. None of these are disqualifying by themselves.

Who benefits: practitioners who want a parameter-efficient channel-wise transformer with strong baseline numbers, and researchers working on state-space memory in forecasting. The flawed causal story doesn't invalidate the empirical contribution, but it needs to be fixed or removed before I'd trust the interpretation.

I'd send it to peer review. Ask for a parameter-matched ablation of the constraint, error bars on the headline datasets, and a rewrite of Section 3.2. If they do that, this is a solid paper.

Referee Report

3 major / 5 minor

Summary. The paper proposes SCFormer, a multivariate time series forecasting architecture that combines a channel-wise Transformer with two modifications: (i) a HiPPO-based cumulative historical state that encodes the full history before the look-back window, and (ii) structured linear transformations, implemented either as triangular matrices or 1D convolutions, intended to enforce temporal causality in query, key, value, and feed-forward projections. The authors report improvements over nine baselines on ETT, PEMS, Solar-Energy, ECL, Exchange, Weather, and Traffic, with the largest gains against iTransformer, and they claim these gains stem from the temporal constraints and the HiPPO state. Ablations and analyses of HiPPO variants, look-back length, and attention patterns are provided.

Significance. If the claims were fully supported, the paper would offer a simple and parameter-efficient way to inject long-range history into channel-wise Transformers, and the public code release is a strength. However, the evidence in the manuscript does not support the central attribution of the gains to the structured temporal constraints. In particular, Table 4 shows that on ECL, Weather, and Traffic, SCFormer-triangular achieves exactly the same MSE/MAE as the '+HiPPO' variant of the iTransformer-like baseline, indicating that the triangular constraint contributes nothing on those datasets. The paper's own ablations on ETT show only tiny, sometimes negative differences. The useful result appears to be the HiPPO cumulative-state injection, which is itself borrowed from prior work; the claimed novelty of structural temporal constraints is not experimentally established. The comparison is also confounded because SCFormer receives more input information (the cumulative history) than the baselines, which use only a 96-step look-back.

major comments (3)

[§4.5, Table 4] On ECL, Weather, and Traffic, SCFormer-triangular reports exactly the same MSE and MAE as the '+HiPPO' row of the Transformer baseline (e.g., ECL average 0.156/0.254, Weather 0.235/0.271, Traffic 0.509/0.359). Since the Transformer row appears to be the channel-wise iTransformer used elsewhere in the paper, this means the triangular temporal constraint yields no measurable benefit on these three datasets. The paper's headline improvements over iTransformer (12.3% on ECL, 8.9% on Weather) are therefore entirely attributable to the HiPPO state, not to the structured matrices. The authors must either explain this exact equality, provide a direct dense-parameter-matched control on all datasets, or substantially revise the claim that temporal constraints improve forecasting.
[§3.2, Eq. (3); §4.4, Table 3(a)] The temporal-causality justification is internally inconsistent and not tested in isolation. The paper states that 'an upper or lower triangular matrix does not affect temporal constraints' (Section 3.2), but under its own definition in Eq. (3), a lower-triangular matrix allows later timesteps to influence earlier ones; the two orientations are not equivalent with respect to the stated principle. More importantly, Table 3(a) compares SCFormer to Transformer-HiPPO on ETT only, varying sparsity and parameter count simultaneously, with differences that are tiny and sometimes negative (e.g., ETTm1 horizon 96: 0.318 vs 0.315). No constraint-only ablation with matched parameter count is reported on the datasets driving the headline results. The assertion in Section 4.4 that temporal constraints 'help mitigate overfitting' is not supported by the evidence presented.
[§3.1, Eq. (2); §4.2, Table 2] The experimental comparison is confounded by the cumulative historical state. Baselines are evaluated with a fixed look-back of 96, while SCFormer additionally receives the HiPPO state computed from the entire history. The improvement over iTransformer could therefore come from the extra input information rather than from the model architecture. Table 4 strongly suggests this is the case, as iTransformer+HiPPO matches SCFormer on the headline datasets. A controlled comparison should include baselines augmented with the same cumulative-state input, or an SCFormer variant without HiPPO evaluated under the same conditions. Without such a control, the claim that SCFormer 'significantly outperforms mainstream baselines' is not a fair attribution of architectural benefit.

minor comments (5)

[§4.3, Table 2] No error bars, confidence intervals, or significance tests are reported for any method, despite claims of 'significant' improvement. For the small differences in Table 3(a), multiple random seeds with standard deviations are necessary to support the conclusions.
[§4.2] Baseline results are taken from their original papers rather than rerun under a unified protocol; this should be disclosed alongside every comparison and ideally accompanied by a reproducibility statement for the baselines.
[§3.2, Eq. (4)] The constraint is written as 'A_ij, B_ij, C_ij = 0 if i > j', but the value matrix is denoted E, not C; this notation error makes the equation ambiguous.
[§3.3, Eq. (10)] The claim that 'at most ceil((d-k)/(k-1)) + 1 layers of convolution' generate a complete upper triangular matrix needs a proof or a citation; as stated, it is not obvious and is not derived in the text.
[Table 4] The model labeled 'Transformer' in Table 4 is not defined in the text; if it is intended to be iTransformer (as the numbers suggest), this should be stated explicitly, and the caption should describe how the 'channel-wise strategy' is applied to Reformer, Informer, Flowformer, and Flashformer.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SCFormer's claims rest on external HiPPO theory, standard benchmarks, and component ablations.

full rationale

The paper makes no derivation that reduces to its own target metric. The triangular and convolutional structured linear transformations are defined directly in Eqs. (4)–(10) as constrained versions of ordinary linear projections, and the HiPPO cumulative historical state is imported from the external HiPPO literature (Gu et al., NeurIPS 2020) rather than invented or fitted here. Benchmarks are standard public datasets (ETT, ECL, Traffic, Weather, Exchange, PEMS, Solar-Energy), and baseline numbers are taken from the respective external papers. The ablations in Table 3 remove the temporal constraint, remove HiPPO, or remove the look-back window, and they report independent test-set MSE/MAE; none of those ablations fits a parameter to the reported outcome. There are no self-citations to prior work by the same authors that carry a load-bearing premise, no uniqueness theorem imported from the authors' own previous papers, and no ansatz smuggled in via self-citation. The skeptical concern that the paper's temporal-causality justification is under-tested — because the Table 3(a) ablation varies sparsity and parameter count simultaneously, and the paper itself states that upper or lower triangular matrices are equivalent for temporal constraints — is a correctness/identification concern, not a circularity concern, since the reported gains are not definitionally forced by the construction. The central claim remains externally falsifiable, so the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or theoretical entities are introduced; SCFormer reuses existing components, namely Transformer, HiPPO, and convolutions. The free parameters are hyperparameters chosen by hand, and the axioms are modeling assumptions that the paper does not independently prove.

free parameters (3)

HiPPO order = 512
Set by hand as the state dimension for cumulative history; affects model capacity and the amount of remembered detail.
Number of convolutional layers and kernel size = 3 layers, kernel 32, stride 1
Hand-chosen for SCFormer-conv; the receptive field and parameter count depend on these values.
Transformer base hyperparameters = iTransformer defaults
The paper reuses iTransformer's configuration, so hidden sizes, layer count, and heads are fixed by that choice rather than derived.

assumptions (4)

domain assumption Temporal causality principle: feature computations on a time series must not let later elements influence earlier ones.
Stated in Section 1 and Section 3.2; load-bearing for the proposed structuring, but not proven and debatable for a whole-window-to-future forecasting task.
domain assumption A fixed look-back window forecasting model approximates a first-order Markov transition matrix.
Used in Fig. 1 and Section 1 to motivate cumulative history; this is a loose analogy, since the predictor maps an L-length window to a future horizon, not a one-step transition.
domain assumption Baseline numbers taken from their original papers are directly comparable to the authors' runs.
Section 4.2 states all baseline results are taken from respective papers; this assumes identical setup, train and validation splits, and hardware-independent convergence.
domain assumption HiPPO projection with the chosen order captures useful cumulative historical information without distribution shift.
Section 3.1 and the ablations in Section 4.4 assume the fixed-size polynomial coefficients preserve the relevant history; the paper tests this only empirically, with mixed results on Traffic.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SCFormer: Structured Channel-wise Transformer with Cumulative Historical State for Multivariate Time Series Forecasting." pith.science (2026). https://pith.science/paper/HOPUVFBO

@misc{pith2026250502655,
  author       = {Pith},
  title        = {Pith review of: SCFormer: Structured Channel-wise Transformer with Cumulative Historical State for Multivariate Time Series Forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HOPUVFBO}},
  note         = {Machine review of arXiv:2505.02655}
}

read the original abstract

The Transformer model has shown strong performance in multivariate time series forecasting by leveraging channel-wise self-attention. However, this approach lacks temporal constraints when computing temporal features and does not utilize cumulative historical series effectively.To address these limitations, we propose the Structured Channel-wise Transformer with Cumulative Historical state (SCFormer). SCFormer introduces temporal constraints to all linear transformations, including the query, key, and value matrices, as well as the fully connected layers within the Transformer. Additionally, SCFormer employs High-order Polynomial Projection Operators (HiPPO) to deal with cumulative historical time series, allowing the model to incorporate information beyond the look-back window during prediction. Extensive experiments on multiple real-world datasets demonstrate that SCFormer significantly outperforms mainstream baselines, highlighting its effectiveness in enhancing time series forecasting. The code is publicly available at https://github.com/ShiweiGuo1995/SCFormer

Figures

Figures reproduced from arXiv: 2505.02655 by the authors.

**Figure 2.** Overall structure of SCFormer. For forecasting at a given moment, the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** The computation process of HiPPO: The coefficients [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: The effect of look-back length: The 720 window size look-back (green) [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

**Figure 5.** Figure 5: (a) Channels(multivariate) correlations: Left-Top: the future correlations [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 19 canonical work pages

[1]

arXiv preprint arXiv:2402.05956 (2024)

Chen, P., Zhang, Y., Cheng, Y., Shu, Y., Wang, Y., Wen, Q., Yang, B., Guo, C.: Pathformer: Multi-scale transformers with adaptive pathways for time series forecasting. arXiv preprint arXiv:2402.05956 (2024)

arXiv 2024
[2]

Information Fusion97, 101819 (2023)

Chen, Z., Ma, M., Li, T., Wang, H., Li, C.: Long sequence time-series forecasting with deep learning: A survey. Information Fusion97, 101819 (2023)

work page 2023
[3]

Advances in Neural Information Pro- cessing Systems35, 16344–16359 (2022)

Dao, T., Fu, D., Ermon, S., Rudra, A., Ré, C.: Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Pro- cessing Systems35, 16344–16359 (2022)

2022
[4]

Transactions on Machine Learning Research (2023)

Das, A., Kong, W., Leach, A., Mathur, S.K., Sen, R., Yu, R.: Long-term forecasting with tide: Time-series dense encoder. Transactions on Machine Learning Research (2023)

work page 2023
[5]

arXiv preprint arXiv:2312.00752 (2023) Structured Channel-wise Transformer with Cumulative Historical State 15

Gu, A., Dao, T.: Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023) Structured Channel-wise Transformer with Cumulative Historical State 15

arXiv 2023
[6]

Gu, A., Dao, T., Ermon, S., Rudra, A., Ré, C.: Hippo: Recurrent memory with optimalpolynomialprojections.Advancesinneuralinformationprocessingsystems 33, 1474–1487 (2020)

2020
[7]

Advances in neural information processing systems34, 15908–15919 (2021)

Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., Wang, Y.: Transformer in transformer. Advances in neural information processing systems34, 15908–15919 (2021)

2021
[8]

IEEE Communications Magazine 57(6), 114–119 (2019)

Hua, Y., Zhao, Z., Li, R., Chen, X., Liu, Z., Zhang, H.: Deep learning with long short-term memory for time series prediction. IEEE Communications Magazine 57(6), 114–119 (2019)

work page 2019

Show all 38 references

[9]

arXiv preprint arXiv:2402.10198 (2024)

Ilbert, R., Odonnat, A., Feofanov, V., Virmaux, A., Paolo, G., Palpanas, T., Redko, I.:Unlockingthepotentialoftransformersintimeseriesforecastingwithsharpness- aware minimization and channel-wise attention. arXiv preprint arXiv:2402.10198 (2024)

2024 arXiv
[10]

arXiv preprint arXiv:2108.05542 (2021)

Kalyan, K.S., Rajasekharan, A., Sangeetha, S.: Ammus: A survey of transformer- based pretrained models in natural language processing. arXiv preprint arXiv:2108.05542 (2021)

2021 arXiv
[11]

In: In- ternational Conference on Learning Representations (2022),https://openreview

Kim, T., Kim, J., Tae, Y., Park, C., Choi, J.H., Choo, J.: Reversible instance normalization for accurate time-series forecasting against distribution shift. In: In- ternational Conference on Learning Representations (2022),https://openreview. net/forum?id=cGDAkQo1C0p

2022
[12]

In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Kiranyaz, S., Ince, T., Abdeljaber, O., Avci, O., Gabbouj, M.: 1-d convolutional neural networks for signal processing applications. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 8360–8364. IEEE (2019)

2019
[13]

arXiv preprint arXiv:2001.04451 (2020)

Kitaev, N., Kaiser, Ł., Levskaya, A.: Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 (2020)

2020 arXiv
[14]

In: The 41st international ACM SIGIR conference on research & development in information retrieval

Lai, G., Chang, W.C., Yang, Y., Liu, H.: Modeling long-and short-term tempo- ral patterns with deep neural networks. In: The 41st international ACM SIGIR conference on research & development in information retrieval. pp. 95–104 (2018)

2018
[15]

In: 2017 IEEE 19th international conference on e- health networking, applications and services (Healthcom)

Li, D., Zhang, J., Zhang, Q., Wei, X.: Classification of ecg signals based on 1d convolution neural network. In: 2017 IEEE 19th international conference on e- health networking, applications and services (Healthcom). pp. 1–6. IEEE (2017)

2017
[16]

arXiv preprint arXiv:2305.10721 (2023)

Li, Z., Qi, S., Li, Y., Xu, Z.: Revisiting long-term time series forecasting: An investigation on linear mapping. arXiv preprint arXiv:2305.10721 (2023)

2023 arXiv
[17]

Advances in Neural Information Processing Systems35, 5816–5828 (2022)

Liu, M., Zeng, A., Chen, M., Xu, Z., Lai, Q., Ma, L., Xu, Q.: Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems35, 5816–5828 (2022)

2022
[18]

Pattern Recognition Letters160, 26–33 (2022)

Liu, Y., Wang, Z., Yu, X., Chen, X., Sun, M.: Memory-based transformer with shorter window and longer horizon for multivariate time series forecasting. Pattern Recognition Letters160, 26–33 (2022)

2022
[19]

Pattern Recognition Letters160, 26–33 (2022).https://doi.org/https: //doi.org/10.1016/j.patrec.2022.05.010,https://www.sciencedirect.com/ science/article/pii/S0167865522001623

Liu, Y., Wang, Z., Yu, X., Chen, X., Sun, M.: Memory-based transformer with shorter window and longer horizon for multivariate time series forecast- ing. Pattern Recognition Letters160, 26–33 (2022).https://doi.org/https: //doi.org/10.1016/j.patrec.2022.05.010,https://www.scie...

2022 doi
[20]

IEEE Transactions on Neural Networks and Learning Systems (2023)

Liu, Y., Zhang, Y., Wang, Y., Hou, F., Yuan, J., Tian, J., Zhang, Y., Shi, Z., Fan, J., He, Z.: A survey of visual transformers. IEEE Transactions on Neural Networks and Learning Systems (2023)

2023
[21]

arXiv preprint arXiv:2310.06625 (2024) 16 Shiwei Guo et al

Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., Long, M.: itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625 (2024) 16 Shiwei Guo et al

2024 arXiv
[22]

In: The Eleventh International Conference on Learning Representations (2022)

Nie, Y., Nguyen, N.H., Sinthong, P., Kalagnanam, J.: A time series is worth 64 words: Long-term forecasting with transformers. In: The Eleventh International Conference on Learning Representations (2022)

2022
[23]

Engineering Applications of Artificial Intelligence122, 106126 (2023)

Parvaiz, A., Khalid, M.A., Zafar, R., Ameer, H., Ali, M., Fraz, M.M.: Vision trans- formers in medical computer vision—a contemplative retrospection. Engineering Applications of Artificial Intelligence122, 106126 (2023)

2023
[24]

Nature communi- cations11(1), 5575 (2020)

Tetko, I.V., Karpov, P., Van Deursen, R., Godin, G.: State-of-the-art augmented nlp transformer models for direct and single-step retrosynthesis. Nature communi- cations11(1), 5575 (2020)

2020
[25]

arXiv preprint arXiv:1607.08022 (2016)

Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)

2016 arXiv
[26]

In: Proceedings of the IEEE/CVF International Conference on Computer Vision

Vasu, P.K.A., Gabriel, J., Zhu, J., Tuzel, O., Ranjan, A.: Fastvit: A fast hy- brid vision transformer using structural reparameterization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5785–5795 (2023)

2023
[27]

Advances in neural information pro- cessing systems30(2017)

Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems30(2017)

2017
[28]

In: The Twelfth International Conference on Learning Representations (2024)

Wang, X., Zhou, T., Wen, Q., Gao, J., Ding, B., Jin, R.: Card: Channel aligned robust blend transformer for time series forecasting. In: The Twelfth International Conference on Learning Representations (2024)

2024
[29]

In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 conference on empirical methods in natural language processing: s...

2020
[30]

In: The eleventh international conference on learning representations (2022)

Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., Long, M.: Timesnet: Temporal 2d- variation modeling for general time series analysis. In: The eleventh international conference on learning representations (2022)

2022
[31]

arXiv preprint arXiv:2202.06258 (2022)

Wu, H., Wu, J., Xu, J., Wang, J., Long, M.: Flowformer: Linearizing transformers with conservation flows. arXiv preprint arXiv:2202.06258 (2022)

2022 arXiv
[32]

Advances in neural information processing systems34, 22419–22430 (2021)

Wu, H., Xu, J., Wang, J., Long, M.: Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems34, 22419–22430 (2021)

2021
[33]

Zeng, A., Chen, M., Zhang, L., Xu, Q.: Are transformers effective for time se- ries forecasting? In: Proceedings of the AAAI conference on artificial intelligence. vol. 37, pp. 11121–11128 (2023)

2023
[34]

Bioinformatics Ad- vances3(1), vbad001 (2023)

Zhang, S., Fan, R., Liu, Y., Chen, S., Liu, Q., Zeng, W.: Applications of transformer-based language models in bioinformatics: a survey. Bioinformatics Ad- vances3(1), vbad001 (2023)

2023
[35]

In: The eleventh international conference on learning representations (2022)

Zhang, Y., Yan, J.: Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In: The eleventh international conference on learning representations (2022)

2022
[36]

Machine Intelligence Research20(4), 514–538 (2023)

Zhao, Y., Zhang, J., Zong, C.: Transformer: A general framework from machine translation to others. Machine Intelligence Research20(4), 514–538 (2023)

2023
[37]

In: Proceed- ings of the AAAI conference on artificial intelligence

Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., Zhang, W.: Informer: Beyond efficient transformer for long sequence time-series forecasting. In: Proceed- ings of the AAAI conference on artificial intelligence. vol. 35, pp. 11106–11115 (2021)

2021
[38]

In: International conference on machine learning

Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., Jin, R.: Fedformer: Frequency en- hanced decomposed transformer for long-term series forecasting. In: International conference on machine learning. pp. 27268–27286. PMLR (2022)

2022

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

arXiv preprint arXiv:2402.05956 (2024)

Chen, P., Zhang, Y., Cheng, Y., Shu, Y., Wang, Y., Wen, Q., Yang, B., Guo, C.: Pathformer: Multi-scale transformers with adaptive pathways for time series forecasting. arXiv preprint arXiv:2402.05956 (2024)

arXiv 2024

[2] [2]

Information Fusion97, 101819 (2023)

Chen, Z., Ma, M., Li, T., Wang, H., Li, C.: Long sequence time-series forecasting with deep learning: A survey. Information Fusion97, 101819 (2023)

work page 2023

[3] [3]

Advances in Neural Information Pro- cessing Systems35, 16344–16359 (2022)

Dao, T., Fu, D., Ermon, S., Rudra, A., Ré, C.: Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Pro- cessing Systems35, 16344–16359 (2022)

2022

[4] [4]

Transactions on Machine Learning Research (2023)

Das, A., Kong, W., Leach, A., Mathur, S.K., Sen, R., Yu, R.: Long-term forecasting with tide: Time-series dense encoder. Transactions on Machine Learning Research (2023)

work page 2023

[5] [5]

arXiv preprint arXiv:2312.00752 (2023) Structured Channel-wise Transformer with Cumulative Historical State 15

Gu, A., Dao, T.: Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023) Structured Channel-wise Transformer with Cumulative Historical State 15

arXiv 2023

[6] [6]

Gu, A., Dao, T., Ermon, S., Rudra, A., Ré, C.: Hippo: Recurrent memory with optimalpolynomialprojections.Advancesinneuralinformationprocessingsystems 33, 1474–1487 (2020)

2020

[7] [7]

Advances in neural information processing systems34, 15908–15919 (2021)

Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., Wang, Y.: Transformer in transformer. Advances in neural information processing systems34, 15908–15919 (2021)

2021

[8] [8]

IEEE Communications Magazine 57(6), 114–119 (2019)

Hua, Y., Zhao, Z., Li, R., Chen, X., Liu, Z., Zhang, H.: Deep learning with long short-term memory for time series prediction. IEEE Communications Magazine 57(6), 114–119 (2019)

work page 2019