Pith. sign in

REVIEW 4 major objections 4 minor 61 references

Standard attention's static token representations limit multivariate time series forecasting; the paper proposes prime attention, which modulates keys and values per token pair with a learnable filter, and reports consistent forecasting gai

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 16:38 UTC pith:FYH7EDN5

load-bearing objection A clean formalization of dynamic relational attention and a plausible modulation mechanism, but the empirical case is compromised by a warm-start protocol and a missing random-init baseline. the 4 major comments →

arxiv 2509.12196 v2 pith:FYH7EDN5 submitted 2025-09-15 cs.LG cs.AI

Dynamic Relational Priming Improves Transformer in Multivariate Time Series

Hunjae Lee , Corey Clark This is my paper
classification cs.LG cs.AI
keywords prime attentiondynamic relational learningstatic relational learningmultivariate time series forecastingtransformerchannel dependencelead-lag correlationattention modulation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that standard attention's static relational learning—where each token presents the same key and value to every other token—is a poor match for multivariate time series, because different channel pairs can obey different physical laws or temporal dynamics. It introduces prime attention, which learns a per-pair modulation filter that element-wise reshapes keys and values before each interaction, letting a token present a different perspective to different partners. This dynamic relational learning is formalized with definitions and theorems, and experiments on three recent transformer backbones across eleven benchmarks report consistent gains, up to 6.5% in forecasting accuracy, with the same asymptotic computational complexity. The paper also reports that prime attention can match standard attention's performance with up to 40% less look-back history on some datasets, suggesting improved data efficiency.

Core claim

Prime attention replaces the static keys and values of standard attention with pair-modulated versions, e k_j = k_j ⊙ F_ij and e v_j = v_j ⊙ F_ij, where F_ij is a learnable filter (optionally seeded with estimated lead-lag and instantaneous correlations). The paper proves that standard attention implements static relational learning (a token's representation is invariant across interaction partners) and that prime attention implements dynamic relational learning (the representation of token j when interacting with i can differ from its representation when interacting with v). It further shows prime attention contains standard attention as a special case (F = 1), so it inherits universal appr

What carries the argument

The central object is the pair-specific modulation filter F_ij ∈ R^{d_model}, applied element-wise to the key and value of token j when it interacts with token i. This converts attention from static relational learning (one representation per token per layer) to dynamic relational learning (a tailored representation per pair), while leaving the attention coefficient computation and the O(N²·d) asymptotic complexity intact. The filter can be learned from random initialization or seeded with estimated lead-lag and instantaneous correlation features through an MLP.

Load-bearing premise

The benchmark gains assume prime attention is trained under the same protocol as standard attention, but the appendix says the transformer is first fine-tuned on standard attention and its attention module is then replaced with prime attention, so the improvements may partly be a warm-start artifact rather than dynamic relational learning itself.

What would settle it

Run a from-scratch training comparison on the Weather dataset with look-back 96, identical seeds and hyperparameters, between standard attention and prime attention with the filter randomly initialized near identity. If random-initialized prime attention does not beat standard attention, or if only lead-lag-initialized filters do, then the claim that pair-specific modulation per se drives the gains would be contradicted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Any transformer for multivariate time series can be upgraded to prime attention by swapping the attention block, yielding consistent forecast accuracy gains on heterogeneous-channel data without increasing asymptotic complexity.
  • On datasets where channels measure different physical quantities (Weather, Solar), gains reach up to 6.5%, while homogeneous datasets (ECL, Traffic) show only marginal changes, supporting the claim that the benefit comes from modeling pair-specific relational heterogeneity.
  • Prime attention can match standard attention's accuracy with up to 40% shorter history (for example L=48 or 64 versus L=96), which is relevant for data-scarce or low-latency forecasting settings.
  • The paper's formal distinction between static and dynamic relational learning gives a concrete vocabulary for what is missing in standard attention and a target for other relational architectures to aim at.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism generalizes beyond channel-wise time series, the same per-pair modulation could benefit attention on other data with heterogeneous pair relations, such as knowledge graphs, multi-agent systems, or sensor networks.
  • The ablation study shows random-initialized filters underperform filters seeded with lead-lag or instantaneous information, but it does not report whether random-initialized prime attention beats standard attention from scratch; if it does not, the reported gains may come from the injected pairwise priors rather than from dynamic modulation per se.
  • The reported fine-tuning protocol—first training on standard attention, then swapping in prime attention—leaves open a warm-start effect; a from-scratch training comparison would separate the benefit of the mechanism from the benefit of a better starting point.
  • The outlined GNN-based sparsification (reducing the filter memory from O(N²) to O(|E|)) is not evaluated empirically; if it works, it could make prime attention practical for high-dimensional systems where dense per-pair filters are prohibitive.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes 'prime attention' for multivariate time series (MTS) forecasting. In standard attention, each token presents the same key/value to every interaction; prime attention computes pair-specific modulations e_k_j = k_j ⊙ F_{i,j} and e_v_j = v_j ⊙ F_{i,j}, with F generated or learned as a function of pairwise statistics. The paper formalizes static vs. dynamic relational learning, proves that standard attention is static (Thm 4.3) and prime attention is dynamic (Thm 5.1), gives a universal-approximation inheritance argument, provides gradient-flow formulas, and reports experiments replacing standard attention in iTransformer, Timer-XL, and FreDF across 11 datasets, claiming consistent improvements and up to 6.5% forecasting accuracy gains, plus a sequence-length efficiency result.

Significance. The core idea is simple, well-motivated, and potentially practical: a drop-in attention modification with the same asymptotic cost could improve MTS forecasting across multiple backbones. The paper's formal categorization of static vs. dynamic relational learning is useful, and the empirical scope is broad, covering three recent transformer architectures and several standard benchmarks. The main strengths are the explicit pair-wise modulation formulation and the extensive backbone-level evaluation. However, the empirical claim is currently undercut by an inconsistency in the training protocol and by an ablation that leaves open whether the gains come from hand-crafted pairwise priors rather than from learnable dynamic priming. If the controlled comparison is fixed, the contribution would be solid; at present the headline result is not yet established.

major comments (4)
  1. [§7.1 vs. Appendix G] The main text says prime-attention variants are obtained 'by simply swapping out the main attention block ... changing nothing else and keeping with the original hyper-parameters except dropout' (§7.1). Appendix G, however, states: 'the transformer is first fine-tuned on standard attention and its attention module is subsequently replaced with prime attention only adjusting dropout rate.' If the latter is literal, prime attention begins from a converged standard-attention checkpoint and receives additional fine-tuning, while the standard baselines are trained from scratch. This is a warm-start and extra-training-budget artifact that alone could explain many of the reported gains. The paper must clarify the actual protocol, retrain both variants from scratch under identical budgets, or report both settings.
  2. [F.1, Figure 3] The filter ablation shows that random initialization of F underperforms lead-lag, instantaneous, and combined initializations, but it never reports whether randomly-initialized prime attention still beats standard attention. If random-init prime attention does not outperform standard attention, then the claimed improvements are attributable to the hand-crafted pairwise statistics (lead-lag, Pearson correlation, rolling std, rank transformation) rather than to dynamic relational priming, which is the paper's central claim. Please report random-init prime vs. standard attention on the main benchmarks, under the same training protocol.
  3. [Theorem 5.1 / Learnable Modulator Generation] Theorem 5.1 proves dynamic relational learning by treating F_{ji} and F_{jv} as 'independent learnable parameters' (Eqs. 19–20). In the actual construction, F_{i,j} = MLP(fs_{ij}) with a shared MLP over pairwise statistics (§5). The parameters are not independent; the outputs can differ only if the inputs differ. The conclusion may still hold, but the proof as written overstates the implementation. Similarly, Theorem D.1's containment argument assumes F can be set to the identity vector for all pairs, which a shared-MLP parameterization may not realize exactly. Please state the precise parameterization and revise the proofs accordingly.
  4. [Tables 1 and 6] All reported results are single MSE/MAE values with no error bars, multiple seeds, or significance tests. Several differences are very small (e.g., ETTm2 FreDF Avg. 0.289/0.327 vs. 0.289/0.327; ECL iTransformer Avg. 0.174/0.267 vs. 0.175/0.266). Given the warm-start ambiguity identified above, variance information is needed to support the claim of consistent improvement, especially for datasets where the gap is below 1%.
minor comments (4)
  1. [Section 1] Typo: 'comapre' should be 'compare'.
  2. [Figures 1 and 8] Figure 8 in Appendix F.4 duplicates Figure 1; the figure numbering and cross-references are inconsistent. Please unify or clearly distinguish the two.
  3. [Table 4] The table reports 10 epochs for all datasets, but it is unclear whether the standard-attention baselines were trained for the same number of epochs or to convergence. Please clarify, since this bears on the fairness of the comparison.
  4. [Section 7.2] The claim that prime attention 'can match or slightly outperform standard attention with up to 40% less sequence length' is based on only two datasets shown in Fig. 1. Please state explicitly that this is a partial result and avoid generalizing it across all benchmarks.

Circularity Check

1 steps flagged

The 'dynamic relational learning' theorem is definitional (pair-specific F restates Def. 4.2), and Appendix G's fine-tuning protocol differs from the claimed 'simply swapping'; the empirical comparisons are otherwise not circular.

specific steps
  1. self definitional [Section 5, Theorem 5.1 (Eqs. 11-12, 14-20); Definition 4.2]
    "Definition 4.2 defines dynamic relational learning by h_{j→i}=ψ(x_j;θ_j,φ_ij), so h_{j→n} is not constrained to equal h_{j→i}. Theorem 5.1 then proves prime attention is dynamic: 'since F_ji and F_jv are independent learnable parameters, they are not constrained to be equal... Therefore, prime attention performs dynamic relational learning.'"

    Dynamic relational learning is defined as pair-specific conditioning of a token's representation (φ_ij in Def. 4.2). Prime attention constructs exactly that: per-pair learnable F_ij modulates key/value (Eqs. 11-12). The proof of Theorem 5.1 simply observes F_ji and F_jv are independent parameters, i.e., it substitutes F for φ and restates the definition. The claimed theoretical result is therefore true by construction, not derived from an independent principle; it adds no content beyond the method's definition.

full rationale

The paper's headline empirical claim—that prime attention lowers MSE/MAE relative to standard attention (Tables 1/6)—is an external benchmark comparison, not a quantity computed from a fitted parameter and then renamed a prediction; no self-citation is load-bearing (all baselines are external works). The main circularity-adjacent issue is Section 5's Theorem 5.1: the 'dynamic relational learning' property is built into the definition of F_ij, so the proof is a restatement rather than an independent result. Section D's UA theorem is a valid containment argument (F=1 recovers standard attention) and is not circular. Appendix G, however, states the transformer is 'first fine-tuned on standard attention and its attention module is subsequently replaced with prime attention only adjusting dropout rate,' which contradicts Section 7.1's 'simply swapping out the main attention block.' This is a correctness/fairness risk for the empirical comparison (warm-start and extra training budget), but it is not a circularity because the reported numbers are still empirical measurements; the paper should clarify or rerun from-scratch training. Overall, no derivation reduces to its own input in a way that forces the central empirical conclusion, so the circularity score is low.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new physical entities are introduced. The ledger's main entries are the added modulator parameters and the domain priors on which the empirical gains may actually depend. The theoretical results are largely definitional, so the burden falls on the empirical comparison, which is under-specified.

free parameters (3)
  • Pair-wise modulation F_ij (or parameters of the generating MLP) = Learned end-to-end; initialized from lead-lag plus instantaneous statistics (full initialization)
    This is the added capacity that defines prime attention; the paper does not report its parameter count or update details.
  • Projection matrix W mapping pairwise statistics to d_model = Not specified; used in s_ij initialization, F_ij = MLP(sigma(R)W)
    Introduced in Section 5; part of the modulator pipeline.
  • Instantaneous-correlation statistics (Pearson, rolling std, rank transformation) and their combination = Not specified
    Used in the full initialization ablation and in the final model, but the exact formulas are left to a brief appendix description in F.1.
axioms (4)
  • standard math Standard attention satisfies the universal approximation theorem (Cybenko 1989; Yun et al. 2020), and setting F_ij = 1 embeds standard attention in prime attention.
    Section D relies on this containment to claim prime attention inherits universal approximation.
  • domain assumption MTS channel-pair relationships are heterogeneous and static token representations limit their modeling.
    Motivating premise of Section 1; not independently tested.
  • domain assumption FFT-based lead-lag and instantaneous correlation priors are useful initializations for F.
    Section 5 and F.1 show these initializations outperform random init; the final model uses full initialization.
  • ad hoc to paper A shared MLP generating F_ij from pairwise statistics preserves the independence of F_ji and F_jv assumed in Theorem 5.1.
    Theorem 5.1 treats F_ij as independent parameters; Section 5 describes F_ij = MLP(s_ij), a shared function. The two parameterizations are not shown to be equivalent.

pith-pipeline@v1.3.0-alltime-deepseek · 23545 in / 17583 out tokens · 190437 ms · 2026-08-04T16:38:14.103001+00:00 · methodology

0 comments
read the original abstract

Standard attention mechanisms in transformers employ static token representations that remain unchanged across all pair-wise computations in each layer. This limits their representational alignment with the potentially diverse relational dynamics of each token-pair interaction. While they excel in domains with relatively homogeneous relationships, standard attention's static relational learning struggles to capture the diverse, heterogeneous inter-channel dependencies of multivariate time series (MTS) data--where different channel-pair interactions within a single system may be governed by entirely different physical laws or temporal dynamics. To better align the attention mechanism for such domain phenomena, we propose attention with dynamic relational priming (prime attention). Unlike standard attention where each token presents an identical representation across all of its pair-wise interactions, prime attention tailors each token dynamically (or per interaction) through learnable modulations to best capture the unique relational dynamics of each token pair, optimizing each pair-wise interaction for that specific relationship. This representational plasticity of prime attention enables effective extraction of relationship-specific information in MTS while maintaining the same asymptotic computational complexity as standard attention. Our results demonstrate that prime attention consistently outperforms standard attention across benchmarks, achieving up to 6.5\% improvement in forecasting accuracy. In addition, we find that prime attention achieves comparable or superior performance using up to 40\% less sequence length compared to standard attention, further demonstrating its superior relational modeling capabilities.

Figures

Figures reproduced from arXiv: 2509.12196 by Corey Clark, Hunjae Lee.

Figure 1
Figure 1. Figure 1: Comparing performance between standard (blue) and prime (yellow) attention at various [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Ablation study on sparsity of pair-wise modulator [PITH_FULL_IMAGE:figures/full_fig_p021_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Effect that different initialization strategies for pair-wise modulation primer [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Analysis of memory (left) and computation time (right) for standard (blue) and prime [PITH_FULL_IMAGE:figures/full_fig_p022_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Attention maps of standard attention (left), prime attention (middle), and their difference [PITH_FULL_IMAGE:figures/full_fig_p023_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Attention maps of standard attention (left), prime attention (middle), and their difference [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Attention maps of standard attention (left), prime attention (middle), and their difference [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗
Figure 1
Figure 1. Figure 1: fig. 1 [PITH_FULL_IMAGE:figures/full_fig_p024_1.png] view at source ↗
Figure 8
Figure 8. Figure 8: Comparing performance between standard (blue) and prime (yellow) attention at various [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Forecasting visualization on ETTh1 dataset using predictions (in blue) made from stan [PITH_FULL_IMAGE:figures/full_fig_p027_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Forecasting visualization on Solar dataset using predictions (in blue) made from standard [PITH_FULL_IMAGE:figures/full_fig_p027_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

61 extracted references · 16 linked inside Pith

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014

    Pith/arXiv arXiv 2014

  3. [3]

    Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference on Learning Representations, 2025

    Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veli c kovi \'c . Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=GtvuNrk58a

    2025

  4. [4]

    E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials

    Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13 0 (1): 0 2453, 2022

    2022

  5. [5]

    How attentive are graph attention networks? arXiv preprint arXiv:2105.14491, 2021

    Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? arXiv preprint arXiv:2105.14491, 2021

    Pith/arXiv arXiv 2021

  6. [6]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp.\ 213--229. Springer, 2020

    2020

  7. [7]

    Simple TM : A simple baseline for multivariate time series forecasting

    Hui Chen, Viet Luong, Lopamudra Mukherjee, and Vikas Singh. Simple TM : A simple baseline for multivariate time series forecasting. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=oANkBaVci5

    2025

  8. [8]

    From similarity to superiority: Channel clustering for time series forecasting

    Jialin Chen, Jan Eric Lenssen, Aosong Feng, Weihua Hu, Matthias Fey, Leandros Tassiulas, Jure Leskovec, and Rex Ying. From similarity to superiority: Channel clustering for time series forecasting. Advances in Neural Information Processing Systems, 37: 0 130635--130663, 2024

    2024

  9. [9]

    Tsmixer: An all-mlp architecture for time series forecasting

    Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan O Arik, and Tomas Pfister. Tsmixer: An all-mlp architecture for time series forecasting. arXiv preprint arXiv:2303.06053, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    George V. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2: 0 303--314, 1989. URL https://api.semanticscholar.org/CorpusID:3958369

    1989

  11. [11]

    Expander graph propagation

    Andreea Deac, Marc Lackenby, and Petar Veli c kovi \'c . Expander graph propagation. In Learning on Graphs Conference, pp.\ 38--1. PMLR, 2022

    2022

  12. [12]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp.\ 4171--4186, 2019

    2019

  13. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    Pith/arXiv arXiv 2010

  14. [14]

    Neural message passing for quantum chemistry

    Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp.\ 1263--1272. Pmlr, 2017

    2017

  15. [15]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp.\ 249--256, Chia Laguna Resort, Sardinia, Italy,...

    2010

  16. [16]

    Attention mechanisms in computer vision: A survey

    Meng-Hao Guo, Tian-Xing Xu, Jiang-Jiang Liu, Zheng-Ning Liu, Peng-Tao Jiang, Tai-Jiang Mu, Song-Hai Zhang, Ralph R Martin, Ming-Ming Cheng, and Shi-Min Hu. Attention mechanisms in computer vision: A survey. Computational visual media, 8 0 (3): 0 331--368, 2022

    2022

  17. [17]

    Inductive representation learning on large graphs

    Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017

    2017

  18. [18]

    Graph representation learning

    William L Hamilton. Graph representation learning. Morgan & Claypool Publishers, 2020

    2020

  19. [19]

    The capacity and robustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting

    Lu Han, Han-Jia Ye, and De-Chuan Zhan. The capacity and robustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting. IEEE Transactions on Knowledge and Data Engineering, 36 0 (11): 0 7129--7142, 2024

    2024

  20. [20]

    Heterogeneous graph transformer

    Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Proceedings of the web conference 2020, pp.\ 2704--2710, 2020

    2020

  21. [21]

    Transformers are graph neural networks

    Chaitanya K Joshi. Transformers are graph neural networks. arXiv preprint arXiv:2506.22084, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    On the expressive power of geometric graph neural networks

    Chaitanya K Joshi, Cristian Bodnar, Simon V Mathis, Taco Cohen, and Pietro Lio. On the expressive power of geometric graph neural networks. In International conference on machine learning, pp.\ 15330--15355. PMLR, 2023

    2023

  23. [23]

    Reversible instance normalization for accurate time-series forecasting against distribution shift

    Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=cGDAkQo1C0p

    2022

  24. [24]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    Pith/arXiv arXiv 2014

  25. [25]

    Semi-supervised classification with graph convolutional networks

    TN Kipf. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016

    Pith/arXiv arXiv 2016

  26. [26]

    Modeling long-and short-term temporal patterns with deep neural networks

    Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp.\ 95--104, 2018

    2018

  27. [27]

    Selective attention improves transformer

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Selective attention improves transformer. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=v0FzmPCd1e

    2025

  28. [28]

    Lagts: Toward adaptive lag relationship modeling for multivariate time series forecasting

    Ciyi Liu, Jiaqi Ye, Zhenpeng Yu, Shubao Zhao, Zhaoxiang Hou, Chengyi Yang, Yanlong Wen, and Xiaojie Yuan. Lagts: Toward adaptive lag relationship modeling for multivariate time series forecasting. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 1--5, 2025 a . doi:10.1109/ICASSP49660.2025.10890275

    arXiv 2025

  29. [29]

    Non-stationary transformers: Exploring the stationarity in time series forecasting

    Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in neural information processing systems, 35: 0 9881--9893, 2022

    2022

  30. [30]

    itransformer: Inverted transformers are effective for time series forecasting

    Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=JePfAI8fah

    2024

  31. [31]

    Timer- XL : Long-context transformers for unified time series forecasting

    Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer- XL : Long-context transformers for unified time series forecasting. In The Thirteenth International Conference on Learning Representations, 2025 b . URL https://openreview.net/forum?id=KMCJXjlDDr

    2025

  32. [32]

    A time series is worth 64 words: Long-term forecasting with transformers

    Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730, 2022

    Pith/arXiv arXiv 2022

  33. [33]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

    2019

  34. [34]

    A comprehensive survey of deep learning for multivariate time series forecasting: A channel strategy perspective

    Xiangfei Qiu, Hanyin Cheng, Xingjian Wu, Jilin Hu, Chenjuan Guo, and Bin Yang. A comprehensive survey of deep learning for multivariate time series forecasting: A channel strategy perspective. arXiv preprint arXiv:2502.10721, 2025

    arXiv 2025

  35. [35]

    Recipe for a general, powerful, scalable graph transformer

    Ladislav Ramp \'a s ek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems, 35: 0 14501--14515, 2022

    2022

  36. [36]

    Modeling relational data with graph convolutional networks

    Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference, pp.\ 593--607. Springer, 2018

    2018

  37. [37]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  38. [38]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  39. [39]

    Graph attention networks

    Petar Veli c kovi \'c , Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017

    Pith/arXiv arXiv 2017

  40. [40]

    softmax is not enough (for sharp out-of-distribution), 2025

    Petar Veli c kovi \'c , Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu. softmax is not enough (for sharp out-of-distribution), 2025. URL https://openreview.net/forum?id=wMj6PgKVuJ

    2025

  41. [41]

    Building powerful and equivariant graph neural networks with structural message-passing

    Clement Vignac, Andreas Loukas, and Pascal Frossard. Building powerful and equivariant graph neural networks with structural message-passing. Advances in neural information processing systems, 33: 0 14143--14155, 2020

    2020

  42. [42]

    Fre DF : Learning to forecast in the frequency domain

    Hao Wang, Lichen Pan, Yuan Shen, Zhichao Chen, Degui Yang, Yifei Yang, Sen Zhang, Xinggao Liu, Haoxuan Li, and Dacheng Tao. Fre DF : Learning to forecast in the frequency domain. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=4A9IdSa1ul

    2025

  43. [43]

    Timemixer++: A general time series pattern machine for universal predictive analysis

    Shiyu Wang, Jiawei Li, Xiaoming Shi, Zhou Ye, Baichuan Mo, Wenze Lin, Shengtong Ju, Zhixuan Chu, and Ming Jin. Timemixer++: A general time series pattern machine for universal predictive analysis. arXiv preprint arXiv:2410.16032, 2024 a

    Pith/arXiv arXiv 2024

  44. [44]

    Timemixer: Decomposable multiscale mixing for time series forecasting

    Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and JUN ZHOU. Timemixer: Decomposable multiscale mixing for time series forecasting. In International Conference on Learning Representations (ICLR), 2024 b

    2024

  45. [45]

    Timemixer: Decomposable multiscale mixing for time series forecasting

    Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and Jun Zhou. Timemixer: Decomposable multiscale mixing for time series forecasting. arXiv preprint arXiv:2405.14616, 2024 c

    Pith/arXiv arXiv 2024

  46. [46]

    Heterogeneous graph attention network

    Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Heterogeneous graph attention network. In The world wide web conference, pp.\ 2022--2032, 2019

    2022

  47. [47]

    Timexer: Empowering transformers for time series forecasting with exogenous variables

    Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Guo Qin, Haoran Zhang, Yong Liu, Yunzhong Qiu, Jianmin Wang, and Mingsheng Long. Timexer: Empowering transformers for time series forecasting with exogenous variables. Advances in Neural Information Processing Systems, 37: 0 469--498, 2024 d

    2024

  48. [48]

    Cayley graph propagation

    JJ Wilson, Maya Bechler-Speicher, and Petar Veli c kovi \'c . Cayley graph propagation. arXiv preprint arXiv:2410.03424, 2024

    Pith/arXiv arXiv 2024

  49. [49]

    Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting

    Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34: 0 22419--22430, 2021

    2021

  50. [50]

    How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018

    Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018

    Pith/arXiv arXiv 2018

  51. [51]

    Differential transformer

    Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=OvoCm1gGhN

    2025

  52. [52]

    Revitalizing multivariate time series forecasting: Learnable decomposition with inter-series dependencies and intra-series variations modeling

    Guoqi Yu, Jing Zou, Xiaowei Hu, Angelica I Aviles-Rivero, Jing Qin, and Shujun Wang. Revitalizing multivariate time series forecasting: Learnable decomposition with inter-series dependencies and intra-series variations modeling. arXiv preprint arXiv:2402.12694, 2024

    Pith/arXiv arXiv 2024

  53. [53]

    Are transformers universal approximators of sequence-to-sequence functions? In International Conference on Learning Representations, 2020

    Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=ByxRM0Ntvr

    2020

  54. [54]

    Graph transformer networks

    Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph transformer networks. Advances in neural information processing systems, 32, 2019

    2019

  55. [55]

    Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp.\ 11121--11128, 2023

    Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp.\ 11121--11128, 2023

    2023

  56. [56]

    Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting

    Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=vSVLM2j9eie

    2023

  57. [57]

    Rethinking channel dependence for multivariate time series forecasting: Learning from leading indicators

    Lifan Zhao and Yanyan Shen. Rethinking channel dependence for multivariate time series forecasting: Learning from leading indicators. arXiv preprint arXiv:2401.17548, 2024

    Pith/arXiv arXiv 2024

  58. [58]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  59. [59]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  60. [60]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  61. [61]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...