REVIEW 4 major objections 5 minor 40 references
Higher Order Transformers: Enhancing Stock Movement Prediction On Multimodal Time-Series Data
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A tensorized transformer hits 72.9% accuracy on stock moves
desk verdict A useful empirical paper whose 'higher-order' claim isn't validated by the implementation, but the factorized attention model itself is decent and worth reviewing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the fourth-order attention tensor $\mathcal{S} \in \mathbb{R}^{N \times N \times T \times T}$ defined in Equations (5)-(8), encoding how stock $i$ at time $t$ attends to stock $j$ at time $\tau$. The paper approximates it as a low-rank Kronecker decomposition, $\mathcal{S} \approx \sum_i S_i^{(1)} \otimes S_i^{(2)}$, with $S^{(1)}$ over stocks and $S^{(2)}$ over time, obtained by pooling the query and key tensors along one dimension before taking scaled dot products (Equations (16)-(18)). Kernelized linear attention then replaces the softmax attention matrices with a feature-map approximation, bringing the near-quadratic cost down to $O(d^2 N T)$. This machinery is what lets the model claim to be a higher-order transformer rather than two independent attention layers.
What would settle it
On a small subset of StockNet where the full attention tensor is computable (e.g., $N=10$ stocks, $T=5$ days), compute the exact $\mathcal{S}$ from Equations (5)-(8) and the factorized approximation from Equations (16)-(18), and compare their agreement; if the approximation error is large, the reported gains come from sequential stock-wise and time-wise attention rather than from a true higher-order interaction.
Extended reading notes
Core claim
The paper's central claim is that a fourth-order attention mechanism over both the stock dimension and the time dimension captures market dynamics that single-dimension attention misses. Concretely, the model tensorizes queries, keys, and values, defines attention scores over all pairs of stock-and-time tokens, then approximates the resulting attention tensor by a sum of Kronecker products of a stock-wise attention matrix and a time-wise attention matrix. Because computing the full tensor is costly, the approximation is paired with kernelized linear attention, which the paper states reduces complexity to $O(d^2 N T)$. In an encoder-decoder setup where tweet text is encoded and price history is decoded, the model reports 72.94% accuracy and 0.516 MCC on StockNet, second only to NL-LSTM, and its ablations attribute the gain to attention on both dimensions and to the multimodal input.
Load-bearing premise
The model assumes that the two attention matrices computed from pooled query and key embeddings faithfully reproduce the interactions captured by the full four-dimensional attention tensor, yet no experiment measures how close that low-rank approximation is.
Editorial extensions
If this is right
- Higher-order attention over both stocks and time improves stock movement prediction over attention restricted to either dimension alone, as shown by the ablation in Table 3.
- Kernelized linear attention does not just cut cost; in the reported ablations it also improves accuracy over standard attention for every modality tested.
- Combining tweet text with historical prices outperforms either single modality, with text contributing the larger part of the gain.
- Theorem 5.1 implies that any fourth-order attention tensor has an exact Kronecker representation at finite rank, so the factorization is a representation, not a heuristic, at full rank.
- The stock-level CLS token carried through the transformer yields a compact representation used for the binary movement classification.
Reading between the lines
- If the pooled factorization faithfully approximates the full attention tensor, the same construction extends to any number of modes beyond stock and time, such as sector or region, with analogous pooling and linear kernels.
- The paper leaves the approximation error between the full attention tensor and its factorized form unmeasured; a small-scale comparison would tell whether the higher-order label carries the improvement or whether two sequential attention layers would match it.
- Because text carries most of the signal on StockNet, the method's edge may be dataset-dependent; on price-dominated markets the ranking of modalities could reverse.
- NL-LSTM beats this model by a wide margin, suggesting that higher-order attention and neutrosophic sentiment modeling may capture complementary signals and could be combined.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces Higher Order Transformers, an architecture for stock movement prediction on multivariate time-series data. The model extends self-attention to a fourth-order tensor over stocks and time, proposes a low-rank Kronecker decomposition with pooled attention matrices S^(1) and S^(2), and applies kernelized linear attention to reduce complexity. A multimodal encoder-decoder combines FinBERT tweet embeddings with price features, and experiments on StockNet report ACC 72.94, F1 72.23, and MCC 0.516, placing the method second behind NL-LSTM. Ablation studies indicate that multimodal inputs and attention over both stock and time dimensions both improve accuracy.
Significance. If the factorized attention construction were shown to be a faithful low-rank approximation of the fourth-order attention tensor, the paper would make a useful contribution to multivariate time-series transformers: it proposes an attention mechanism over two interaction modes with linear-in-NT complexity and demonstrates competitive stock-movement accuracy on a standard benchmark. The paper also contains a formal existence result for Kronecker decompositions of attention matrices and a set of ablation experiments that clearly separate the contributions of modality and attention dimension. However, the central methodological claim is currently unverified: the connection between Theorem 5.1 and the implemented pooling-based attention matrices is not established, and no experiment measures the approximation error of the factorized attention tensor. The empirical comparison also lacks error bars and significance tests, so the headline performance claim is only a point estimate.
major comments (4)
- [Section 5.2.3, Eqs. (16)-(18), Theorem 5.1] The central claim that the model implements a low-rank Kronecker approximation of the higher-order attention tensor is not supported. Theorem 5.1 proves only that any matrix can be expressed as a sum of Kronecker products at sufficiently high rank, but the implemented S^(1) and S^(2) are computed from pooled query and key tensors in Eqs. (16)-(17), not from the SVD of the true attention tensor. The sequential application in Eq. (18) corresponds per head to the rank-one tensor \hat S_{ijt\tau} = S^(1)_{ij} S^(2)_{t\tau}. The manuscript provides neither a theoretical bound nor an experimental measure of the relative error between the full attention tensor S from Eqs. (5)-(8) and this factorized approximation. Without such evidence, the model may be two independently applied attention layers, and the reported empirical gains are not attributable to the proposed tensor-decomposition mechanism. Please add a fidelity experiment (e.g., relative Frobenius error on trained models or on random inputs) or a theoretical bound, or substantially revise the claim of what the architecture computes.
- [Section 6.1, label construction] The label threshold definition is incomplete. The text labels movements of at least 0.55% as positive and at most 0.5% as negative, leaving the interval (0.5%, 0.55%) unaccounted for. The paper does not state whether samples with movement percentages in this gap are discarded, relabeled, or assigned randomly. The same paragraph also says that samples lacking either prices or tweets are discarded, but the number of discarded samples is not reported. Since the baselines in Table 1 were reported under their own preprocessing pipelines, this ambiguity directly affects the comparability of accuracy and MCC values and must be clarified with exact counts.
- [Section 6.4 and Table 1] The performance comparison reports a single point estimate per method without standard deviations, number of independent runs, or significance tests, while hyperparameters are selected on validation F1. Consequently, the statement in Section 6.5 that the model outperforms all baselines except NL-LSTM is not statistically supported. Please report mean and standard deviation over multiple seeds and include a paired significance test against the strongest baselines (e.g., TEANet, CapTE, MAN-SF) under identical preprocessing and evaluation splits.
- [Section 5.2.3 and 5.2.4, multi-head formulation] The relationship between the rank-R Kronecker sum in Eq. (9) and the multi-head implementation is unclear. The text states that having R attention matrices acts similarly to multi-head attention and then says the multi-head mechanism is adopted instead of the summation, but it does not specify whether each head uses a separate pair (S^(1)_h, S^(2)_h), how the number of heads relates to R, or how Eq. (16)-(18) are combined across heads. This ambiguity prevents reproduction of the architecture and makes the complexity analysis in Section 5.2.4 difficult to verify. Please provide the exact per-head computation and how it maps to the Kronecker decomposition.
minor comments (5)
- [Section 5.2.4, Eq. (21)] Equation (21) uses g1(Q) and g1(K), but only the pooling functions f and g are defined; it should likely be f(Q) and f(K) for the stock-wise attention. Please correct the notation.
- [Section 5.2.4] The kernel attention description says the same kernel function as in Choromanski et al. is used, but it does not specify the feature-map dimension, kernel type, or approximation parameters. These details are needed for reproducibility and for verifying the O(d^2 N T) complexity claim.
- [Section 6.6, Table 3] The 'None' attention row is not defined. It should state whether this configuration uses no attention at all (e.g., only MLP layers) or pooling without attention, as the interpretation of the ablation depends on this definition.
- [Section 6.5] The phrase 'significantly outperforms' is used without any statistical test. Please use informal language such as 'achieves higher point estimates than' unless significance is actually established.
- [Abstract and Section 5.2.4] The claim of 'linear computational complexity' should specify that complexity is linear in the product N*T and still scales quadratically in the feature dimension d, which is stated as O(d^2 N T) but is described too loosely as 'linear with respect to the data size' in the abstract.
Circularity Check
No circular derivation found: the empirical claims are benchmarked externally and the linear-algebra theorem is independent and self-contained.
full rationale
The paper's central performance claim is validated against external baselines on the StockNet dataset (Table 1), so the reported accuracy, F1, and MCC are not constructed from the method's own assumptions. The low-rank Kronecker approximation is motivated by Theorem 5.1, which is a self-contained SVD-based existence result and does not depend on the model's outputs. The specific attention matrices S^(1) and S^(2) in Equations (16)-(17) are defined by pooling, and Equation (18) applies them sequentially; this is a modeling choice rather than a circular reduction. The kernelized linear attention is imported from the cited Performer work [6], which is external to this paper and not a self-citation. There are no load-bearing self-citations: the reference list contains no prior work by the present authors that is invoked to justify the central claims. The clearest weakness is that the paper asserts that the pooled factorized attention approximates the full fourth-order attention tensor, but it never measures the approximation error or connects Theorem 5.1 to the particular pooling construction. This is a correctness risk about whether the implemented model truly realizes the claimed higher-order tensor attention, but it is not circularity: the approximation is not defined in terms of the prediction target, and no fitted parameter is renamed as a prediction. Hyperparameters are selected on the validation set and the best model is reported on test, which is standard practice and not a fitted-input-called-prediction pattern. Therefore the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (3)
- Grid-search hyperparameters (hidden size, heads, blocks, dropout) =
not reported (search ranges: [32,64,128], [1,4,8,16], [2,4,6], [0,0.2,0.4])
- Kronecker rank R and actual attention factor count =
not reported
- Data preprocessing label thresholds =
positive >= 0.55%, negative <= 0.5%
assumptions (4)
- standard math Standard SVD rank decomposition is applicable to the reshaped attention tensor (Theorem 5.1).
- ad hoc to paper Permutation-invariant pooling functions (sum, mean, product) preserve enough information for S1 and S2.
- domain assumption The kernel from Choromanski et al. [6] faithfully approximates softmax attention in this financial time-series setting.
- domain assumption Baseline numbers from prior papers are directly comparable to the authors' runs.
Cite this review
Pith. "Pith review of Higher Order Transformers: Enhancing Stock Movement Prediction On Multimodal Time-Series Data." pith.science (2026). https://pith.science/paper/UICIBPCU
@misc{pith2026241210540,
author = {Pith},
title = {Pith review of: Higher Order Transformers: Enhancing Stock Movement Prediction On Multimodal Time-Series Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/UICIBPCU}},
note = {Machine review of arXiv:2412.10540}
}
read the original abstract
In this paper, we tackle the challenge of predicting stock movements in financial markets by introducing Higher Order Transformers, a novel architecture designed for processing multivariate time-series data. We extend the self-attention mechanism and the transformer architecture to a higher order, effectively capturing complex market dynamics across time and variables. To manage computational complexity, we propose a low-rank approximation of the potentially large attention tensor using tensor decomposition and employ kernel attention, reducing complexity to linear with respect to the data size. Additionally, we present an encoder-decoder model that integrates technical and fundamental analysis, utilizing multimodal signals from historical prices and related tweets. Our experiments on the Stocknet dataset demonstrate the effectiveness of our method, highlighting its potential for enhancing stock movement prediction in financial markets.
Figures
Reference graph
Works this paper leans on
-
[1]
Jeffrey S Abarbanell and Brian J Bushee. 1997. Fundamental analysis, future earnings, and stock prices. Journal of accounting research 35, 1 (1997), 1–24
work page 1997
-
[2]
Bassant A. Abdelfattah, Saad M. Darwish, and Saleh M. Elkaffas. 2024. Enhanc- ing the Prediction of Stock Market Movement Using Neutrosophic-Logic-Based Sentiment Analysis. Journal of Theoretical and Applied Electronic Commerce Research 19, 1 (2024), 116–134. https://doi.org/10.3390/jtaer19010007
-
[3]
Dogu Araci. 2019. FinBERT: Financial Sentiment Analysis with Pre-trained Language Models. arXiv:1908.10063 [cs.CL]
arXiv 2019
-
[4]
Tim Bollerslev. 1986. Generalized autoregressive conditional heteroskedasticity. Journal of econometrics 31, 3 (1986), 307–327
1986
-
[5]
Robert Goodell Brown. 2004. Smoothing, forecasting and prediction of discrete time series. Courier Corporation
work page 2004
-
[6]
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, An- dreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2022. Rethinking Attention with Performers. arXiv:2009.14794 [cs.LG]
arXiv 2022
-
[7]
Divyanshu Daiya and Che Lin. 2021. Stock movement prediction and portfolio management via multimodal learning with transformer. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 3305–3309
work page 2021
-
[8]
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024. Vision Transformers Need Registers. arXiv:2309.16588 [cs.CV]
arXiv 2024
Show all 40 references
-
[9]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs.CL]
2019 arXiv
-
[10]
Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. 2015. Deep learning for event-driven stock prediction. In Twenty-fourth international joint conference on artificial intelligence
2015
-
[11]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognit...
2021 arXiv
-
[12]
Fuli Feng, Xiangnan He, Xiang Wang, Cheng Luo, Yiqun Liu, and Tat-Seng Chua
-
[13]
Ronghang Hu and Amanpreet Singh. 2021. UniT: Multimodal Multitask Learning with a Unified Transformer. arXiv:2102.10772 [cs.CV]
2021 arXiv
-
[14]
Ziniu Hu, Weiqing Liu, Jiang Bian, Xuanzhe Liu, and Tie-Yan Liu. 2018. Listen- ing to chaotic whispers: A deep learning framework for news-oriented stock trend prediction. In Proceedings of the eleventh ACM international conference on web search and data mining. 261–269
2018
-
[15]
Ziniu Hu, Weiqing Liu, Jiang Bian, Xuanzhe Liu, and Tie-Yan Liu. 2019. Listen- ing to Chaotic Whispers: A Deep Learning Framework for News-oriented Stock Trend Prediction. arXiv:1712.02136 [cs.SI]
2019 arXiv
-
[16]
Joshi Kalyani, Prof Bharathi, Prof Jyothi, et al. 2016. Stock trend prediction using news sentiment analysis. arXiv preprint arXiv:1607.01958 (2016). Higher Order T ransformers: Enhancing Stock Movement Prediction On Multimodal Time-Series Data
2016 arXiv
-
[17]
Natthawut Kertkeidkachorn, Rungsiman Nararatwong, Ziwei Xu, and Ryutaro Ichise. 2023. FinKG: A Core Financial Knowledge Graph for Financial Analysis. In 2023 IEEE 17th International Conference on Semantic Computing (ICSC) . IEEE, 90–93
2023
-
[18]
Raehyun Kim, Chan Ho So, Minbyul Jeong, Sanghoon Lee, Jinkyu Kim, and Jaewoo Kang. 2019. Hats: A hierarchical graph attention network for stock movement prediction. arXiv preprint arXiv:1908.07999 (2019)
2019 arXiv
-
[19]
Qing Li, Jinghua Tan, Jun Wang, and Hsinchun Chen. 2020. A multimodal event- driven LSTM model for stock prediction using online news. IEEE Transactions on Knowledge and Data Engineering 33, 10 (2020), 3323–3337
2020
-
[20]
Jintao Liu, Hongfei Lin, Xikai Liu, Bo Xu, Yuqi Ren, Yufeng Diao, and Liang Yang. 2019. Transformer-Based Capsule Network For Stock Movement Prediction. In Proceedings of the First Workshop on Financial Technology and Natural Language Processing, Chung-Chi Chen, Hen-Hsen Huang...
2019
-
[21]
Daiki Matsunaga, Toyotaro Suzumura, and Toshihiro Takahashi. 2019. Exploring graph neural networks for stock market predictions with rolling window analysis. arXiv preprint arXiv:1909.10660 (2019)
2019 arXiv
-
[22]
Mainul Ah- san, Maishameem Meherin Muhu, Shahidul Islam Khan, and Mohammad Shafiul Alam
Tashreef Muhammad, Anika Bintee Aftab, Muhammad Ibrahim, Md. Mainul Ah- san, Maishameem Meherin Muhu, Shahidul Islam Khan, and Mohammad Shafiul Alam. 2023. Transformer-Based Deep Learning Model for Stock Price Pre- diction: A Case Study on Bangladesh Stock Market. Internationa...
2023 doi
-
[23]
Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Hena Ghonia, Rishika Bhagwatkar, Arian Khorasani, Mohammad Javad Darvishi Bayazi, George Adamopoulos, Roland Riachi, Nadhir Hassen, Marin Biloš, Sahil Garg, Anderson Schneider, Nicolas Chapados, Alexandre Drouin, Valentina Zan...
2024 arXiv
-
[24]
Ramit Sawhney, Shivam Agarwal, Arnav Wadhwa, and Rajiv Shah. 2020. Deep attentive learning for stock movement prediction from social media text and company correlations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 8415–8426
2020
-
[25]
Ramit Sawhney, Shivam Agarwal, Arnav Wadhwa, and Rajiv Ratn Shah. 2020. Spatiotemporal hypergraph convolution network for stock movement forecasting. In 2020 IEEE International Conference on Data Mining (ICDM). IEEE, 482–491
2020
-
[26]
Yejun Soun, Jaemin Yoo, Minyong Cho, Jihyeong Jeon, and U Kang. 2022. Accurate Stock Movement Prediction with Self-supervised Learning from Sparse Noisy Tweets. In 2022 IEEE International Conference on Big Data (Big Data). 1691–1700. https://doi.org/10.1109/BigData55660.2022.10020720
2022
-
[27]
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864 [cs.CL]
2023 arXiv
-
[28]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guil- laume Lample. 2023. LLaMA: Open and Efficient Foundation ...
2023 arXiv
-
[29]
Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal Transformer for Un- aligned Multimodal Language Sequences. arXiv:1906.00295 [cs.CL]
2019 arXiv
-
[30]
Gomez, Lukasz Kaiser, and Illia Polosukhin
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL]
2023 arXiv
-
[31]
Qianqian Xie, Weiguang Han, Yanzhao Lai, Min Peng, and Jimin Huang. 2023. The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges. arXiv:2304.05351 [cs.CL]
2023 arXiv
-
[32]
Yumo Xu and Shay B Cohen. 2018. Stock movement prediction from tweets and historical prices. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1970–1979
2018
-
[33]
Zheng Yangjia, Li Xia, Ma Junteng, and Chen Yuan. 2022. Fundamental Analysis based Neural Network for Stock Movement Prediction. In Proceedings of the 21st Chinese National Conference on Computational Linguistics, Maosong Sun, Yang Liu, Wanxiang Che, Yang Feng, Xipeng Qiu, Gao...
2022
-
[34]
Deniz Can Yıldırım, Ismail Hakkı Toroslu, and Ugo Fiore. 2021. Forecasting di- rectional movement of Forex data using LSTM with technical and macroeconomic indicators. Financial Innovation 7 (2021), 1–36
2021
-
[35]
Libo Yin and Qingyuan Yang. 2016. Predicting the oil prices: do technical indicators help? Energy Economics 56 (2016), 338–350
2016
-
[36]
Biao Zhang and Rico Sennrich. 2019. Root Mean Square Layer Normalization. arXiv:1910.07467 [cs.LG]
2019 arXiv
-
[37]
Qiuyue Zhang, Chao Qin, Yunfeng Zhang, Fangxun Bao, Caiming Zhang, and Peide Liu. 2022. Transformer-based attention network for stock movement prediction. Expert Systems with Applications 202 (2022), 117239. https://doi. org/10.1016/j.eswa.2022.117239
2022
-
[38]
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. arXiv:2012.07436 [cs.LG]
2021 arXiv
-
[39]
Jinan Zou, Haiyao Cao, Lingqiao Liu, Yuhao Lin, Ehsan Abbasnejad, and Javen Qinfeng Shi. 2022. Astock: A New Dataset and Automated Stock Trading based on Stock-specific News Analyzing Model. arXiv:2206.06606 [cs.CL]
2022 arXiv
-
[2019]
ACM Transactions on Information Systems (TOIS) 37, 2 (2019), 1–30
Temporal relational ranking for stock prediction. ACM Transactions on Information Systems (TOIS) 37, 2 (2019), 1–30
2019
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.