{"id":"3fd7c364-66e8-4c25-88d4-258856b3b917","arxiv_id":"2412.10540","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A factorized 'higher-order' transformer with kernelized linear attention and tweet plus price inputs reaches 72.94% accuracy and 0.516 MCC on StockNet, behind only NL-LSTM among the baselines compared.","lead":"This paper reports a transformer variant that applies attention separately over stocks and over time, adds tweet text through an encoder-decoder design, and achieves 72.94% accuracy on StockNet stock movement prediction, second among the listed baselines. It is worth reading as a candidate efficient multimodal time-series model, but the authors release no code, report no error bars, and the 'higher-order' description overstates what the factorized attention actually computes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The implemented attention is not shown to be the claimed low-rank Kronecker approximation; without an error measure, the higher-order contribution and the reported gains are not tied to the proposed tensor mechanism.","rationale":"The load-bearing point is that the paper sells a specific architectural innovation—'Higher Order Transformers'—and then evaluates it. The innovation is defined by the fourth-order attention tensor in Eqs. (5)-(8) and the claim that Eq. (18) approximates it at linear cost. The actual model replaces the pairwise logits z_{ijtτ} from Eq. (8) by pooled logits that are additively separable in stock and time (after softmax, multiplicatively separable). This is a strong structural assumption: it says the attention a stock-time pair pays to another pair factors as a stock-only factor times a time-only factor. Nothing in Theorem 5.1 shows that the softmax attention logits of the actual Transformer have this separability, and the theorem's universality at full rank does not make a rank-1 (or R-term) product of two independently normalized softmax matrices a good approximation at the chosen R. No experiment checks the approximation error, so the paper's central contribution is currently a heuristic with an unverified equivalence. This does not make the empirical numbers false, but it means the results do not establish the higher-order mechanism as the source of the gains. A reader should not accept the contribution as stated without either (i) measuring the approximation error, or (ii) explicitly reframing the model as factorized/sequential attention and dropping the tensor-decomposition justification. This matches the reader's CONDITIONAL verdict and strengthens it. Credit is due for the ablation study and the linear-attention implementation, but those do not close the gap.","tokens_in":10628,"tokens_out":8468,"duration_ms":81033,"concrete_test":"Using a trained model on StockNet (or a reimplementation, since no code is provided), fix a validation batch and compute the exact softmax tensor S from Eq. (7) with the model's Q and K (N=88, T=5, so about 1.9e5 entries per head). Compare it to the factorized \\hat S = S^(1)⊗S^(2) used in Eq. (18) by relative Frobenius error, per head and averaged. If the median relative error exceeds 0.5 (or is close to the error of a random rank-1 baseline), the implemented attention is not a faithful low-rank approximation and the paper should be reframed as two sequential attention mechanisms; if the error is small, the higher-order claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (18) computes attention by applying a stock-wise matrix S^(1) and then a time-wise matrix S^(2) sequentially. This is equivalent to a per-head attention tensor \\hat S_{ijtτ} = (S^(1))_{ij}(S^(2))_{tτ}. The paper's central methodological claim is that this factorized, pooled construction is a low-rank Kronecker approximation of the full fourth-order attention tensor defined by Eqs. (5)-(8). Theorem 5.1 only establishes that every matrix can be written as a sum of Kronecker products at sufficiently high rank; it does not show that the particular S^(1) and S^(2) obtained from pooled queries and keys in Eqs. (16)-(17) are the Kronecker factors of the true attention tensor, nor does it give an error bound. No experiment measures ||S − \\hat S||_F / ||S||_F or any related fidelity metric. If the factorized S^(1)⊗S^(2) is far from the softmax attention tensor S, then the model is not a higher-order transformer in the sense claimed; it is two independently applied attention layers whose empirical success does not validate the higher-order/tensor-decomposition narrative. This is a correctness risk for the contribution's central claim, separate from reproducibility of the accuracy numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces Higher Order Transformers, an architecture for stock movement prediction on multivariate time-series data. The model extends self-attention to a fourth-order tensor over stocks and time, proposes a low-rank Kronecker decomposition with pooled attention matrices S^(1) and S^(2), and applies kernelized linear attention to reduce complexity. A multimodal encoder-decoder combines FinBERT tweet embeddings with price features, and experiments on StockNet report ACC 72.94, F1 72.23, and MCC 0.516, placing the method second behind NL-LSTM. Ablation studies indicate that multimodal inputs and attention over both stock and time dimensions both improve accuracy.","tokens_in":10870,"tokens_out":5562,"duration_ms":53404,"significance":"If the factorized attention construction were shown to be a faithful low-rank approximation of the fourth-order attention tensor, the paper would make a useful contribution to multivariate time-series transformers: it proposes an attention mechanism over two interaction modes with linear-in-NT complexity and demonstrates competitive stock-movement accuracy on a standard benchmark. The paper also contains a formal existence result for Kronecker decompositions of attention matrices and a set of ablation experiments that clearly separate the contributions of modality and attention dimension. However, the central methodological claim is currently unverified: the connection between Theorem 5.1 and the implemented pooling-based attention matrices is not established, and no experiment measures the approximation error of the factorized attention tensor. The empirical comparison also lacks error bars and significance tests, so the headline performance claim is only a point estimate.","major_comments":[{"comment":"The central claim that the model implements a low-rank Kronecker approximation of the higher-order attention tensor is not supported. Theorem 5.1 proves only that any matrix can be expressed as a sum of Kronecker products at sufficiently high rank, but the implemented S^(1) and S^(2) are computed from pooled query and key tensors in Eqs. (16)-(17), not from the SVD of the true attention tensor. The sequential application in Eq. (18) corresponds per head to the rank-one tensor \\hat S_{ijt\\tau} = S^(1)_{ij} S^(2)_{t\\tau}. The manuscript provides neither a theoretical bound nor an experimental measure of the relative error between the full attention tensor S from Eqs. (5)-(8) and this factorized approximation. Without such evidence, the model may be two independently applied attention layers, and the reported empirical gains are not attributable to the proposed tensor-decomposition mechanism. Please add a fidelity experiment (e.g., relative Frobenius error on trained models or on random inputs) or a theoretical bound, or substantially revise the claim of what the architecture computes.","section":"Section 5.2.3, Eqs. (16)-(18), Theorem 5.1"},{"comment":"The label threshold definition is incomplete. The text labels movements of at least 0.55% as positive and at most 0.5% as negative, leaving the interval (0.5%, 0.55%) unaccounted for. The paper does not state whether samples with movement percentages in this gap are discarded, relabeled, or assigned randomly. The same paragraph also says that samples lacking either prices or tweets are discarded, but the number of discarded samples is not reported. Since the baselines in Table 1 were reported under their own preprocessing pipelines, this ambiguity directly affects the comparability of accuracy and MCC values and must be clarified with exact counts.","section":"Section 6.1, label construction"},{"comment":"The performance comparison reports a single point estimate per method without standard deviations, number of independent runs, or significance tests, while hyperparameters are selected on validation F1. Consequently, the statement in Section 6.5 that the model outperforms all baselines except NL-LSTM is not statistically supported. Please report mean and standard deviation over multiple seeds and include a paired significance test against the strongest baselines (e.g., TEANet, CapTE, MAN-SF) under identical preprocessing and evaluation splits.","section":"Section 6.4 and Table 1"},{"comment":"The relationship between the rank-R Kronecker sum in Eq. (9) and the multi-head implementation is unclear. The text states that having R attention matrices acts similarly to multi-head attention and then says the multi-head mechanism is adopted instead of the summation, but it does not specify whether each head uses a separate pair (S^(1)_h, S^(2)_h), how the number of heads relates to R, or how Eq. (16)-(18) are combined across heads. This ambiguity prevents reproduction of the architecture and makes the complexity analysis in Section 5.2.4 difficult to verify. Please provide the exact per-head computation and how it maps to the Kronecker decomposition.","section":"Section 5.2.3 and 5.2.4, multi-head formulation"}],"minor_comments":[{"comment":"Equation (21) uses g1(Q) and g1(K), but only the pooling functions f and g are defined; it should likely be f(Q) and f(K) for the stock-wise attention. Please correct the notation.","section":"Section 5.2.4, Eq. (21)"},{"comment":"The kernel attention description says the same kernel function as in Choromanski et al. is used, but it does not specify the feature-map dimension, kernel type, or approximation parameters. These details are needed for reproducibility and for verifying the O(d^2 N T) complexity claim.","section":"Section 5.2.4"},{"comment":"The 'None' attention row is not defined. It should state whether this configuration uses no attention at all (e.g., only MLP layers) or pooling without attention, as the interpretation of the ablation depends on this definition.","section":"Section 6.6, Table 3"},{"comment":"The phrase 'significantly outperforms' is used without any statistical test. Please use informal language such as 'achieves higher point estimates than' unless significance is actually established.","section":"Section 6.5"},{"comment":"The claim of 'linear computational complexity' should specify that complexity is linear in the product N*T and still scales quadratically in the feature dimension d, which is stated as O(d^2 N T) but is described too loosely as 'linear with respect to the data size' in the abstract.","section":"Abstract and Section 5.2.4"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope and the empirical result on StockNet is interesting, but the main architectural claim rests on an unverified equivalence between the pooled factorized attention and the fourth-order attention tensor. This is a fixable issue: the authors could measure approximation fidelity or reframe their contribution as a factorized two-mode attention mechanism. The lack of error bars and the ambiguous label threshold are additional load-bearing concerns that should be addressed before publication. I do not see evidence of circularity; the benchmarks are external and the SVD theorem is standard linear algebra. If the fidelity question is resolved, the paper may be suitable for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nQuick take: this is a readable empirical paper with an interesting architecture, but the main theoretical hook—that the model is a low-rank approximation of a fourth-order attention tensor—is not actually supported. What is implemented is two standard attention matrices computed on pooled features and applied sequentially, which is equivalent to a Kronecker-product attention matrix. Theorem 5.1 is just SVD; it doesn't show that the specific S1 and S2 from Equations (16)-(17) are the factors of the true attention tensor, and no experiment measures the approximation error. So the 'higher-order' framing is largely a narrative, and the actual model is a factorized attention transformer.\n\nThat said, the empirical story is not bad. The combination of stock-wise and time-wise factorized attention with kernelized linear attention and a multimodal encoder-decoder appears to be new relative to the cited literature. The ablations are sensible: they show kernelized attention helps over standard attention, and both dimensions of attention help over single-dimension. The reported 72.94% accuracy and 0.516 MCC on StockNet would make it second-best after NL-LSTM, with a comfortable margin over the rest. The linear complexity claim O(d^2 N T) checks out for the factorized implementation.\n\nThe soft spots beyond the theory gap: no code, no error bars or significance tests, and baselines are not rerun, so the comparisons are point estimates from prior papers. The label threshold has a 0.5%/0.55% gap, and samples with missing modalities are dropped. These are standard weaknesses in this literature, but they matter. Also, the exact Kronecker rank R mentioned in Section 5.2.3 never appears in the configuration (they use multi-head attention instead, effectively), so the connection between the decomposition and the number of heads is vague.\n\nIs it worth a serious referee? Yes. The empirical result is interesting enough, and the theory-vs-implementation gap is fixable by either implementing the full Kronecker approximation and measuring error, or honestly reframing the contribution as factorized attention. I'd send it to review with a request for code and variance. It's not a desk reject; it's a conditional accept with major revision.","headline":"A useful empirical paper whose 'higher-order' claim isn't validated by the implementation, but the factorized attention model itself is decent and worth reviewing.","tokens_in":11422,"tokens_out":2802,"would_cite":false,"duration_ms":22567,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A tensorized transformer hits 72.9% accuracy on stock moves","keywords":["Stock Movement Prediction","Multivariate Time-Series","Self-Attention","Transformer","Tensor Decomposition","Kernel Attention","Multimodal Signals","Higher-Order Attention"],"falsifier":"On a small subset of StockNet where the full attention tensor is computable (e.g., $N=10$ stocks, $T=5$ days), compute the exact $\\mathcal{S}$ from Equations (5)-(8) and the factorized approximation from Equations (16)-(18), and compare their agreement; if the approximation error is large, the reported gains come from sequential stock-wise and time-wise attention rather than from a true higher-order interaction.","tokens_in":10383,"feed_emoji":"📈","tokens_out":5911,"duration_ms":48037,"temperature":0.7,"pith_summary":"The paper claims that extending self-attention from matrices to higher-order tensors, so that a transformer can attend over stocks and time simultaneously, improves stock movement prediction on multimodal price-and-tweet data. On the StockNet benchmark, the proposed Higher Order Transformer reaches 72.94% accuracy, 72.23 F1, and 0.516 MCC, outperforming nearly all listed baselines; only one recent model, NL-LSTM, scores higher. To keep the larger attention tensor tractable, the authors factor it into stock-wise and time-wise attention matrices and use kernelized linear attention, claiming linear complexity in the data size. The ablation study shows that attending over both dimensions beats attending over either one alone, and that combining text with prices beats either modality alone.","feed_headline":"A tensorized transformer hits 72.9% accuracy on stock moves","feed_subtitle":"Higher-order attention over stocks and days tops nearly all baselines on StockNet, with linear-time kernel attention.","key_machinery":"The central object is the fourth-order attention tensor $\\mathcal{S} \\in \\mathbb{R}^{N \\times N \\times T \\times T}$ defined in Equations (5)-(8), encoding how stock $i$ at time $t$ attends to stock $j$ at time $\\tau$. The paper approximates it as a low-rank Kronecker decomposition, $\\mathcal{S} \\approx \\sum_i S_i^{(1)} \\otimes S_i^{(2)}$, with $S^{(1)}$ over stocks and $S^{(2)}$ over time, obtained by pooling the query and key tensors along one dimension before taking scaled dot products (Equations (16)-(18)). Kernelized linear attention then replaces the softmax attention matrices with a feature-map approximation, bringing the near-quadratic cost down to $O(d^2 N T)$. This machinery is what lets the model claim to be a higher-order transformer rather than two independent attention layers.","core_discovery":"The paper's central claim is that a fourth-order attention mechanism over both the stock dimension and the time dimension captures market dynamics that single-dimension attention misses. Concretely, the model tensorizes queries, keys, and values, defines attention scores over all pairs of stock-and-time tokens, then approximates the resulting attention tensor by a sum of Kronecker products of a stock-wise attention matrix and a time-wise attention matrix. Because computing the full tensor is costly, the approximation is paired with kernelized linear attention, which the paper states reduces complexity to $O(d^2 N T)$. In an encoder-decoder setup where tweet text is encoded and price history is decoded, the model reports 72.94% accuracy and 0.516 MCC on StockNet, second only to NL-LSTM, and its ablations attribute the gain to attention on both dimensions and to the multimodal input.","pith_inferences":["If the pooled factorization faithfully approximates the full attention tensor, the same construction extends to any number of modes beyond stock and time, such as sector or region, with analogous pooling and linear kernels.","The paper leaves the approximation error between the full attention tensor and its factorized form unmeasured; a small-scale comparison would tell whether the higher-order label carries the improvement or whether two sequential attention layers would match it.","Because text carries most of the signal on StockNet, the method's edge may be dataset-dependent; on price-dominated markets the ranking of modalities could reverse.","NL-LSTM beats this model by a wide margin, suggesting that higher-order attention and neutrosophic sentiment modeling may capture complementary signals and could be combined."],"forward_implications":["Higher-order attention over both stocks and time improves stock movement prediction over attention restricted to either dimension alone, as shown by the ablation in Table 3.","Kernelized linear attention does not just cut cost; in the reported ablations it also improves accuracy over standard attention for every modality tested.","Combining tweet text with historical prices outperforms either single modality, with text contributing the larger part of the gain.","Theorem 5.1 implies that any fourth-order attention tensor has an exact Kronecker representation at finite rank, so the factorization is a representation, not a heuristic, at full rank.","The stock-level CLS token carried through the transformer yields a compact representation used for the binary movement classification."],"supporting_citations":[{"why":"Supplies the StockNet dataset, the binary movement labeling, and the StockNet baseline the model must beat.","marker":"[32]"},{"why":"Provides the data-processing methodology (5-day lag windows, sample filtering) and the MAN-SF baseline.","marker":"[24]"},{"why":"Supplies the kernelized linear attention mechanism used to reduce computational complexity to $O(d^2 N T)$.","marker":"[6]"},{"why":"Supplies the original transformer self-attention formulation that the paper extends to higher order.","marker":"[30]"},{"why":"Provides the pretrained financial language model used to encode tweet text.","marker":"[3]"},{"why":"Describes NL-LSTM, the strongest baseline that outperforms the paper's model.","marker":"[2]"}],"fun_headline_variants":["Tensorized stock transformer hits 72.9% with linear-time attention","Fourth-order attention over stocks and days: 72.9% on StockNet","Multimodal transformer with tensor attention approaches 72.9% on StockNet","Tensorized transformer fuses tweets and prices, 72.9% on StockNet"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model assumes that the two attention matrices computed from pooled query and key embeddings faithfully reproduce the interactions captured by the full four-dimensional attention tensor, yet no experiment measures how close that low-rank approximation is.","fun_headline_variants_meta":{"raw":{"variants":["Tensorized stock transformer hits 72.9% with linear-time attention","Fourth-order attention over stocks and days: 72.9% on StockNet","Multimodal transformer with tensor attention approaches 72.9% on StockNet","Tensorized transformer fuses tweets and prices, 72.9% on StockNet"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000851,"raw_usage":{"total_tokens":3650,"prompt_tokens":844,"completion_tokens":2806,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":2732}},"tokens_in":460,"tokens_out":2806,"duration_ms":19163,"temperature":1.0,"reasoning_tokens":2732,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:52:08.219424+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a small subset of StockNet where the full attention tensor is computable (e.g., $N=10$ stocks, $T=5$ days), compute the exact $\\mathcal{S}$ from Equations (5)-(8) and the factorized approximation from Equations (16)-(18), and compare their agreement; if the approximation error is large, the reported gains come from sequential stock-wise and time-wise attention rather than from a true higher-order interaction.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the StockNet dataset, the binary movement labeling, and the StockNet baseline the model must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the data-processing methodology (5-day lag windows, sample filtering) and the MAN-SF baseline."},{"cited_title":"Abdelfattah, Saad M","cited_arxiv_id":null,"evidence_quote":"Describes NL-LSTM, the strongest baseline that outperforms the paper's model."}],"review_version":1}