Pith. sign in

REVIEW 3 major objections 5 minor 60 references

Continual Low-Rank Scaled Dot-product Attention

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces the Continual Nyströmformer, a Nyström-based Scaled Dot-product Attention whose incremental updates process streaming tokens in linear time and cut FLOPs by up to three orders of magnitude while retaining predictive…

desk verdict Good idea, useful fixed-landmark training trick, but the central update equation is algebraically wrong as written and the memory numbers ignore the token buffer; needs a careful revision before it can be trusted. read the letter →

arxiv 2412.03214 v4 pith:4ACUJGQW submitted 2024-12-04 cs.CV cs.LG

Ginés Carreto Picón , Illia Oleksiienko , Lukas Hedegaard , Arian Bakhtiarnia , Alexandros Iosifidis This is my paper · ORCID
classification cs.CVcs.LG
keywords ContinualInferenceScaledDot-productAttentionNyströmapproximationlow-rankTransformerlandmarkselectiononlineactiondetectionaudioclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to make Transformer attention usable in continual inference, where a model must produce a prediction for every incoming token of a stream without recomputing the whole sliding window. It introduces the Continual Nyströmformer, a Nyström-based low-rank formulation of Scaled Dot-product Attention whose per-step updates reuse cached intermediate matrices, giving linear per-step cost in the sequence length and, for the fixed-landmark single-output variant, cost independent of sequence length. The central move is to decouple the numerator and denominator of the softmax inside the Nyström approximation and update them incrementally as tokens slide through the window. In Audio Classification, Online Action Detection, and electricity forecasting experiments, the reported FLOP counts drop by up to three orders of magnitude relative to standard attention while accuracy remains comparable to the original and Continual Transformer baselines.

What carries the argument

The load-bearing object is the Nyström approximation of the softmax attention matrix, $s(QK^\top/\sqrt d)\approx s(Q\tilde K^\top/\sqrt d)\,s(\tilde Q\tilde K^\top/\sqrt d)^\dagger\,s(\tilde QK^\top/\sqrt d)$, where $\tilde Q,\tilde K$ are $m\times d$ landmark matrices and $\dagger$ denotes the Moore–Penrose pseudo-inverse. The paper makes this object continual by splitting each softmax factor into numerator and denominator, caching the row-sum vectors $\phi(B),\phi(\Gamma),\phi(\Delta)$ and the product $\Delta V$, and updating them incrementally as tokens enter and leave the window. The pseudo-inverse $(\Gamma_\phi)^\dagger$ is the one piece that resists incremental updating: it is recomputed with an iterative method when landmarks change, and computed once during training when landmarks are fixed.

What would settle it

Implement the fixed-landmark single-output Continual Nyströmformer on a tiny synthetic stream (e.g., $n=4$, $m=2$, $d=2$) and compare its output to the exact Nyström attention recomputed from scratch at each step; the outputs will differ whenever the departing and arriving tokens both have nonzero similarity, because Eq. (24) conflates $1/(a-b+c)$ with $1/a-1/b+1/c$. For a concrete numeric check, take a row whose row-sum moves from 1 to 0.75 to 0.5: the true updated inverse is $1/(1-0.75+0.5)=4/3$, while Eq. (24) gives $1-4/3+2=5/3$.

Watch

Extended reading notes

Core claim

This paper's central claim is that the Nyström approximation of softmax attention can be maintained incrementally over a sliding window, making low-rank attention genuinely continual. The Continual Nyströmformer writes the attention as $\mathrm{Att}_{\mathrm{CoNy}}(q_{\mathrm{new}},k_{\mathrm{new}},v_{\mathrm{new}}) = (B_\phi(\Gamma_\phi)^\dagger \Delta_\phi)V$, where $B=\rho(Q,\tilde K)$, $\Gamma=\rho(\tilde Q,\tilde K)$, and $\Delta=\rho(\tilde Q,K)$ are exponential similarity matrices computed with $m$ landmark tokens $\tilde Q,\tilde K$, and the subscript $\phi$ denotes the softmax numerator/denominator decoupling $\Omega_\phi=\phi(\Omega)^{-1}\odot\Omega$. When a new token arrives, the similarity matrices are updated by removing the departing token's contribution and adding the new token's; the landmark matrices themselves are refreshed every $n/m$ steps using segment means (continual landmarks) or fixed after training by clustering the training data's Q and K features (fixed landmarks). The paper argues that this yields $O(nd)$ per-step cost, and $O(dm+m^2)$ for the fixed-landmark single-output variant, with accuracy comparable to the original Transformer and Continual Transformer on the evaluated tasks.

Load-bearing premise

The load-bearing premise is that Eq. (24) correctly updates the inverse row-sum vector $\phi(\Delta)^{-1}$ by subtracting and adding inverses of individual exponential terms; this is algebraically false, since the inverse of a sum is not the sum of inverses, and the fixed-landmark and non-updated-landmark variants rely on that update.

Editorial extensions

If this is right

  • With the fixed-landmark single-output attention, per-step cost becomes $O(dm+m^2)$ and no longer depends on the sequence length $n$, so streams can be processed with bounded latency as the window grows.
  • The incremental update reuses cached numerator and denominator quantities, eliminating the $n$-fold redundant recomputation that a sliding-window Transformer performs when each frame is processed multiple times.
  • On GTZAN audio classification, the single-layer fixed-landmark model reports a relative FLOP reduction of about $1000\times$ versus standard attention, at 90.76% accuracy versus 93.12% for the original attention baseline.
  • Because every intermediate layer is itself a Nyström layer, more Transformer layers can be stacked with lower computational overhead than in Continual Transformers, whose intermediate layers use full attention.
  • The same decouple-and-update pattern is proposed as a template for adapting other low-rank attention approximations to continual inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if Eq. (24) is corrected to recompute $\phi(\Delta)^{-1}$ by inverting the updated row sums elementwise, the fixed-landmark and non-updated-landmark variants remain well-defined, and their reported cost savings do not depend on the erroneous identity.
  • Beyond the paper: the fixed-landmark variant's sequence-length-independent cost suggests a drop-in attention module for latency-critical streaming settings such as robotics control, a use the paper does not test.
  • Beyond the paper: the observation that a few landmarks often match full attention may reflect redundancy in audio and video token streams; evaluating the method on non-redundant inputs, such as shuffled tokens or diverse text, would test how much of the accuracy retention relies on stream redundancy.
  • Beyond the paper: the amortized cost of the continually updated landmarks depends on the ratio $n/m$; selecting $m$ adaptively from stream statistics could trade accuracy against computation without changing the update equations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces the Continual Nyströmformer, a continual-inference adaptation of the Nyström-based low-rank approximation of Scaled Dot-product Attention. The authors derive update rules for the B, Γ, and Δ matrices that appear in the Nyström attention formula under a sliding window, and propose two landmark-selection schemes: continually updated landmarks based on segment means, and fixed landmarks obtained by k-means clustering of training tokens followed by fine-tuning. They report asymptotic computational and memory costs, FLOPs measurements, and experiments on audio classification (GTZAN), online action detection (THUMOS14), and electricity forecasting, claiming up to three orders of magnitude FLOPs reduction relative to standard attention while retaining accuracy comparable to competing continual and Nyström-based baselines.

Significance. The problem of making transformer-style attention applicable to online, resource-constrained inference is timely, and the idea of extending the Nyströmformer approximation to a continually updated sliding window is a natural and potentially useful contribution. The paper also introduces a fixed-landmark variant whose memory cost, if the derivation were correct, could be attractive. The authors provide a public code repository, a detailed FLOPs analysis, and experiments on multiple tasks, which are strengths. However, the core algebraic derivation of the non-updated-landmark and fixed-landmark variants contains a load-bearing error in the inverse row-sum update, an internal inconsistency in the updated-landmark update, and the memory accounting omits the key and value buffers needed to perform the sliding-window updates. These issues must be resolved before the reported results can be taken as validating the claimed method.

major comments (3)
  1. [Section IV-A2, Eq. (24)] Equation (24) is algebraically incorrect as the update for the inverse row-sum vector. The row sums of Δ satisfy φ(Δ) = φ(Δ)_{prev} − ρ(˜Q, k_old) + ρ(˜Q, k_new), so the correct inverse update is the elementwise reciprocal of that sum, i.e., 1/(φ(Δ)_{prev} − ρ(˜Q, k_old) + ρ(˜Q, k_new)). Equation (24) instead writes φ(Δ)^{-1} = φ(Δ)^{-1}_{prev} − ρ(˜Q, k_old)^{-1} + ρ(˜Q, k_new)^{-1}, which asserts that the reciprocal of a sum equals a sum of reciprocals. This is false for every nontrivial window. Because φ(Δ)^{-1} is used directly in Eqs. (23) and (26) and in the fixed-landmark variants of Section IV-B, the formulas as written do not compute the Nyström attention defined by Eq. (9). The reported results for the non-updated and fixed-landmark variants therefore do not establish the paper's central claim unless the released code implements the correct recurrence; the paper should be corrected and the code checked against the corrected equation.
  2. [Section IV-A1, Eq. (19)] Equation (19) contains an internal inconsistency: the subtracted and added terms in the first row block are both ρ(˜Q_mem, ˜k_new). Since the landmark update removes the oldest landmark ˜k_old and adds ˜k_new, the subtracted term should be ρ(˜Q_mem, ˜k_old), not another ρ(˜Q_mem, ˜k_new). As written, the update cancels these two identical terms and does not account for the eviction of the old landmark. This makes the updated-landmark derivation inconsistent with the stated landmark update in Eqs. (13)-(14).
  3. [Table I and Appendices O/P] The memory accounting omits the input token buffers needed to perform the sliding-window updates. For the non-updated and fixed-landmark variants, equations (24)-(25) require ρ(˜Q, k_old) and ρ(˜Q, k_new), and equation (25) additionally requires v_old and v_new. While k_new and v_new are available from the current input, k_old and v_old must have been stored from earlier tokens, so the model must maintain buffers for the key and value matrices (or equivalently for the raw token representations). The valley costs listed for AttFix_CoNySi and AttFix_CoNyRe in Table I and in Appendices O and P do not include these O(nd) buffers. The claimed valley memory of O(dm + m^2) for the fixed-landmark single-output variant is therefore understated by O(nd), which directly affects the comparisons in Figure 4 and the corresponding claims in Section V-B.
minor comments (5)
  1. [Conclusion, Section VI] The sentence "The use of landmarks may translate into a slightly lower performance at the expense of a slightly lower performance" is garbled; it appears that the second occurrence should refer to reduced computational cost, and the sentence should be rewritten.
  2. [Abstract vs. Conclusion] The abstract claims a reduction in the number of operations of "up to three orders of magnitude," while the conclusion states "up to two orders of magnitude." Table III does report a relative reduction of ×1028 for the one-layer fixed-landmark variant, so the abstract is supportable in that case, but the inconsistency should be resolved.
  3. [Table V] The relative FLOPs entries "144,04" and "294,35" use commas as decimal separators, which is inconsistent with the decimal-point notation used in Tables III and IV and with the surrounding text.
  4. [Figures 5-7 captions] The figure captions contain typesetting artifacts such as "....red.......lines" and "...red......lines"; these should be cleaned up.
  5. [Section V-C, experimental setup] The description of the online action detection experimental setup is brief; in particular, the number of training frames used for the learned fixed landmarks on THUMOS14 is specified as 50,000 tokens, but no information is given about how the five data seeds were combined with model initializations to produce the reported means and standard deviations in Table IV, making the uncertainty estimates difficult to interpret.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Continual Nyströmformer is derived algebraically from the Nyströmformer and Continual Transformer equations, and the reported FLOPs/memory/accuracy figures come from experiments against independent benchmarks rather than from the equations.

full rationale

I walked the claimed derivation chain. The central construction, Eq. (9), is a definitional decoupling of the Nyströmformer softmax approximation into numerator/denominator factors, and Eqs. (15)-(26) are algebraic update rules for B, Γ, Δ and ΔV in terms of cached quantities from the previous step; no parameter is fitted to make a target output come out. The complexity claims in Table I and Appendix A-I follow from counting operations on matrices of stated sizes (n, d, m), not from measured results. The empirical FLOPs, accuracy, mAP and MSE numbers in Tables III-V and Figures 4-7 are obtained by running the architectures on GTZAN, THUMOS14 and Electricity benchmarks, with landmarks selected on training data and evaluated on test data; no test result is used as an input to the derivation. Self-citations to [27] and [31]-[33] supply the Continual Inference definitions and are prior published work; they are not used as a forced uniqueness argument or as the only support for the new algebraic steps. The main weakness found—Eq. (24) updates φ(Δ)^{-1} by subtracting and adding reciprocals of individual exponential terms, which is not equivalent to taking the reciprocal after the row-sum update—is a mathematical correctness flaw in the written derivation, not a circularity, and per the scoring rubric it does not raise the circularity score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the Nyström approximation formula, the iterative pseudo-inverse method, and the correctness of the derived continual updates. No new physical or conceptual entities are introduced beyond the landmark abstraction inherited from Nyströmformer.

free parameters (2)
  • Number of landmarks m = 4, 16, 32, 64 in experiments
    User-chosen hyperparameter controlling approximation rank; evaluated over a range, not fitted to a target.
  • Subset size for k-means landmark selection = 50,000 tokens in THUMOS14
    Randomly chosen subset of training tokens used to compute fixed landmarks.
assumptions (4)
  • domain assumption The Nyström approximation of the softmax attention matrix (Eq. 8) is a valid low-rank approximation.
    Adopted from Nyströmformer [19].
  • standard math The Moore-Penrose pseudo-inverse can be approximated by the iterative method of Razavi et al. [44] with 24m^3 operations.
    Used to compute (Gamma_phi)^dagger; the paper cites [44].
  • domain assumption In continual inference, incremental updates to attention can produce outputs close to the non-continual counterpart.
    From Continual Transformers [31]; for Nyström approximation, approximate equality is assumed.
  • ad hoc to paper Landmarks computed as segment means or k-means cluster centroids provide sufficiently accurate Nyström approximation for the tested tasks.
    The paper chooses these landmark selection schemes without theoretical error bounds.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continual Low-Rank Scaled Dot-product Attention." pith.science (2026). https://pith.science/paper/4ACUJGQW

@misc{pith2026241203214,
  author       = {Pith},
  title        = {Pith review of: Continual Low-Rank Scaled Dot-product Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ACUJGQW}},
  note         = {Machine review of arXiv:2412.03214}
}
read the original abstract

Transformers are widely used for their ability to capture data relations in sequence processing, with great success for a wide range of static tasks. However, the computational and memory footprint of their main component, i.e., the Scaled Dot-product Attention, is commonly overlooked. This makes their adoption in applications involving stream data processing with constraints in response latency, computational and memory resources infeasible. Some works have proposed methods to lower the computational cost of Transformers, i.e. low-rank approximations, sparsity in attention, and efficient formulations for Continual Inference. In this paper, we introduce a new formulation of the Scaled Dot-product Attention based on the Nystr\"om approximation that is suitable for Continual Inference. In experiments on Online Audio Classification and Online Action Detection tasks, the proposed Continual Scaled Dot-product Attention can lower the number of operations by up to three orders of magnitude compared to the original Transformers while retaining the predictive performance of competing models.

Figures

Figures reproduced from arXiv: 2412.03214 by the authors.

Figure 1
Figure 1. Continual landmarks’ calculation (for n = 20 and m = 4). Until enough new input tokens have been received, the landmarks remain fixed. This includes the landmark calculated by some old tokens which have been shifted out of the current attention window (top). When enough input tokens have been received (bottom), a new landmark is computed using all the new tokens, replacing the q˜old landmark. The same approach is ap… view at source ↗
Figure 2
Figure 2. Continual Inference with updated landmarks for a sequence of [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Continual Inference with non-updated landmarks. As landmarks remain unchanged, we can update most of the previous matrices to save computations. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: (a) and (b) contain a comparison of computational cost in FLOPs [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Accuracy on the GTZAN dataset of the models using Nystr [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: mAP on the THUMOS14 dataset of the models using Nystr [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: mAP on the THUMOS14 dataset of the models using Nystr [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 46 canonical work pages

  1. [1]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , pages 5998–6008, 2017

    2017

  2. [2]

    Trans- formers in the real world: A survey on NLP applications

    Narendra Patwardhan, Stefano Marrone, and Carlo Sansone. Trans- formers in the real world: A survey on NLP applications. Information, 14(4):242, 2023

    work page 2023

  3. [3]

    Transformer architecture and attention mechanisms in genome data analysis: A comprehensive review

    Sanghyuk Roy Choi and Minhyeok Lee. Transformer architecture and attention mechanisms in genome data analysis: A comprehensive review. Biology, 12(7):1033, 2023

    work page 2023

  4. [4]

    Vision transformers for action recognition: A survey

    Anwaar Ulhaq, Naveed Akhtar, Ganna Pogrebna, and Ajmal Mian. Vision transformers for action recognition: A survey. arXiv:2209.05700, 2022

    arXiv 2022

  5. [5]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weis- senborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021

    2021

  6. [6]

    Singh, Muskaan Chopra, Sudhakar Kumar, and Francesco Colace

    Kriti Aggarwal, Sunil K. Singh, Muskaan Chopra, Sudhakar Kumar, and Francesco Colace. Deep Learning in Robotics for Strengthening Industry 4.0.: Opportunities, Challenges and Future Directions , volume 1030, chapter 1, pages 1–19. Springer International Publishing, 2022

    work page 2022

  7. [7]

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav M...

    work page 2023

  8. [8]

    Miller, Mohammed Aldosari, Farah Saeed, Nasid Habib Barna, Subas Rana, Ismailcem Budak Arpinar, and Ninghao Liu

    John A. Miller, Mohammed Aldosari, Farah Saeed, Nasid Habib Barna, Subas Rana, Ismailcem Budak Arpinar, and Ninghao Liu. A survey of deep learning and foundation models for time series forecasting. arXiv:2401.13912, 2024

    arXiv 2024

Show all 60 references
  1. [9]

    Single-layer Vision Transformers for more accurate early exits with less overhead

    Arian Bakhtiarnia, Qi Zhang, and Alexandros Iosifidis. Single-layer Vision Transformers for more accurate early exits with less overhead. Neural Networks, 153:461–473, 2022

    2022

  2. [10]

    Efficient High-Resolution Deep Learning: A Survey

    Arian Bakhtiarnia, Qi Zhang, and Alexandros Iosifidis. Efficient High-Resolution Deep Learning: A Survey. ACM Computing Surveys , 56(7):181:1–181:35, 2024

    2024

  3. [11]

    Reducing transformer depth on demand with structured dropout

    Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations , 2020

    2020

  4. [12]

    Transformer multivariate forecasting: Less is more? arXiv:2401.00230, 2024

    Jingjing Xu, Caesar Wu, Yuan-Fang Li, and Pascal Bouvry. Transformer multivariate forecasting: Less is more? arXiv:2401.00230, 2024

    2024 arXiv

  5. [13]

    Informer: Beyond efficient transformer for long sequence time-series forecasting

    Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI Conference on Artificial Intelligence, pages 11106–11115, 2021

    2021

  6. [14]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv:1904.10509, 2019

    1904 arXiv

  7. [15]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv:2004.05150, 2020

    2004 arXiv

  8. [16]

    Colwell, and Adrian Weller

    Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Do- han, Xingyou Song, Andreea Gane, Tam ´as Sarl ´os, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performer...

    2021

  9. [17]

    Petros Drineas and Michael W. Mahoney. On the nystr ¨om method for approximating a gram matrix for improved kernel-based learning. Journal on Machine Learning Research , 6:2153–2175, 2005

    2005

  10. [18]

    On compressing deep models by low rank and sparse decomposition

    Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In IEEE Conference on Computer Vision and Pattern Recognition , pages 67–76, 2017

    2017

  11. [19]

    Nystr ¨omformer: A nystr ¨om- based algorithm for approximating self-attention

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystr ¨omformer: A nystr ¨om- based algorithm for approximating self-attention. In AAAI Conference on Artificial Intelligence , pages 14138–14148, 2021

    2021

  12. [20]

    Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis E. H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to- token vit: Training vision transformers from scratch on imagenet. In IEEE/CVF International Conference on Computer Vision , pages 538– 547, 2021

    2021

  13. [21]

    Cvt: Introducing convolutions to vision transformers

    Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In IEEE/CVF International Conference on Computer Vision, pages 22–31, 2021

    2021

  14. [22]

    Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah

    Salman H. Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM Computing Surveys , 54(10s):200:1–200:41, 2022

    2022

  15. [23]

    Is space-time attention all you need for video understanding? In International Conference on Machine Learning , pages 813–824, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In International Conference on Machine Learning , pages 813–824, 2021

    2021

  16. [24]

    Video swin transformer

    Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3192–3201, 2022

    2022

  17. [25]

    Vivit: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. Vivit: A video vision transformer. In IEEE/CVF International Conference on Computer Vision , pages 6816– 6826, 2021

    2021

  18. [26]

    Multiscale vision transformers

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In IEEE/CVF International Conference on Computer Vision, pages 6804–6815, 2021

    2021

  19. [27]

    Continual inference: A library for efficient online inference with deep neural networks in pytorch

    Lukas Hedegaard and Alexandros Iosifidis. Continual inference: A library for efficient online inference with deep neural networks in pytorch. In European Conference on Computer Vision Workshops, pages 21–34, 2022

    2022

  20. [28]

    Robin M. Schmidt. Recurrent neural networks (rnns): A gentle intro- duction and overview. abs:1912.05911, abs/1912.05911, 2019

    1912 arXiv

  21. [29]

    Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network

    Alex Sherstinsky. Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. abs:1808.03314, abs/1808.03314, 2018

    2018 arXiv

  22. [30]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. abs:2312.00752, abs/2312.00752, 2023

    2023 arXiv

  23. [31]

    Con- tinual transformers: Redundancy-free attention for online inference

    Lukas Hedegaard, Arian Bakhtiarnia, and Alexandros Iosifidis. Con- tinual transformers: Redundancy-free attention for online inference. In International Conference on Learning Representations , 2023

    2023

  24. [32]

    Continual spatio-temporal graph convolutional networks

    Lukas Hedegaard, Negar Heidari, and Alexandros Iosifidis. Continual spatio-temporal graph convolutional networks. Pattern Recognition , 140(3):109528, 2023

    2023

  25. [33]

    Continual 3d convolutional neural networks for real-time processing of videos

    Lukas Hedegaard and Alexandros Iosifidis. Continual 3d convolutional neural networks for real-time processing of videos. In European Conference Computer Vision, pages 369–385, 2022

    2022

  26. [34]

    Vision xformers: Efficient attention for image classification

    Pranav Jeevan and Amit Sethi. Vision xformers: Efficient attention for image classification. arXiv:2107.02239, 2021

    2021 arXiv

  27. [35]

    Bernhard Sch ¨olkopf and Alexander J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond . The MIT Press, 2001. 15

    2001

  28. [36]

    Improving CUR matrix decomposition and the nystr ¨om approximation via adaptive sampling

    Shusen Wang and Zhihua Zhang. Improving CUR matrix decomposition and the nystr ¨om approximation via adaptive sampling. Journal of Machine Learning Research , 14(1):2729–2769, 2013

    2013

  29. [37]

    Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and Jo˜ao F

    Mandela Patrick, Dylan Campbell, Yuki M. Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and Jo˜ao F. Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. In Advances in Neural Information Processing Systems , pages 1...

    2021

  30. [38]

    Eventful transformers: Leveraging temporal redundancy in vision transformers

    Matthew Dutson, Yin Li, and Mohit Gupta. Eventful transformers: Leveraging temporal redundancy in vision transformers. In IEEE/CVF International Conference on Computer Vision , pages 16865–16877, 2023

    2023

  31. [39]

    SOFT: softmax-free transformer with linear complexity

    Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. SOFT: softmax-free transformer with linear complexity. In Advances in Neural Information Processing Systems, 2021

    2021

  32. [40]

    Adaptive multi-resolution attention with linear complexity

    Yao Zhang, Yunpu Ma, Thomas Seidl, and V olker Tresp. Adaptive multi-resolution attention with linear complexity. In International Joint Conference on Neural Networks, IJCNN , pages 1–8. IEEE, 2023

    2023

  33. [41]

    Kwok, Slobodan Vucetic, and Bahram Parvin

    Kai Zhang, Liang Lan, James T. Kwok, Slobodan Vucetic, and Bahram Parvin. Scaling up graph-based semisupervised learning via Prototype Vector Machines. IEEE Transactions on Neural Networks and Learning Systems, 26(3):444–457, 2015

    2015

  34. [42]

    Scaling Up Class-Specific Kernel Discriminant Analysis for Large-Scale Face Verification

    Alexandros Iosifidis and Moncef Gabbouj. Scaling Up Class-Specific Kernel Discriminant Analysis for Large-Scale Face Verification. IEEE Transactions on Information Forensics and Security, 11(11):2453–2465, 2016

    2016

  35. [43]

    Kai Zhang and James T. Kwok. Clustered nystr ¨om method for large scale manifold learning and dimension reduction. IEEE Trans. Neural Networks, 21, 2010

    2010

  36. [44]

    Razavi, A

    M. Razavi, A. Kerayechian, Mortaza Gachpazan, and Stanford Shateyi. A new iterative method for finding approximate inverses of complex matrices. Abstract and Applied Analysis , 2014(1):1–7, 2014

    2014

  37. [45]

    Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Brad- bury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K ¨opf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Stein...

    2019

  38. [46]

    Deep Learning

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016

    2016

  39. [47]

    A scale for the measurement of the psychological magnitude pitch

    Stanley Smith Stevens, John V olkmann, and Edwin Broomell Newman. A scale for the measurement of the psychological magnitude pitch. The Journal of the Acoustical Society of America , 8(3):185–190, 1937

    1937

  40. [48]

    Keunwoo Choi, Gy ¨orgy Fazekas, and Mark B. Sandler. Automatic tagging using deep convolutional neural networks. In International Society for Music Information Retrieval Conference , pages 805–811, 2016

    2016

  41. [49]

    Rethinking CNN models for audio classification

    Kamalesh Palanisamy, Dipika Singhania, and Angela Yao. Rethinking CNN models for audio classification. arXiv:2007.11154, 2020

    2007 arXiv

  42. [50]

    George Tzanetakis and Perry R. Cook. Musical genre classification of audio signals. IEEE Transactions on Speech and Audio Processing , 10(5):293–302, 2002

    2002

  43. [51]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations , 2015

    2015

  44. [52]

    Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, and Kevin W. Wilson. CNN architectures for large-scale audio classification. In IEEE I...

    2017

  45. [53]

    Online action detection

    Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees Snoek, and Tinne Tuytelaars. Online action detection. In European Conference on Computer Vision , pages 269–284, 2016

    2016

  46. [54]

    Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah

    Haroon Idrees, Amir R. Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The THUMOS challenge on action recognition for videos ”in the wild”. Computer Vision and Image Understanding, 155:1–23, 2017

    2017

  47. [55]

    Learning to discriminate information for online action detection: Analysis and application

    Sumin Lee, Hyunjun Eun, Jinyoung Moon, Seokeon Choi, Yoonhyung Kim, Chanho Jung, and Changick Kim. Learning to discriminate information for online action detection: Analysis and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5918– 5934, 2023

    2023

  48. [56]

    Temporal segment networks for action recognition in videos

    Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks for action recognition in videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(11):2740–2755, 2019

    2019

  49. [57]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Car- los Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In IEEE Conference on Computer Vision and Pattern Recognition, pages 961–970, 2015

    2015

  50. [58]

    Quo vadis, action recognition? A new model and the kinetics dataset

    Jo ˜ao Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In IEEE Conference on Computer Vision and Pattern Recognition , pages 4724–4733, 2017

    2017

  51. [59]

    ElectricityLoadDiagrams20112014

    Artur Trindade. ElectricityLoadDiagrams20112014. UCI Machine Learning Repository, 2015. DOI: https://doi.org/10.24432/C58C86

    2015 doi

  52. [60]

    Decoupled weight decay regular- ization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regular- ization. In 7th International Conference on Learning Representations, ICLR, 2019

    2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.