Pith. sign in

REVIEW 3 major objections 5 minor 118 references

A Survey of Retentive Network

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This survey argues that RetNet's retention mechanism—a recurrent state update that is algebraically equivalent to a masked attention with exponential decay—makes it a practical, efficient alternative to the Transformer, and presents the…

desk verdict A workmanlike catalog of RetNet applications, but the architecture recap is sloppy and the 'comprehensive survey' claim outruns the stated method. read the letter →

arxiv 2506.06708 v1 pith:UO3XAJQQ submitted 2025-06-07 cs.CL

Haiqi Yang , Zhiyuan Li , Yi Chang , Yuan Wu This is my paper
classification cs.CL
keywords RetentiveNetworksequencemodelinglinear-timeinferenceretentionmechanismTransformeralternativessurveymulti-scale
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that Retentive Network (RetNet) is a viable successor to the Transformer for sequence modeling, because its retention mechanism combines recurrent and attention-style computation: it trains in parallel like attention, infers in linear time with constant memory, and keeps up with Transformer quality. It further claims to be the first comprehensive survey of RetNet, organizing the architecture's three computational representations (parallel, recurrent, and chunkwise recurrent), its gated multi-scale retention, and its spread across natural language processing, computer vision, speech, natural science, and engineering. The survey's purpose is to consolidate a fast-growing literature so that researchers can see what RetNet has already achieved and where it still falls short. If the survey is right, RetNet is not an exotic variant but a mainstream architecture family with broad, demonstrated applicability, and its main open problems are adaptive decay, hardware optimization, security, and ecosystem maturity. The paper's own Limitations section concedes that certain notable works may have been omitted and that some discussed studies rely on older benchmarks, which tempers the comprehensiveness claim.

What carries the argument

The load-bearing object is the retention mechanism and its three computational representations: parallel, recurrent, and chunkwise recurrent. The mechanism is defined by a recurrent update $s_n = \gamma s_{n-1} + K_n^\top v_n$ with output $o_n = Q_n s_n$, which unrolls into a masked attention formula $\text{Retention}(X) = (QK^\top \odot D)V$ where $D_{nm} = \gamma^{n-m}$ for $n \geq m$ and zero otherwise. The decay mask $\gamma$ is set per head via $\gamma = 1 - 2^{-5-\text{arange}(0,h)}$, and positional information enters through the xPos rotational encoding. This identity is what carries the argument: it shows that recurrence and attention are two views of the same computation, so RetNet can choose parallel representation for training and recurrent representation for inference without changing the model.

What would settle it

A systematic literature search that turns up a substantial cluster of RetNet variants or application areas absent from this survey, or a modern long-context benchmark where RetNet's quality clearly falls below Transformer or state-space baselines under controlled conditions, would undercut the survey's representation of the field.

Watch

Extended reading notes

Core claim

On its own terms, the paper documents the central insight that RetNet replaces self-attention with a retention mechanism derived from a recurrent state update $s_n = \gamma s_{n-1} + K_n^\top v_n$, whose unrolled form equals a causally masked attention $QK^\top \odot D$ with an exponential decay mask $\gamma^{n-m}$. This dual form is the key discovery the survey presents: the same architecture can be computed as parallel trainable attention, as a recurrent inference loop with constant per-step memory, and as a chunkwise hybrid that combines both for long sequences. Multi-scale retention assigns a different decay factor $\gamma$ per head, allowing the model to capture dependencies at different scales simultaneously. The survey argues that this design yields Transformer-comparable quality with linear-time inference and reduced memory overhead, and catalogs the evidence that the same module transfers across domains as diverse as language modeling, image segmentation, speech enhancement, radar perception, and drug-target interaction prediction.

Load-bearing premise

The survey's value rests on the assumption that the selected collection of cited works gives an accurate and adequately complete picture of RetNet's capabilities, an assumption the paper itself softens by conceding that notable works may have been inadvertently omitted and that some studies use older benchmarks.

Editorial extensions

If this is right

  • RetNet can train like a Transformer (fully parallel) and infer like an RNN with $O(1)$ per-step memory, so long-context applications such as document-level language modeling and streaming speech processing become cheaper to deploy.
  • The same retention module transfers across modalities, with the survey showing applications in NLP, computer vision, video, 3D point clouds, audio, chemistry, physics, biology, and traffic engineering.
  • The decay-mask formulation gives a natural way to encode spatial or temporal locality, as evidenced by vision variants that extend the 1D decay to a Manhattan-distance-based 2D decay for images and point clouds.
  • The main open problems identified are adapting the fixed decay to dynamic task needs, building dedicated hardware accelerators, addressing security and bias risks, and developing standardized benchmarks for fair comparison.
  • The paper's Limitations section states that some studied works rely on earlier benchmarks or smaller-scale evaluations, which may not reflect RetNet's performance in large-scale real-world settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If RetNet's dual recurrent-attention form proves as general as the survey suggests, the Transformer-versus-RNN distinction becomes a design choice rather than a fundamental trade-off, and any future architecture that couples a masked attention term with a recurrent state update belongs to the same family.
  • A testable extension: head-to-head comparisons of RetNet with selective state-space models such as Mamba and with linear-attention transformers on modern long-context benchmarks, controlling for training budget and hardware, would sharpen the survey's claim of Transformer-quality parity.
  • The breadth of applications collected here suggests a meta-observation: the decay-mask building block may become as standard in sequence modeling as self-attention is now, and surveys like this one may need frequent updates to track variants as they proliferate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper is a survey of Retentive Network (RetNet), a Transformer alternative based on a retention mechanism that unifies recurrent and attention-based sequence modeling. The paper reviews RNN and Transformer background, reproduces the three computational forms of retention (parallel, recurrent, and chunkwise recurrent), describes the multi-scale retention block, catalogues applications under NLP, computer vision, natural science, social engineering, audio, and other headings, and closes with challenges and future directions. The authors claim to offer the first detailed and comprehensive survey of RetNet, filling a gap in the survey literature.

Significance. If the coverage and technical exposition are reliable, the survey will be a useful entry point for researchers approaching RetNet, because it consolidates a large and heterogeneous set of application papers into one place and organizes them by domain. The bibliography is broad, roughly eighty items, and the structuring by application area is helpful. The paper introduces no new derivations or experiments; its contribution is organizational and expository. The authors are to be credited for including an explicit Limitations section, but as detailed below, that section undercuts the Abstract's 'comprehensive review' claim, and the technical core of Section 3 contains notation errors that need correction. I therefore regard the central claim as defensible but not yet fully supported.

major comments (3)
  1. [Abstract, Introduction, and Limitations] The paper claims to be 'the first detailed survey' and a 'comprehensive review' of RetNet, but its own Limitations section concedes that 'certain notable works may have been inadvertently omitted' and that several discussed studies 'rely on earlier benchmarks or smaller-scale evaluations.' No literature-search protocol, such as databases, query terms, inclusion and exclusion criteria, or screening statistics, is reported anywhere. Because the survey's value and the 'first detailed survey' claim rest on representative coverage, the omission rate must be bounded or the claims must be softened. I request either a methodology paragraph describing the search and screening process, or a revised Abstract and Conclusion that drop 'comprehensive' and state explicitly that the coverage is an entry-point overview rather than an exhaustive review.
  2. [Section 3, Eq. (12)] The chunkwise retention formula cannot be reproduced as written. The symbols i and j in 'ξij = γ^{i+1}, ζij = γ^{B−i−1}' are never defined, and i is already used as the chunk index in R_i and Retention(X[i]); the decay factors appear to be functions of the row index inside the chunk, not of the chunk index. In addition, D in Eq. (10) is defined as an |j|×|j| causal mask for the full sequence, but in Eq. (12) it is applied to a chunk of length B. The authors should define D ∈ R^{B×B} for the chunk and specify ζ and ξ with distinct indices for the chunk and the intra-chunk position, or explicitly refer to the chunkwise equations of Sun et al. (2023).
  3. [Section 3, Eqs. (7)–(10)] The architecture exposition contains multiple errors that impede verification. The sequence length is written as |j| without defining j, and later the input is given as {x_i}_{i=1}^{|j|}. In Eq. (8), the phrase 'according to Euler's formula e^{iθ} = [cosθ1, sinθ2, ..., cosθ_{d−1}, sinθ_d]' is not a correct statement of Euler's formula, and the bracket expression mixes element and index notation. In Eq. (10), Θ is called a position-dependent modulation term and 'denotes its complex conjugate,' but the complex conjugate is never explicitly defined. Since Section 3 is the survey's account of the central innovation, these notational problems should be fixed before publication.
minor comments (5)
  1. [Section 4.1] The sentence 'He et al. (2024) introduced DenseRetNet' is unsupported by the bibliography; the listed He et al. (2024) entry is 'Dense-Mamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models,' which is about Mamba, not RetNet. Please check the intended reference and correct either the citation or the description.
  2. [Section 4.4] The heading 'Social Engineering' is misleading for the content of the section, which covers building change detection, fire detection, earthquake early warning, traffic forecasting, and similar civil and urban applications. A title such as 'Societal Applications' or 'Urban and Civil Engineering' would be clearer.
  3. [Figure 1 and References] There are typographical and formatting inconsistencies: 'GRetNe' should be 'GRetNet'; the citation 'RangeRet (MOSCO, 2023)' and the reference entry 'SIMONE MOSCO. 2023' should be formatted as an author citation such as 'Mosco, S. (2023)'.
  4. [Section 5] The challenges in Section 5, including adversarial robustness and bias, are stated without supporting citations or evidence. Since the rest of the paper consistently cites sources for claims, please add references for these claims or explicitly mark them as speculative.
  5. [References] The reference list uses inconsistent citation keys, for example 'Afzal et al.' appears without a year in Section 4.1 and Figure 1, and the corresponding reference list entry lacks a year. Please harmonize all citations and years.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: a survey that reports external architecture and applications without fitting parameters or self-derived predictions.

full rationale

This is a survey paper, not a derivation. The architecture description in Section 3 is presented as a report of Sun et al. (2023), and the application sections summarize results from external papers; the survey performs no fitting, no parameter estimation, and no prediction from its own construction that could be circular. The only self-references (Li et al. 2024, a prior RWKV survey; Li et al. 2025, SegRet) are background and application citations, not premises that support the survey's central claims. The Limitations section candidly concedes possible omissions and reliance on older benchmarks; this concerns coverage and evidence quality, not circularity. No equation in the paper defines a target in terms of itself, and no fitted quantity is renamed as a prediction. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No new entities, parameters, or mechanisms are introduced; the survey only describes the existing RetNet architecture.

assumptions (2)
  • domain assumption The cited primary sources accurately describe RetNet and its applications.
    The survey's utility depends on trusting the claims in Sun et al. (2023) and the application papers it catalogs; it does not independently verify any experimental result.
  • domain assumption The set of included papers is representative of the field.
    The paper claims to be a comprehensive or first survey, but provides no systematic search protocol; the Limitations section admits that some notable works may be omitted.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Survey of Retentive Network." pith.science (2026). https://pith.science/paper/UO3XAJQQ

@misc{pith2026250606708,
  author       = {Pith},
  title        = {Pith review of: A Survey of Retentive Network},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UO3XAJQQ}},
  note         = {Machine review of arXiv:2506.06708}
}
read the original abstract

Retentive Network (RetNet) represents a significant advancement in neural network architecture, offering an efficient alternative to the Transformer. While Transformers rely on self-attention to model dependencies, they suffer from high memory costs and limited scalability when handling long sequences due to their quadratic complexity. To mitigate these limitations, RetNet introduces a retention mechanism that unifies the inductive bias of recurrence with the global dependency modeling of attention. This mechanism enables linear-time inference, facilitates efficient modeling of extended contexts, and remains compatible with fully parallelizable training pipelines. RetNet has garnered significant research interest due to its consistently demonstrated cross-domain effectiveness, achieving robust performance across machine learning paradigms including natural language processing, speech recognition, and time-series analysis. However, a comprehensive review of RetNet is still missing from the current literature. This paper aims to fill that gap by offering the first detailed survey of the RetNet architecture, its key innovations, and its diverse applications. We also explore the main challenges associated with RetNet and propose future research directions to support its continued advancement in both academic research and practical deployment.

Figures

Figures reproduced from arXiv: 2506.06708 by the authors.

Figure 1
Figure 1. Structure of this paper. ing. By employing the retention mechanism, the de￾cay mask makes RetNet very versatile for a wide range of applications, from NLP (Cheng et al., 2024), CV (Fan et al., 2024), natural science (Luo et al., 2025) to social engineering (Yan et al., 2025). With the rapid expansion of research and applica￾tions of RetNet, this survey aims to shed light on current progress in this field. As depicte… view at source ↗
Figure 2
Figure 2. Dual form of RetNet. “GN” denotes GroupNorm. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overall architecture of RetNet. Retention mechanism with a dual form of re￾cursion and parallelism is the key to the success of RetNet. Project the input X ∈ R |j|×dmodel to vn = Xn · wv, where wv is the trainable matrix that maps inputs to value vectors. Then make the projection Q, K: Q = XWQ, K = XWK, (7) where WQ, WK ∈ R d×d are learnable matrices. Consider a sequence modeling problem, through the state sn ∈ R d×… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

118 extracted references · 54 canonical work pages

  1. [1]

    Lion: A bidirectional framework that trains like a transformer and infers like an rnn

    Arshia Afzal, Elias Abad Rocamora, Leyla Naz Candogan, Pol Puigdemont, Francesco Tonin, Yongtao Wu, Mahsa Shoaran, and Volkan Cevher. Lion: A bidirectional framework that trains like a transformer and infers like an rnn

  2. [2]

    Arshia Afzal, Elias Abad Rocamora, Leyla Naz Candogan, Pol Puigdemont, Francesco Tonin, Yongtao Wu, Mahsa Shoaran, and Volkan Cevher. 2025. Linear attention for efficient bidirectional sequence modeling. arXiv preprint arXiv:2502.16249

    arXiv 2025

  3. [3]

    Rajat Kumar Arya, Subhojit Paul, and Rajeev Srivastava. 2025. An efficient hyperspectral image classification method using retentive network. Advances in Space Research, 75(2):1701--1718

    2025

  4. [4]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450

    arXiv 2016

  5. [5]

    Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157--166

    1994

  6. [6]

    Yueyang Cang, Pingge Hu, Xiaoteng Zhang, Xingtong Wang, Yuhang Liu, and Li Shi. 2024. Retcompletion: High-speed inference image completion with retentive network. arXiv preprint arXiv:2410.04056

    work page Pith review arXiv 2024

  7. [7]

    Qian Chang, Xia Li, and Xiufeng Cheng. 2024. Graph retention networks for dynamic graphs. arXiv preprint arXiv:2411.11259

    work page Pith review arXiv 2024

  8. [8]

    Guanxu Chen, Fangfang Yang, Weiwen Peng, Yuqian Fan, and Ximin Lyu. 2024. State-of-health estimation for lithium-ion batteries based on kullback--leibler divergence and a retentive network. Applied Energy, 376:124266

    2024

Show all 118 references
  1. [9]

    Yanrui Chen, Guangwu Chen, and Peng Li. 2025. Named entity recognition in track circuits based on multi-granularity fusion and multi-scale retention mechanism. Electronics, 14(5):828

    2025

  2. [10]

    Jun Cheng, Tao Meng, Xiao Ao, and Xiaohua Wu. 2024. Pre-training retnet of simulating entities and relations as sentences for knowledge graph reasoning. In 2024 4th Asia Conference on Information Engineering (ACIE), pages 6--10. IEEE

    2024

  3. [11]

    Lei Cheng and Siyang Cao. 2025. Transrad: Retentive vision transformer for enhanced radar object detection. IEEE Transactions on Radar Systems

    2025

  4. [12]

    Kyunghyun Cho, Bart Van Merri \"e nboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259

    2014 arXiv

  5. [13]

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, and 1 others. 2020. Rethinking attention with performers. arXiv preprint arXiv:2009.14794

    2020 arXiv

  6. [14]

    Jinghui Chu, Wangtao Liu, Qi Tian, and Wei Lu. 2024. Pfprnet: A phase-wise feature pyramid with retention network for polyp segmentation. IEEE Journal of Biomedical and Health Informatics

    2024

  7. [15]

    Lior Cohen, Kaixin Wang, Bingyi Kang, and Shie Mannor. 2024. Improving token-based world models with parallel observation prediction. arXiv preprint arXiv:2402.05643

    2024 arXiv

  8. [16]

    Griffin: Mixing gated linear recurrences with local attention for efficient language models, 2024

    Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, and 1 others. Griffin: Mixing gated linear recurrences with local attention for efficient language models, 2024. UR...

    2024 arXiv

  9. [17]

    JE Dom \' nguez-Vidal and Alberto Sanfeliu. 2024. Force and velocity prediction in human-robot collaborative transportation tasks through video retentive networks. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9307--9313. IEEE

    2024

  10. [18]

    Shreyas Dongre and Shrushti Mehta. 2024. Retvit: Retentive vision transformers. In 2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT), pages 1--8. IEEE

    2024

  11. [19]

    Khaled ELKarazle, Valliappan Raman, Caslon Chua, and Patrick Then. 2023. Retseg: Retention-based colorectal polyps segmentation network. arXiv preprint arXiv:2310.05446

    2023 arXiv

  12. [20]

    Gopi Krishna Erabati and Helder Araujo. 2024. Retformer: Embracing point cloud transformer with retentive network. IEEE Transactions on Intelligent Vehicles

    2024

  13. [21]

    Gopi Krishna Erabati and Helder Araujo. 2025. Retseg3d: Retention-based 3d semantic segmentation for autonomous driving. Computer Vision and Image Understanding, 250:104231

    2025

  14. [22]

    Lily Erickson. 2023. https://arxiv.org/abs/2311.07184 Cross-axis transformer with 3d rotary positional embeddings . Preprint, arXiv:2311.07184

    2023 arXiv

  15. [23]

    Dengxin Fan and Songyan Liu. 2024. Monoretnet: A self-supervised model for monocular depth estimation with bidirectional half-duplex retention. In International Conference on Intelligent Computing, pages 361--372. Springer

    2024

  16. [24]

    Qihang Fan, Huaibo Huang, Mingrui Chen, Hongmin Liu, and Ran He. 2024. Rmt: Retentive networks meet vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5641--5651

    2024

  17. [25]

    Meng Feiyu. 2024. Cfpsg: Collaborative filtering poi similarity graph enhanced retentive network for next poi recommendation. In 2024 21st International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP), pages 1--4. IEEE

    2024

  18. [26]

    Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752

    2023 arXiv

  19. [27]

    Albert Gu, Karan Goel, and Christopher R \'e . 2021. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396

    2021 arXiv

  20. [28]

    Chen Guo, Xinran Li, Jiaman Ma, Yimeng Li, Yuefan Liu, Haiying Qi, Li Zhang, and Yuhan Jin. 2024. Vl-mfer: A vision-language multimodal pretrained model with multiway-fuzzy-experts bidirectional retention network. IEEE Transactions on Fuzzy Systems

    2024

  21. [29]

    Ayse Asu Guvenli and Bora Isildak. 2024. B-jet tagging with retentive networks: A novel approach and comparative study. arXiv preprint arXiv:2412.08134

    2024 arXiv

  22. [30]

    Jia Han, Zhiyong Yu, and Jian Yang. 2025. Radio frequency-retentive network for automatic modulation classification. Electronics Letters, 61(1):e70203

    2025

  23. [31]

    Zhu Han, Shuyi Xu, Lianru Gao, Zhi Li, and Bing Zhang. 2024. Gretnet: Gaussian retentive network for hyperspectral image classification. IEEE Geoscience and Remote Sensing Letters

    2024

  24. [32]

    Ali Hatamizadeh, Michael Ranzinger, and Jan Kautz. 2023 a . Vir: Vision retention networks. arXiv. org

    2023

  25. [33]

    Ali Hatamizadeh, Michael Ranzinger, Shiyi Lan, Jose M Alvarez, Sanja Fidler, and Jan Kautz. 2023 b . Vir: Towards efficient vision retention backbones. arXiv preprint arXiv:2310.19731

    2023 arXiv

  26. [34]

    Wei He, Kai Han, Yehui Tang, Chengcheng Wang, Yujie Yang, Tianyu Guo, and Yunhe Wang. 2024. Densemamba: State space models with dense hidden connection for efficient large language models. arXiv preprint arXiv:2403.00818

    2024 arXiv

  27. [35]

    Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415

    2016 arXiv

  28. [36]

    Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735--1780

    1997

  29. [37]

    Jia Cheng Hu, Roberto Cavicchioli, and Alessandro Capotondi. 2024 a . Shifted window fourier transform and retention for image captioning. arXiv preprint arXiv:2408.13963

    2024 arXiv

  30. [38]

    Jingjing Hu, Dan Guo, Kun Li, Zhan Si, Xun Yang, and Meng Wang. 2024 b . Maskable retentive network for video moment retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 1476--1485

    2024

  31. [39]

    Jianan Huang, Xuebing Liu, Qing Zhu, Yaonan Wang, Mingtao Feng, Jiaming Zhou, Zhen Zhou, Lin Chen, and Danwei Wang. 2025 a . Rampgrasp: Retentive attention-based multiscale perception grasp detection network. IEEE Transactions on Circuits and Systems for Video Technology

    2025

  32. [40]

    Jingjia Huang, Jingyan Tu, Ge Meng, Yingying Wang, Yuhang Dong, Xiaotong Tu, Xinghao Ding, and Yue Huang. 2024 a . Efficient perceiving local details via adaptive spatial-frequency information integration for multi-focus image fusion. In Proceedings of the 32nd ACM Internation...

    2024

  33. [41]

    Kai-Wei Huang and Chia-Ping Chen. 2024. Long audio file speaker diarization with feasible end-to-end models. In 2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1--6. IEEE

    2024

  34. [42]

    Qihe Huang, Zhengyang Zhou, Kuo Yang, Gengyu Lin, Zhongchao Yi, and Yang Wang. 2024 b . Leret: Language-empowered retentive network for time series forecasting. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24

    2024

  35. [43]

    Yuyao Huang, Kai Chen, Wei Tian, and Lu Xiong. 2025 b . Boost query-centric network efficiency for multi-agent motion forecasting. IEEE Robotics and Automation Letters

    2025

  36. [44]

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ois Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156--5165. PMLR

    2020

  37. [45]

    Sangwon Kim, In-su Jang, and Byoung Chul Ko. 2024. Domain-free fire detection using the spatial--temporal attention transform of the yolo backbone. Pattern Analysis and Applications, 27(2):45

    2024

  38. [46]

    Oliver Knitter. 2024. Exploration of Quantum-Inspired Deep Learning Architectures for Fundamental Applications in Scientific Computing. Ph.D. thesis

    2024

  39. [47]

    Oliver Knitter, Dan Zhao, James Stokes, Martin Ganahl, Stefan Leichenauer, and Shravan Veerapaneni. 2024. Retentive neural quantum states: Efficient ans \"a tze for ab initio quantum chemistry. Machine Learning: Science and Technology

    2024

  40. [48]

    Bin Lei, Caiwen Ding, and 1 others. 2023. Flashvideo: A framework for swift inference in text-to-video generation. arXiv preprint arXiv:2401.00869

    2023 arXiv

  41. [49]

    Sihan Li and Juhua Huang. 2025. Resgdanet: An efficient residual group attention neural network for medical image classification. Applied Sciences, 15(5):2693

    2025

  42. [50]

    Adaptive gated meta graph retention network: A model for urban traffic flow prediction

    Xing Li and Yuequan Bao. Adaptive gated meta graph retention network: A model for urban traffic flow prediction. Available at SSRN 5170149

  43. [51]

    Zhiyuan Li, Yi Chang, and Yuan Wu. 2025. Segret: An efficient design for semantic segmentation with retentive network. arXiv preprint arXiv:2502.14014

    2025 arXiv

  44. [52]

    Zhiyuan Li, Tingyu Xia, Yi Chang, and Yuan Wu. 2024. A survey of rwkv. arXiv preprint arXiv:2412.14847

    2024 arXiv

  45. [53]

    Di Liang and Xiaofei Li. 2024. Ls-eend: Long-form streaming end-to-end neural diarization with online attractor extraction. arXiv preprint arXiv:2410.06670

    2024 arXiv

  46. [54]

    Huangbin Lin, Qing Zhu, Zhen Zhou, Yaonan Wang, Yongjie Sui, Yijiang Li, Tianming Li, and Zichen Chen. 2024. Single-stage uav geolocation enhancement with masa and self-homologous loss. In 2024 China Automation Congress (CAC), pages 5439--5444. IEEE

    2024

  47. [55]

    Ruixing Lin and Shunmei Piao. 2024. Change detection of building remote sensing images based on rmt-bit. In 2024 4th International Conference on Electronic Information Engineering and Computer Science (EIECS), pages 428--432. IEEE

    2024

  48. [56]

    Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. 2022. A survey of transformers. AI open, 3:111--132

    2022

  49. [57]

    Zhicheng Lin, Rongpu Cui, Limiao Ning, and Jian Peng. 2025 a . Temporal features-fused vision retentive network for echocardiography image segmentation. Sensors, 25(6):1909

    2025

  50. [58]

    Zhijie Lin, Zilong Zhu, Lingling Guo, Jingjing Chen, and Jiyi Wu. 2025 b . Disease detection algorithm for tea health protection based on improved real-time detection transformer. Applied Sciences (2076-3417), 15(4)

    2025

  51. [59]

    Zhou Linhao, Zhong Shenghua, and Xiao Zhijiao. 2024. Discovering multi-relational integration for knowledge tracing with retentive networks. In Proceedings of the 2024 International Conference on Multimedia Retrieval, pages 960--968

    2024

  52. [60]

    Jiaji Liu, ZhiTao Liu, YePeng Wang, and Fang Li. 2024 a . Vvnet: Underwater object detection network based on vision transformer and vision retnet for underwater robot picking. In 2024 International Symposium on Digital Home (ISDH), pages 19--24. IEEE

    2024

  53. [61]

    Jiayuan Liu, Bo Zhou, Xue Wan, Yan Pan, Zicong Li, and Yuanbin Shao. 2025. Mar-vo: A match-and-refine framework for uav’s monocular visual odometry in planetary environments. IEEE Transactions on Geoscience and Remote Sensing

    2025

  54. [62]

    Jing Liu, Donglin Jing, Yanyan Cao, Ying Wang, Chaoping Guo, Peijun Shi, and Haijing Zhang. 2024 b . Lightweight progressive fusion calibration network for rotated object detection in remote sensing images. Electronics, 13(16):3172

    2024

  55. [63]

    Zhe Liu, Jinghua Hou, Xinyu Wang, Xiaoqing Ye, Jingdong Wang, Hengshuang Zhao, and Xiang Bai. 2024 c . Lion: Linear group rnn for 3d object detection in point clouds. Advances in Neural Information Processing Systems, 37:13601--13626

    2024

  56. [64]

    Ziyu Liu, Yi Shen, Yunliang Jiang, Hancan Zhu, Hailong Hu, Yanlei Kang, Ming Chen, and Zhong Li. 2024 d . Variation and evolution analysis of sars-cov-2 using self-game sequence optimization. Frontiers in Microbiology, 15:1485748

    2024

  57. [65]

    Baichao Long, Wang Zhu, and Jianli Xiao. 2024. St-retnet: A long-term spatial-temporal traffic flow prediction method. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 3--16. Springer

    2024

  58. [66]

    Junwei Luo, Jiaojiao Wang, Jingjing Wei, Chaokun Yan, and Huimin Luo. 2025. Deephapnet: a haplotype assembly method based on retnet and deep spectral clustering. Briefings in Bioinformatics, 26(1):bbae656

    2025

  59. [67]

    Omayma Mahjoub, Sasha Abramowitz, Ruan de Kock, Wiem Khlifi, Simon du Toit, Jemma Daniel, Louay Ben Nessir, Louise Beyers, Claude Formanek, Liam Clark, and Arnu Pretorius. 2025. https://arxiv.org/abs/2410.01706 Sable: a performant, efficient and scalable sequence model for mar...

    2025 arXiv

  60. [68]

    Runyu Miao, Danlin Liu, Liyun Mao, Xingyu Chen, Leihao Zhang, Zhen Yuan, Shanshan Shi, Honglin Li, and Shiliang Li. 2024. Gr-p k a: a message-passing neural network with retention mechanism for p k a prediction. Briefings in Bioinformatics, 25(5):bbae408

    2024

  61. [69]

    Zhaoyi Min, Qianqian Xiao, Muhammad Abbas, and Duanjin Zhang. 2025. Retentive network-based time series anomaly detection in cyber-physical systems. Engineering Applications of Artificial Intelligence, 145:110215

    2025

  62. [70]

    SIMONE MOSCO. 2023. Exploiting retentive networks in 3d lidar semantic segmentation

    2023

  63. [71]

    Cheng Nian, Weiyi Zhang, Fasih Ud Din Farrukh, Liting Niu, Dapeng Jiang, Fei Chen, and Chun Zhang. 2024. A 77.79 gops/w retentive network fpga inference accelerator with optimized workload. In IECON 2024-50th Annual Conference of the IEEE Industrial Electronics Society, pages ...

    2024

  64. [72]

    Masashi Okada, Mayumi Komatsu, and Tadahiro Taniguchi. 2024. A contact model based on denoising diffusion to learn variable impedance control for contact-rich manipulation. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7286--7293. IEEE

    2024

  65. [73]

    Sidike Paheding, Nusrat Zahan, Abel A Reyes, Ernesto Martinez Jr, and Eung-Joo Lee. 2024. Hyperspectral image classification with retentive network. In Pattern Recognition and Tracking XXXV, volume 13040, pages 15--21. SPIE

    2024

  66. [74]

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, and 1 others. 2023. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048

    2023 arXiv

  67. [75]

    Lihong Peng, Xin Liu, Min Chen, Wen Liao, Jiale Mao, and Liqian Zhou. 2024. Mgndti: A drug-target interaction prediction framework based on multimodal representation learning and the gating mechanism. Journal of Chemical Information and Modeling, 64(16):6684--6698

    2024

  68. [76]

    Zhen Qin, Songlin Yang, and Yiran Zhong. 2023. Hierarchically gated recurrent neural network for sequence modeling. Advances in Neural Information Processing Systems, 36:33202--33221

    2023

  69. [77]

    Yufan Qiu, Yaping Liu, and Shuo Zhang. 2024. Rn-ete: A retentive network-based encryption traffic encoder. In Proceedings of the 2024 3rd International Conference on Cryptography, Network Security and Communication Technology, pages 214--219

    2024

  70. [78]

    Prajit Ramachandran, Barret Zoph, and Quoc V Le. 2017. Swish: a self-gated activation function. arXiv preprint arXiv:1710.05941, 7(1):5

    2017 arXiv

  71. [79]

    Hojjat Salehinejad, Sharan Sankar, Joseph Barfett, Errol Colak, and Shahrokh Valaee. 2017. Recent advances in recurrent neural networks. arXiv preprint arXiv:1801.01078

    2017 arXiv

  72. [80]

    Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(11):2673--2681

    1997

  73. [81]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053

    2019 arXiv

  74. [82]

    Ruilin Su, Wanshan Zhang, and Dunhui Yu. 2024. Sequence-to-sequence multi-hop knowledge reasoning based on retentive network. In 2024 7th International Conference on Computer Information Science and Application Technology (CISAT), pages 360--366. IEEE

    2024

  75. [83]

    Yongjie Sui, Yan Zheng, Qing Zhu, Zhen Zhou, Lin Chen, Jianqiao Luo, Yaonan Wang, and Zihao Yang. 2024. An efficient transformer incorporating a spatial decay matrix and attention decomposition for image matching. In 2024 China Automation Congress (CAC), pages 5216--5221. IEEE

    2024

  76. [84]

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. 2023. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621

    2023 arXiv

  77. [85]

    Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. 2022. A length-extrapolatable transformer. arXiv preprint arXiv:2212.10554

    2022 arXiv

  78. [86]

    Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. 2024. You only cache once: Decoder-decoder architectures for language models. Advances in Neural Information Processing Systems, 37:7339--7361

    2024

  79. [87]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  80. [88]

    Baoquan Wang, Yan Zeng, and Dongming Feng. 2025 a . Deep learning-based damage assessment of hinge joints for multi-girder bridges utilizing vehicle-induced bridge responses. Engineering Structures, 333:120148

    2025

  81. [89]

    Bin Wang, Fei Deng, and Peifan Jiang. 2024 a . Eegdir: Electroencephalogram denoising network for temporal information storage and global modeling through retentive network. Computers in Biology and Medicine, 177:108626

    2024

  82. [90]

    Junliang Wang, Wenlong Hang, Shuang Liang, Qiong Wang, Badong Chen, and Jing Qin. 2025 b . Convolutional retentive network for eeg decoding. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1--5. IEEE

    2025

  83. [91]

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768

    2020 arXiv

  84. [92]

    Siyu Wang, Xiaocong Chen, and Lina Yao. 2024 b . Retentive decision transformer with adaptive masking for reinforcement learning based recommendation systems. arXiv preprint arXiv:2403.17634

    2024 arXiv

  85. [93]

    Zeyu Wang, Libo Zhao, Jizheng Zhang, Rui Song, Haiyu Song, Jiana Meng, and Shidong Wang. 2025 c . Multi-text guidance is important: Multi-modality image fusion via large generative vision-language model. International Journal of Computer Vision, pages 1--23

    2025

  86. [94]

    Tengqing Wu. 2024. A diffusion data enhancement retentive model for sequential recommendation. In 2024 7th International Conference on Computer Information Science and Application Technology (CISAT), pages 114--118. IEEE

    2024

  87. [95]

    Yuxin Wu and Kaiming He. 2018. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3--19

    2018

  88. [96]

    Yunyang Xie, Kai Chen, Shenghui Li, Bingqian Li, and Ning Zhang. 2024 a . Uarc: Unsupervised anomalous traffic detection with improved u-shaped autoencoder and retnet based multi-clustering. In International Conference on Information and Communications Security, pages 187--207...

    2024

  89. [97]

    Zexun Xie, Min Xu, Shudong Zhang, and Lijuan Zhou. 2024 b . Rcat: Retentive clip adapter tuning for improved video recognition. Electronics, 13(5):965

    2024

  90. [98]

    Keyu Xu, Chengtian Song, Yue Xie, Lizhi Pan, Xiaozheng Gan, and Gao Huang. 2024. Rmt-yolov9s: An infrared small target detection method based on uav remote sensing images. IEEE Geoscience and Remote Sensing Letters

    2024

  91. [99]

    Zhangyong Xu, Ning Chen, Guangqiang Li, Jing Li, Hongqing Zhu, and Zhiying Zhu. 2025. The mitigation of heterogeneity in temporal scale among different cortical regions for eeg emotion recognition. Knowledge-Based Systems, 309:112826

    2025

  92. [100]

    Yimo Yan, Songyi Cui, Jiahui Liu, Yaping Zhao, Bodong Zhou, and Yong-Hong Kuo. 2025. Multimodal fusion for large-scale traffic prediction with heterogeneous retentive networks. Information Fusion, 114:102695

    2025

  93. [101]

    Wenkui Yang, Zhida Zhang, Xiaoqiang Zhou, Junxian Duan, and Jie Cao. 2024 a . Tt-df: A large-scale diffusion-based dataset and benchmark for human body forgery detection. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 429--443. Springer

    2024

  94. [102]

    Xuan Yang, Yunxuan Dong, Lina Yang, and Thomas Wu. 2024 b . Short-term photovoltaic forecasting model for qualifying uncertainty during hazy weather. arXiv preprint arXiv:2407.19663

    2024 arXiv

  95. [103]

    Xuan Yang, Tao Peng, Haijia Bi, and Jiayu Han. 2024 c . Span-level bidirectional retention scheme for aspect sentiment triplet extraction. Information Processing & Management, 61(5):103823

    2024

  96. [104]

    Yong Yu, Xiaosheng Si, Changhua Hu, and Jianxun Zhang. 2019. A review of recurrent neural networks: Lstm cells and network architectures. Neural computation, 31(7):1235--1270

    2019

  97. [105]

    Yuansong Zeng, Jiancong Xie, Zhuoyi Wei, Yun Su, Ningyuan Shangguan, Shuangyu Yang, Chengyang Zhang, Wenbing Li, Jinbo Zhang, Nan Fang, and 1 others. 2024. Cellfm: a large-scale foundation model pre-trained on transcriptomics of 100 million human cells. bioRxiv, pages 2024--06

    2024

  98. [106]

    Fanlong Zhang, Huanming Chen, Quan Chen, and Jianqi Liu. 2025 a . Cloud software code generation via knowledge graphs and multi-modal learning

    2025

  99. [107]

    Lingzhe Zhang, Zewen Xiao, and Huaiyuan Wang. 2024 a . Transient stability assessment of power system based on time-adapative retnet. In 2024 3rd Asia Power and Electrical Technology Conference (APET), pages 391--396. IEEE

    2024

  100. [108]

    Meng Zhang, Wenzhong Yang, Liejun Wang, Zhonghua Wu, and Danny Chen. 2025 b . Hfa-net: hierarchical feature aggregation network for micro-expression recognition. Complex & Intelligent Systems, 11(3):1--20

    2025

  101. [109]

    Tianning Zhang, Feng Liu, Yuming Yuan, Rui Su, Wanli Ouyang, and Lei Bai. 2024 b . Fast information streaming handler (fish): A unified seismic neural network for single station real-time earthquake early warning. arXiv preprint arXiv:2408.06629

    2024 arXiv

  102. [110]

    Yuxuan Zhang, Zipeng Zhang, Weiwei Guo, Wei Chen, Zhaohai Liu, and Houguang Liu. 2025 c . Lretunet: A u-net-based retentive network for single-channel speech enhancement. Computer Speech & Language, page 101798

    2025

  103. [111]

    Zhikang Zhang, Zhongjie Zhu, Yongqiang Bai, Ming Wang, and Zhijing Yu. 2024 c . Octree-retention fusion: A high-performance context model for point cloud geometry compression. In Proceedings of the 2024 International Conference on Multimedia Retrieval, pages 1150--1154

    2024

  104. [112]

    Zipeng Zhang, Zhencai Zhu, Bin Meng, Zheng Yang, Mingke Wu, Xinyu Cheng, Binhong Li, and Houguang Liu. 2025 d . Intelligent coal gangue identification: A novel amplitude frequency sensitive neural network. Expert Systems with Applications, 274:126880

    2025

  105. [113]

    Kaili Zheng, Feixiang Lu, Yihao Lv, Liangjun Zhang, Chenyi Guo, and Ji Wu. 2024. 3d human pose estimation via non-causal retentive networks. In European Conference on Computer Vision, pages 111--128. Springer

    2024

  106. [114]

    Li Zhou, Dayang Wang, Yongshun Xu, Shuo Han, Bahareh Morovati, Shuyi Fan, and Hengyong Yu. 2024. Gradient guided co-retention feature pyramid network for ldct image denoising. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 153-...

    2024

  107. [115]

    Wang Zhu, Baichao Long, and Jianli Xiao. 2024. Spatial-temporal retentive heterogeneous graph convolutional network for traffic flow prediction. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1--8. IEEE

    2024

  108. [116]

    Zijie Zhu, Feng Ding, Chenglong Chu, and Fangming Zhong. 2025. Retention enhanced cross-modal attention for multi-hop vqa. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1--5. IEEE

    2025

  109. [117]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  110. [118]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.