Pith. sign in

REVIEW 3 major objections 5 minor 39 references

AudioRWKV: Efficient and Stable Bidirectional RWKV for Audio Pattern Recognition

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A-RWKV-B/16 trained from scratch outperforms AST-B/16 and AuM-B/16 on all reported audio benchmarks, with 40.91 AudioSet mAP.

desk verdict A-RWKV is a genuinely new linear-time audio backbone, but the headline wins over AuM are built on imported baselines and need a matched re-run. read the letter →

arxiv 2509.02167 v2 pith:EPRD2454 submitted 2025-09-02 cs.SD

Jing Wang , Maoxiang Wu , Jiayu Xiong , Jianlong Kwan , Jun Xue This is my paper
classification cs.SD
keywords audiopatternrecognitionRWKVlinear-timesequencemodelingbidirectionalWKVstate-spacemodelsMelspectrogramclassificationSet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper seeks to establish that a recurrent architecture with linear, not quadratic, sequence cost can be the best default for audio classification. It adapts RWKV7 to Mel spectrograms by replacing the 1D token shift with a 2D depthwise separable convolution and by running the WKV recurrence both forward and backward over the audio. On five public benchmarks, the base model trained from scratch reports higher scores than both the Audio Spectrogram Transformer and Audio Mamba, including 40.91 mAP on AudioSet-2M versus 35.23 for AST and 32.43 for AuM. The same recipe scales from 6M to 91M parameters without the instability the paper attributes to Mamba-style models, and the RWKV7 kernel sustains near-flat throughput on audio up to about five and a half minutes. If these numbers hold, long-form audio understanding gets a backbone that is both faster and safer to scale than the two dominant sequence operator families.

What carries the argument

The load-bearing object is the bidirectional WKV state recurrence inherited from RWKV7: a linear-cost update that carries global context in a learned-decay state. A-RWKV wraps it in two audio-specific modifications. The first, a 2D depthwise separable convolution replacing the 1D token shift, forms a local residual by subtracting the input from its convolved version and channel-wise interpolates that residual into the recurrent parameters, so local time-frequency structure informs every update. The second, Bi-WKV, runs the same recurrence on the original and time-reversed token sequences and fuses the two context streams with a dynamic gate derived from the same local residual, giving each t

What would settle it

Run the complete A-RWKV-B model end-to-end on a 5m28s spectrogram with the Bi-WKV forward/backward scan and the DWConv2D enabled, on the same GPU used in the efficiency plot, and compare wall-clock latency and peak memory against FlashAttention-based AST. If the full-model speedup is far below 13.3x, or if training a much larger A-RWKV with the same recipe diverges, the efficiency and stability claims would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that A-RWKV-B/16, trained from scratch, surpasses both a quadratic-attention transformer (AST) and a linear-time bidirectional state-space model (AuM) on every benchmark the paper reports, and does so with stable scaling. The mechanism credited is the combination of RWKV7's expressive recurrent state update, a 2D depthwise separable convolution that injects local spectro-temporal structure into the recurrent parameters, and a bidirectional scan that lets every position attend to the full sequence at linear cost. The paper also claims the operator remains stable from 6M to 91M parameters under one training recipe, and that its WKV7 kernel maintains roughly constant token

Load-bearing premise

The 13.3x speedup is measured on the RWKV7 kernel in isolation, and the paper assumes the bidirectional scan and 2D convolution added for audio do not erase that advantage in the complete model.

Editorial extensions

If this is right

  • From-scratch training, not just fine-tuning, yields the best audio classification numbers, so linear-time recurrent models can serve as primary backbones rather than only efficient fallbacks.
  • Audio segments of several minutes become processable on a single 24 GB GPU, where quadratic attention hits memory limits around 20 seconds.
  • A single fixed training recipe works from 6M to 91M parameters, lowering the cost of scaling experiments for audio models.
  • Pre-training on 10-second AudioSet transfers to 1-second Speech Commands better for A-RWKV than for AuM, suggesting the representations handle shifts in sequence length.
  • The Bi-WKV and 2D ConvShift are generic mixer components that other spectrogram-based pipelines could adopt independently.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the reported 13.3x speedup is measured on the causal RWKV7 kernel, so a fair end-to-end replication should measure full A-RWKV inference including the backward scan and the depthwise convolution; the gain may be smaller, though still likely favorable at minute-long contexts.
  • Inference: because the 2D ConvShift already weakens strict 1D ordering, windowed or zigzag scan strategies used in vision Mamba models may be redundant on spectrograms; a controlled scan-order ablation would test this.
  • Inference: the same stable bidirectional recurrent operator should transfer to dense-output audio tasks such as sound event detection or source separation, where quadratic attention is prohibitive at high temporal resolution.
  • Inference: RWKV-style explicit decay may conflict with masked audio pretraining, as the paper acknowledges; a variant that bypasses or learns to disable decay during pretraining could combine linear-time bidirectionality with masked-spectrogram objectives.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes AudioRWKV (A-RWKV), adapting the RWKV7 architecture for audio classification by replacing the 1D token shift with a 2D depthwise separable convolution and making the WKV recurrence bidirectional via a forward/backward scan with a learned fusion gate. The authors claim that A-RWKV-B/16 trained from scratch outperforms both the Audio Spectrogram Transformer (AST) and Audio Mamba (AuM) on all reported benchmarks, that it scales stably from 6M to 91M parameters, and that the underlying RWKV7 operator is substantially more efficient than attention for long audio sequences. The paper includes experiments on AudioSet-2M/20K, VGGSound, NSynth, Speech Commands v2, and ESC-50, plus an ablation study and an efficiency comparison.

Significance. If the central claims hold, the paper would demonstrate that a linear-time recurrent architecture with bidirectional scanning can match or exceed both quadratic-attention and prior state-space baselines on audio pattern recognition, while offering a meaningful efficiency advantage for long-form audio. The manuscript has concrete strengths: code is released, the ablation decomposes the contribution of bidirectional scanning, fusion gating, and 2D convolution, and the efficiency measurements are experimental rather than purely asymptotic. However, the main accuracy and efficiency claims rely on external baseline numbers and on operator-level measurements that do not yet establish the full-model advantage claimed in the abstract and Section 4.2.

major comments (3)
  1. [Section 4.1, Table 1] The central claim that A-RWKV-B/16 'outperforms both attention-based and state-space baselines on every benchmark' rests on imported AuM numbers. Table 1's caption states 'All other results are from AuM [9]', yet Section 4.1 says 'Evaluations use the same training recipe.' These statements are mutually inconsistent. Since AuM was not re-run under the 25-epoch, AdamW, batch-1024, bf16 recipe, the 8.48 mAP gap on AudioSet-2M may not be a controlled comparison. Please re-run AuM (and, where possible, AST without the bf16 caveat) under the identical recipe and report per-seed results, or explicitly soften the claim to state which comparisons are cross-paper.
  2. [Section 4.3, Fig. 2] The headline efficiency result ('13.3x faster', Fig. 2) is measured on the causal RWKV7 kernel, not on the full A-RWKV model. The full model replaces the causal kernel with two Bi-WKV scans and adds DWConv2D operations, which may materially change the latency/token-throughput profile. The abstract also attributes the speedup to 'WKV7' rather than to the full A-RWKV architecture. Please report end-to-end inference time for A-RWKV versus AST/AuM on the same benchmark conditions, or at least measure the overhead of the bidirectional scan and convolution so the reader can assess how much of the kernel-level speedup survives in the actual model.
  3. [Section 4.2, Table 3] The scaling and stability claim is not directly supported by a controlled comparison. Table 3 shows A-RWKV-T/S/B improving across sizes, but it does not train AuM at comparable sizes or report gradient/update statistics to substantiate the claim that A-RWKV 'overcomes the training instability issues that limit competing linear-time models.' The instability of prior SSMs is cited from prior work, but the paper's own contribution claims to solve it. A direct stability comparison (e.g., loss/gradient norm curves, or a large-AuM run under the same recipe) would be needed to make this claim load-bearing.
minor comments (5)
  1. [Abstract and Section 4.3] The notation 'O(L2)' should read O(L^2). Also, 'A-RWKV's operator (RWKV7)' in Section 4.3 contradicts the abstract's 'WKV7' phrasing; use one consistent term for the full model versus the kernel.
  2. [Section 4.2, Scaling Analysis] The sentence 'As summarized in Tab. 2' appears to refer to Table 3, since Table 2 contains fine-tuning results and Table 3 contains the scaling results. Please correct the cross-reference.
  3. [Section 4.1, Datasets] The VGGSound entry ends incomplete: 'VGGSound (VGG) [24], 10s audio-' — finish the sentence and specify whether the full or a subset is used.
  4. [Table 1] The provenance of each row is unclear. The footnote says '*' is the authors' implementation and 'All other results are from AuM [9]', but AST-B/16 without '*' is listed above AuM. Clarify per-row/cell which numbers are re-run, which are imported, and the exact training setup for each.
  5. [Conclusion] The conclusion says A-RWKV 'performs as well as standard Transformers and Mambas', but the paper's own results claim it outperforms both. Rephrase to be consistent with the empirical section.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all central claims rest on measured benchmarks and ablations, not on fitted inputs or load-bearing self-citations.

full rationale

The paper's derivation chain is architectural: it adapts RWKV7's recurrent formulation, replaces 1D token shift with a 2D depthwise separable convolution, and introduces a bidirectional WKV scan. These are design choices, not circular derivations. Every central claim is supported by direct empirical evaluation: Table 1 reports measured mAP/accuracy; Table 2 reports measured fine-tuning results; Table 4 is an ablation comparing measured variants; Fig. 2 is a measured runtime/throughput comparison. No parameter is fitted to a subset of data and then renamed as a prediction, and no equation defines the target metric in terms of the model's own inputs. The paper contains no load-bearing self-citations: references to RWKV7, AuM, V-RWKV, and Vision Mamba are external prior works, and the accuracy/scaling claims do not reduce to those citations. The efficiency measurement is performed on the RWKV7 operator rather than the full A-RWKV model, and the AuM baseline numbers are imported from the AuM paper, but these are fairness/external-validity concerns, not circularity. The conclusion's caveats about inductive biases, decay, and masked pretraining are stated limitations, not self-referential justification. Therefore no circularity pattern is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted physical constants or invented entities. Its claims rest on standard deep-learning assumptions (RWKV7 kernel correctness, comparability of external baselines, representativeness of operator-level efficiency measurements), which are listed above.

assumptions (4)
  • domain assumption The RWKV7 recurrence (Eq. 4) and its stability properties are as described in [12] and transfer to the modified 2D bidirectional variant.
    Invoked in Section 3 and implicitly in the scaling analysis; the paper does not re-derive or verify stability, and does not provide a head-to-head stability control against AuM.
  • ad hoc to paper Baseline numbers from AuM [9] and AST [3] are directly comparable to the paper's A-RWKV runs despite originating from different papers and training setups.
    Table 1 note says 'All other results are from AuM [9]', while Section 4.1 claims 'the same training recipe'; these statements conflict.
  • ad hoc to paper The efficiency measurements in Fig. 2, obtained on the causal RWKV7 kernel, are representative of the full A-RWKV model with Bi-WKV and DWConv2D.
    Fig. 2 uses 'A-RWKV' labels but the abstract and caption describe the WKV7 operator; the bidirectional scan doubles recurrence and the 2D convolution adds compute, neither of which is benchmarked.
  • domain assumption Flattening a 2D Mel spectrogram into a 1D patch sequence with absolute positional embeddings preserves the information needed for classification.
    Standard practice in AST and AuM, but the paper does not analyze the effect of this 2D-to-1D reduction on the reported results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AudioRWKV: Efficient and Stable Bidirectional RWKV for Audio Pattern Recognition." pith.science (2026). https://pith.science/paper/EPRD2454

@misc{pith2026250902167,
  author       = {Pith},
  title        = {Pith review of: AudioRWKV: Efficient and Stable Bidirectional RWKV for Audio Pattern Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EPRD2454}},
  note         = {Machine review of arXiv:2509.02167}
}
read the original abstract

Recently, Transformers (e.g., Audio Spectrogram Transformers, AST) and state-space models (e.g., Audio Mamba, AuM) have achieved remarkable progress in audio modeling. However, the O(L^2) computational complexity of the Transformer architecture hinders efficient long-sequence processing, while the Mamba architecture tends to become unstable when scaling parameters and data. To address these challenges, this paper proposes AudioRWKV (A-RWKV), a highly efficient and stable architecture for audio modeling. Specifically, we inherit the stable and efficient recurrent formulation of RWKV7 and replace its 1D token-shift operation with a 2D depthwise separable convolution to better capture local spectro-temporal patterns. Furthermore, we adapt the original causal WKV kernel into a bidirectional WKV kernel (Bi-WKV), enabling global context modeling over the entire audio sequence while maintaining linear computational complexity. Benefiting from the inherent stability of the RWKV7 foundation, A-RWKV scales seamlessly to larger model sizes. Experimental results demonstrate that, under the same linear-model regime, A-RWKV-S (22M) achieves performance parity with AuM-B (92M) while exhibiting more stable throughput than AST; for long-form audio (~5 minutes 28 seconds), WKV7 achieves up to a 13.3X speedup in processing.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 28 canonical work pages

  1. [9]

    CONCLUSION Audio-RWKV performs as well as standard Transformers and Mambas in audio tasks, while using much less computation and memory, demonstrating the promise of linear models for audio understanding. Although inductive biases and explicit decay are beneficial in low-to-moderate data regimes, they may impair performance at sufficiently large scales [2...

    work page

  2. [1]

    INTRODUCTION Transformer-based architectures [1, 2], particularly the Audio Spectrogram Transformer (AST) [3], have established new performance benchmarks across a variety of audio under- standing tasks, owing to their powerful global information processing capabilities. However, the self-attention mecha- nism, which underpins these models, has a computat...

    work page Pith review arXiv 2025

  3. [2]

    It retains the global mod- eling strengths of AST while reducing computational complexity to a linear scale, offering an efficient solu- tion for long-form audio processing

    We propose A-RWKV , a cost-effective and scalable backbone for audio tasks. It retains the global mod- eling strengths of AST while reducing computational complexity to a linear scale, offering an efficient solu- tion for long-form audio processing

    work page

  4. [3]

    These operators, tailored for spectro- grams, achieve effective and efficient feature aggrega- tion across both local and global scopes

    We develop a bidirectional global attention mechanism (Bi-WKV) combined with a DWConv2D-based token shift method. These operators, tailored for spectro- grams, achieve effective and efficient feature aggrega- tion across both local and global scopes

    work page

  5. [4]

    We demonstrate that by building on the stable RWKV7 foundation, A-RWKV overcomes the training instabil- ity issues that limit the scalability of competing linear- time models like AuM, enabling the successful training of larger models for superior performance

    work page

  6. [5]

    RELATED WORK The field of audio processing has been significantly advanced by Transformer-based models, most notably the Audio Spec- trogram Transformer (AST) [3], which adapted the suc- cess of the Vision Transformer (ViT) [2] to spectrograms. These models excel at capturing global context through self-attention but are inherently constrained by its quad...

    work page

  7. [6]

    The foundational principle is the systematic replacement of the original 1D token shift operation with a 2D depthwise sep- arable convolution (DWConv2D)

    METHODOLOGY This work adapts the RWKV7 architecture [12] for process- ing 2D spatial data, exemplified by audio Mel spectrograms. The foundational principle is the systematic replacement of the original 1D token shift operation with a 2D depthwise sep- arable convolution (DWConv2D). This substitution enables R W K V AG Conv Shift Preprocess R W K V AG Aud...

    work page

  8. [7]

    EXPERIMENTS In this section, we aim to demonstrate the advantages of Bi- WKV as sequence modeling operator over attention and Bi- SSM. 4.1. Settings, Datasets and Baselines. Model Settings. AudioRWKV(-B) is a 12-layer model with a 768d embedding and learnable absolute positional embed- dings. All other architectural details and parameter initializa- tion ...

    work page

Show all 39 references
  1. [8]

    Rwkvtts: Yet another tts based on rwkv-7,

    Liu Xiao et al., “Rwkvtts: Yet another tts based on rwkv-7,” arXiv preprint arXiv:2504.03289, 2025

    2025 arXiv

  2. [10]

    Attention is all you need,

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention is all you need,” Ad- vances in neural information processing systems , vol. 30, 2017

    2017

  3. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint ar...

    2010 arXiv

  4. [12]

    Ast: Audio spectrogram transformer,

    Yuan Gong, Yu-An Chung, and James Glass, “Ast: Audio spectrogram transformer,” arXiv preprint arXiv:2104.01778, 2021

    2021 arXiv

  5. [13]

    Retentive network: A successor to trans- former for large language models,

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei, “Retentive network: A successor to trans- former for large language models,” arXiv preprint arXiv:2307.08621, 2023

    2023 arXiv

  6. [14]

    Mamba: Linear-time sequence modeling with selective state spaces,

    Albert Gu and Tri Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  7. [15]

    Rwkv: Reinventing rnns for the transformer era,

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al., “Rwkv: Reinventing rnns for the transformer era,”arXiv preprint arXiv:2305.13048, 2023

    2023 arXiv

  8. [16]

    Exploring re- ceptance weighted key value model for single-channel speech enhancement,

    Yuanle Li, Yi Zhou, and Hongqing Liu, “Exploring re- ceptance weighted key value model for single-channel speech enhancement,” in 2024 7th International Con- ference on Information Communication and Signal Pro- cessing (ICICSP). IEEE, 2024, pp. 123–127

    2024

  9. [17]

    Random erasing data augmentation,

    Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang, “Random erasing data augmentation,” in Proceedings of the AAAI conference on artificial intelli- gence, 2020, vol. 34, pp. 13001–13008

    2020

  10. [18]

    Audio mamba: Bidirectional state space model for audio representation learning,

    Mehmet Hamza Erol, Arda Senocak, Jiu Feng, and Joon Son Chung, “Audio mamba: Bidirectional state space model for audio representation learning,” IEEE Signal Processing Letters, 2024

    2024

  11. [19]

    Under- standing the differences in foundation models: Atten- tion, state space models, and recurrent neural networks,

    Jerome Sieber, Carmen A Alonso, Alexandre Didier, Melanie N Zeilinger, and Antonio Orvieto, “Under- standing the differences in foundation models: Atten- tion, state space models, and recurrent neural networks,” Advances in Neural Information Processing Systems , vol. 37, pp. ...

    2024

  12. [20]

    Vision-rwkv: Efficient and scalable visual perception with rwkv-like architectures,

    Yuchen Duan, Weiyun Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Hongsheng Li, Jifeng Dai, and Wenhai Wang, “Vision-rwkv: Efficient and scalable visual perception with rwkv-like architectures,” arXiv preprint arXiv:2403.02308, 2024

    2024 arXiv

  13. [21]

    Rwkv-7” goose

    Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Al- caide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, et al., “Rwkv-7” goose” with expressive dynamic state evolution,”arXiv preprint arXiv:2503.14456, 2025

    2025 arXiv

  14. [22]

    Vi- sion mamba: Efficient visual representation learning with bidirectional state space model,

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xin- long Wang, Wenyu Liu, and Xinggang Wang, “Vi- sion mamba: Efficient visual representation learning with bidirectional state space model,” arXiv preprint arXiv:2401.09417, 2024

    2024 arXiv

  15. [23]

    Decoupled weight decay regularization,

    Ilya Loshchilov and Frank Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations (ICLR), 2019

    2019

  16. [24]

    mixup: Beyond empirical risk min- imization,

    Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz, “mixup: Beyond empirical risk min- imization,” arXiv preprint arXiv:1710.09412, 2017

    2017 arXiv

  17. [25]

    Cutmix: Regularization strategy to train strong clas- sifiers with localizable features,

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo, “Cutmix: Regularization strategy to train strong clas- sifiers with localizable features,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 6023–6032

    2019

  18. [26]

    Mlp-mixer: An all-mlp architecture for vision,

    Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jes- sica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al., “Mlp-mixer: An all-mlp architecture for vision,” Advances in neural information processing systems, v...

    2021

  19. [27]

    Deep networks with stochas- tic depth,

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger, “Deep networks with stochas- tic depth,” in European conference on computer vision. Springer, 2016, pp. 646–661

    2016

  20. [28]

    When does label smoothing help?,

    Rafael M ¨uller, Simon Kornblith, and Geoffrey E Hinton, “When does label smoothing help?,”Advances in neural information processing systems, vol. 32, 2019

    2019

  21. [29]

    Audio set: An on- tology and human-labeled dataset for audio events,

    Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter, “Audio set: An on- tology and human-labeled dataset for audio events,” in IEEE International Conference on Acoustics, Speech and Signal Processi...

    2017

  22. [30]

    Esc: Dataset for environmental sound classification,

    Karol J. Piczak, “Esc: Dataset for environmental sound classification,” in ACM international conference on Multimedia (ACM MM), 2015

    2015

  23. [31]

    Speech commands: A dataset for limited-vocabulary speech recognition,

    Pete Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” ArXiv, vol. abs/1804.03209, 2018

    2018 arXiv

  24. [32]

    Neural audio synthesis of musical notes with wavenet autoencoders,

    Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan, “Neural audio synthesis of musical notes with wavenet autoencoders,” inInternational con- ference on machine learning (ICML). PMLR, 2017, pp. 1068–1077

    2017

  25. [33]

    Vggsound: A large-scale audio-visual dataset,

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and An- drew Zisserman, “Vggsound: A large-scale audio-visual dataset,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 721–725

    2020

  26. [34]

    Flashattention: Fast and memory- efficient exact attention with io-awareness,

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R ´e, “Flashattention: Fast and memory- efficient exact attention with io-awareness,” Advances in neural information processing systems , vol. 35, pp. 16344–16359, 2022

    2022

  27. [36]

    Scaling vision transformers,

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer, “Scaling vision transformers,” inProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 12104–12113

    2022

  28. [37]

    Masked autoencoders are scalable vision learners,

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Pi- otr Doll ´ar, and Ross Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16000–16009

    2022

  29. [38]

    Spatial-mamba: Effective vi- sual state space models via structure-aware state fusion,

    Chaodong Xiao, Minghan Li, Zhengqiang Zhang, Deyu Meng, and Lei Zhang, “Spatial-mamba: Effective vi- sual state space models via structure-aware state fusion,” arXiv preprint arXiv:2410.15091, 2024

    2024 arXiv

  30. [39]

    Localmamba: Visual state space model with windowed selective scan,

    Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu, “Localmamba: Visual state space model with windowed selective scan,” in European Conference on Computer Vision . Springer, 2024, pp. 12–22

    2024

  31. [384]

    2, A-RWKV scales smoothly from tiny to small to base with consistent gains across bench- marks, aligning with our motivation for a stable, linear-time sequence operator

    As summarized in Tab. 2, A-RWKV scales smoothly from tiny to small to base with consistent gains across bench- marks, aligning with our motivation for a stable, linear-time sequence operator. Table 3. Scaling and downstream fine-tuning with A-RWKV . All models are pre-trained ...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.