Pith. sign in

REVIEW 4 major objections 5 minor 33 references

ASDA: Audio Spectrogram Differential Attention Mechanism for Self-Supervised Representation Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that replacing the standard Transformer attention with a differential attention mechanism—the difference of two softmax attention maps scaled by a tunable coefficient λ—improves self-supervised audio representation…

desk verdict A modest, plausible audio SSL gain built on Differential Transformer, but the attribution rests on single runs and a hyperparameter sweep on the test benchmarks; the multi-student variant is never ablated. read the letter →

arxiv 2507.02666 v1 pith:4HU37RPZ submitted 2025-07-03 cs.SD cs.AIcs.CLeess.AS

Junyu Wang , Tianrui Wang , Meng Ge , Longbiao Wang , Jianwu Dang This is my paper · ORCID
classification cs.SDcs.AIcs.CLeess.AS
keywords audioself-supervisedlearningdifferentialattentiontransformerclassificationkeywordspottingenvironmentalsoundmaskedautoencoderteacher-student
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper attempts to establish that a standard Transformer's attention mechanism wastes a share of its weights on irrelevant audio content, and that this waste can be suppressed by a differential attention operation: each attention head computes two softmax maps and subtracts the second, scaled by a tunable coefficient λ, from the first. Wrapped in a masked-autoencoder teacher–student setup with a shared EMA teacher and sixteen masked students, this mechanism is claimed to reach state-of-the-art scores on audio classification (49.0% mAP on AudioSet-2M, 41.5% mAP on AS20K), keyword spotting (98.3% accuracy on Speech Commands V2), and environmental sound classification (96.1% accuracy on ESC-50). The appeal is that the change is architectural and task-agnostic: if the claim holds, swapping standard attention for differential attention offers a cheap, general way to improve audio SSL representations across downstream benchmarks.

What carries the argument

The differential attention module is the load-bearing novelty: it replaces the single softmax attention map with the difference of two softmax maps, $\mathrm{softmax}(Q_1K_1^T/\sqrt{d}) - \lambda\,\mathrm{softmax}(Q_2K_2^T/\sqrt{d})$, intended to cancel the 'noise portion' of attention. Around it, the architecture combines a masked autoencoder target: a student encoder sees 20% unmasked patches, a teacher encoder (updated by exponential moving average) sees full input, a six-layer CNN decoder reconstructs frame-level spectrogram targets, and the sum of a frame-level MSE loss and an utterance-level contrastive loss on a head CLS token drives learning. The multi-student single-teacher variant runs sixteen students with different masks under one teacher to stabilize learning and spread compute.

What would settle it

Run the ASDA architecture identically with λ=0 and λ=0.3, across at least five seeds, on AS20K and AS-2M; if the mAP difference does not consistently exceed the seed-to-seed standard deviation, the claimed benefit of differential attention is not established. Alternatively, a direct inspection of attention maps could test whether standard attention actually assigns more weight to irrelevant spectrogram patches and whether the differential mechanism removes those patches.

Watch

Extended reading notes

Core claim

The paper's central claim is that attention noise in audio Transformers can be measured and removed by a dual-softmax differential mechanism. Concretely, the attention weights for a head are $\mathrm{softmax}(Q_1K_1^T/\sqrt{d}) - \lambda\,\mathrm{softmax}(Q_2K_2^T/\sqrt{d})$, computed from two separate query–key projections, with λ governing suppression strength; the output is LayerNorm(Diff(Z)V). The paper reports that embedding this mechanism in a 12-layer ViT-style encoder, trained with masked spectrogram reconstruction and an utterance-level contrastive loss against an EMA teacher, yields gains over the standard attention baseline (AS20K mAP rises from 41.0 to 41.5 at λ=0.3) and sets new best numbers on AudioSet-2M and ESC-50 among self-supervised methods.

Load-bearing premise

The entire performance story rests on the assumption that the 0.5-point mAP gap between the differential attention model (λ=0.3) and the standard attention model (λ=0) on AS20K reflects a real, repeatable improvement, rather than run-to-run noise, and that the same λ transfers to other tasks.

Editorial extensions

If this is right

  • Replacing standard attention with differential attention improves AS20K mAP from 41.0 (λ=0) to 41.5 (λ=0.3) without any other architectural change.
  • Choosing λ=0.3 is a sweet spot: λ=0.5 drops performance to 41.1 mAP, so the coefficient needs tuning rather than being set to an extreme.
  • The same pre-trained encoder transfers to three different tasks (AudioSet, Speech Commands V2, ESC-50), indicating the learned representation is not overfit to one benchmark.
  • Setting the utterance-loss weight α=0.5 and using a head CLS token each yield consistent small gains over α=1 and mean pooling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not isolate the multi-student design or the contrastive loss in an ablation, so the differential attention mechanism and these other components are confounded; a fair assessment would require ablating each separately.
  • The 0.5 mAP difference between λ=0 and λ=0.3 is reported without error bars; repeating the comparison across seeds would establish whether the attention change alone is responsible.
  • Because the differential operation only alters how attention weights are formed, it could in principle be transplanted into any audio or speech SSL backbone, with λ re-tuned per task—an extension the paper does not test.
  • A direct test of the mechanism's assumed effect would be to visualize attention maps on masked spectrograms and check that the standard softmax concentrates more probability on unrelated patches than the differential map does.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ASDA, a self-supervised audio representation learning method built on a masked autoencoder and a teacher-student framework. The main novelty is replacing standard softmax attention in the student and teacher encoders with a differential attention operation: each attention head computes two softmax attention maps and subtracts a scaled second map (Eq. 2). The model also introduces a multi-student single-teacher design, a CLS token, and a contrastive utterance-level loss in addition to the frame-level reconstruction loss. Experiments on AudioSet AS-2M/AS20K, Speech Commands V2, and ESC-50 report mAP and accuracy; the authors claim state-of-the-art performance among self-supervised pre-training methods. The central claim is that differential attention, together with the multi-student teacher-student setup, improves audio SSL transfer relative to methods such as EAT.

Significance. If the empirical results hold, the paper would show that a relatively small architectural change to attention—dual-softmax subtraction with a tuned coefficient—can improve transfer performance across several audio benchmarks, and that the multi-student single-teacher training scheme is a practical way to use low-cost student models. The paper reports results on standard public benchmarks, includes ablations for the loss weight and differential coefficient, and compares against a broad set of baselines. Its value is primarily empirical; the differential attention mechanism is adapted from prior work, and no formal analysis of the claimed noise suppression is provided. The main weakness is that the central claims rest on single-run point estimates, with small reported margins and hyperparameters selected on the evaluation sets, so the current evidence is thinner than the SOTA language in the abstract suggests.

major comments (4)
  1. [§4.2, Table 3] The only ablation that isolates the differential attention mechanism is run on AS20K with one point estimate per value of λ. The improvement from λ=0 to λ=0.3 is 0.5 mAP (41.0 to 41.5), while λ=0.1 already reaches 41.4. Without repeated seeds, confidence intervals, or a significance test, this difference is within plausible run-to-run noise, so the claim in §4.2 that the mechanism “significantly enhances model performance” is not supported by the evidence.
  2. [Tables 2 and 3, §4.1] The hyperparameters α and λ are selected on the same benchmarks that later define the SOTA claims: α is tuned on AS20K/SPC-2/ESC-50 (Table 2) and λ on AS20K (Table 3). Because the reported margins over EAT on AS-2M and ESC-50 are only 0.4 and 0.2 points, respectively, and SPC-2 is a tie at 98.3%, selection on the evaluation sets can plausibly account for the improvement. A hold-out protocol or nested validation (for example, tuning on AS20K and reporting only AS-2M/SPC-2/ESC-50) is needed to make the numbers predictive rather than selected.
  3. [§2.3, §3.2, Table 1] The full ASDA model differs from EAT in at least three respects: differential attention, a multi-student single-teacher architecture with n=16 students, and an additional contrastive loss. Only the differential coefficient is ablated (Table 3); the multi-student design and the contrastive loss are never removed. Consequently, even if the λ sweep were statistically reliable, the SOTA margin over EAT could not be attributed to differential attention rather than to the other architectural changes. An ablation of each component under matched training budgets is required to support the attribution.
  4. [Abstract and §4.1, Table 1] The abstract states SOTA performance on “multiple benchmarks,” but Table 1 shows the extra-supervised PaSST model at 96.8% on ESC-50 versus 96.1% for ASDA, and ASDA ties SPC-2 at 98.3% with several methods. The qualification that the comparison is among self-supervised methods appears only later in §4.1. The central SOTA claim should be stated accurately from the outset, and the ESC-50 result should not be described as a new SOTA without that qualifier.
minor comments (5)
  1. [§2.2, Eq. (2)] The dimension d in Eq. (2) is not defined precisely; if d is the per-head dimension D′, please state this explicitly.
  2. [§4.2, Table 3] The statement that λ=0 “resembles the standard ViT architecture” is inaccurate, since the model still includes the CLS token, the contrastive loss, and the teacher-student setup; please rephrase to say that only the differential attention operation is removed.
  3. [References] Reference [5] lists wav2vec 2.0 as appearing in ICML 2020; it was published at NeurIPS 2020.
  4. [§3.2] The fine-tuning protocol is not described in detail; please report the fine-tuning epochs, learning rate, and any augmentation or hyperparameter choices to support reproducibility.
  5. [General] No code or pretrained checkpoints are provided; given the small reported margins, releasing them, or at least reporting multiple seeds for the main results and the λ ablation, would materially help verification.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the ASDA claim rests on external benchmarks and an explicitly parameterized attention formula, with only a minor non-load-bearing self-citation.

full rationale

The paper is an empirical architecture study, not a derivation, so there is no claimed reduction chain that could collapse into its inputs. Differential attention is defined explicitly in Eq. (2) as Diff(Z)=softmax(Q1K1^T/sqrt(d)) - lambda*softmax(Q2K2^T/sqrt(d)); lambda and alpha are free hyperparameters selected in Tables 2-3, not quantities derived from the benchmarks they are later used to report. The SOTA claims are comparisons against externally reported baselines (EAT, BEATs, Audio-MAE) on public datasets, so they are externally falsifiable. The only self-citation, [3], is a general SSL reference in the introduction and is not load-bearing. The absence of error bars, selection of alpha and lambda on AS20K and ESC-50, and the unablated multi-student/contrastive components are statistical-validity concerns about whether the gain is real or attributable to differential attention, but they do not make the output equivalent to the input by construction. Under the hard-rule standard, this is not circularity; score 2 reflects only the minor non-load-bearing self-citation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central contribution is empirical; the method borrows differential attention from prior work and adds a multi-student teacher-student setup. The free parameters are hyperparameters tuned on validation or test sets, and the key axioms are standard SSL assumptions plus one mechanism-specific assumption that is not directly verified.

free parameters (4)
  • lambda (differential coefficient) = 0.3
    Tuned on AS20K (Table 3); all headline numbers use this value; no validation on other tasks.
  • alpha (loss weight) = 0.5
    Tuned on AS20K, SPC-2 and ESC-50 (Table 2); used in the total loss, Eq. (6).
  • number of students = 16
    Chosen for the multi-student architecture; no ablation shown; affects compute and effective masking diversity.
  • input masking ratio = 0.2
    Set based on computational efficiency and described as 'empirically validated', but no ablation table is provided.
assumptions (4)
  • domain assumption Pre-training on AudioSet with masked reconstruction and contrastive loss yields transferable representations for downstream tasks.
    Standard SSL premise; not proven in this paper beyond benchmark numbers.
  • domain assumption The teacher model's EMA-updated targets are stable enough to train 16 student models.
    Invoked in Section 2.3; no stability analysis is provided.
  • ad hoc to paper Dual-softmax differential attention with a fixed scalar lambda suppresses irrelevant context better than standard softmax attention.
    Core mechanism borrowed from [15]; the only evidence here is the AS20K ablation, with no attention-level analysis.
  • domain assumption Log-mel filterbank representation with 16x16 patches is an adequate input for audio transformer pre-training.
    Standard in the audio SSL literature; assumed without comparison in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ASDA: Audio Spectrogram Differential Attention Mechanism for Self-Supervised Representation Learning." pith.science (2026). https://pith.science/paper/4HU37RPZ

@misc{pith2026250702666,
  author       = {Pith},
  title        = {Pith review of: ASDA: Audio Spectrogram Differential Attention Mechanism for Self-Supervised Representation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4HU37RPZ}},
  note         = {Machine review of arXiv:2507.02666}
}
read the original abstract

In recent advancements in audio self-supervised representation learning, the standard Transformer architecture has emerged as the predominant approach, yet its attention mechanism often allocates a portion of attention weights to irrelevant information, potentially impairing the model's discriminative ability. To address this, we introduce a differential attention mechanism, which effectively mitigates ineffective attention allocation through the integration of dual-softmax operations and appropriately tuned differential coefficients. Experimental results demonstrate that our ASDA model achieves state-of-the-art (SOTA) performance across multiple benchmarks, including audio classification (49.0% mAP on AS-2M, 41.5% mAP on AS20K), keyword spotting (98.3% accuracy on SPC-2), and environmental sound classification (96.1% accuracy on ESC-50). These results highlight ASDA's effectiveness in audio tasks, paving the way for broader applications.

Figures

Figures reproduced from arXiv: 2507.02666 by the authors.

Figure 1
Figure 1. The overall architecture of the proposed ASDA model for self-supervised learning. dings, followed by a patching operation that segments the em￾beddings into non-overlapping 16 × 16 patches. Each patch is subsequently flattened and projected into a 768-dimensional vector via a linear transformation, forming the patch embed￾dings X ∈ R 50t×768 . Since the Transformer architecture lacks an inherent mech￾anism for posit… view at source ↗
Figure 2
Figure 2. Differential attention Module. mechanism, given an input feature matrix Z ∈ R L×D, we first apply linear transformations to obtain the query, key, and value representations. To achieve effective suppression of extraneous noise, we introduce a dual-path query-key mapping mechanism, mathematically formulated as follows: [Q1, Q2] = ZWQ, [K1, K2] = ZWK, V = ZWV (1) where WQ ∈ R D×2D′ , WK ∈ R D×2D′ , and WV ∈ R D×D′ are… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 30 canonical work pages

  1. [15]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS), 2017, p. 6000–6010

    work page 2017

  2. [1]

    Introduction In recent years, self-supervised learning (SSL) has demon- strated remarkable potential across various domains, including computer vision, natural language processing, and audio signal processing, by leveraging pre-training tasks such as contrastive learning and masked prediction to extract supervisory signals inherent in the data itself [1, ...

    work page

  3. [2]

    ASDA: Audio Spectrogram Differential Attention Mechanism for Self-Supervised Representation Learning

    Method 2.1. Model architecture The overall architecture of the proposed ASDA model is shown in Figure 1. Given a raw audio signal of approximately t sec- onds, we first convert it into a 128-dimensional log-mel filter- bank (fbank) representation. Specifically, a 25 ms Hamming window is applied every 10 ms, yielding an input spectrogram of shape 128 × 100...

    work page Pith review arXiv 2025

  4. [3]

    Experiments Our study leverages the large-scale AudioSet dataset [21] for model pre-training and evaluates its performance across three representative downstream tasks: audio classification (AS-2M and AS20K), keyword spotting (Speech Commands V2) [22], and environmental sound classification (ESC-50) [23]. 3.1. Datasets The AudioSet dataset comprises appro...

    work page

  5. [4]

    -” indicates that the data was not reported in the original paper. “Acc

    Results 4.1. Performance comparison on standard benchmarks Table 1 presents the performance comparison between our model and various classical baseline methods. The ex- perimental results demonstrate that compared to the current best-performing extra-supervised pre-training model [26], our method achieves a significant improvement of 1.9 % mAP on the larg...

    work page

  6. [5]

    By defining such irrelevant information as noise and drawing inspiration from differential denoising techniques, we design a dual-softmax based differential attention mecha- nism

    Conclusions In this paper, we introduce a novel differential attention mecha- nism to address the issue of standard Transformer architectures allocating excessive attention weights to irrelevant contextual information. By defining such irrelevant information as noise and drawing inspiration from differential denoising techniques, we design a dual-softmax ...

    work page

  7. [6]

    Masked autoencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll´ar, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2022, pp. 16 000–16 009

    work page 2022

  8. [7]

    Bert: Pre- training of deep bidirectional transformers for language under- standing,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- training of deep bidirectional transformers for language under- standing,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics (NAACL), 2019, pp. 4171–4186

    work page 2019

Show all 33 references
  1. [8]

    Progressive residual extrac- tion based pre-training for speech representation learning,

    T. Wang, J. Li, Z. Ma, R. Cao, X. Chen, L. Wang, M. Ge, X. Wang, Y . Wang, J. Dang, and N. Tashi, “Progressive residual extrac- tion based pre-training for speech representation learning,” arXiv preprint arXiv:2409.00387, 2024

    2024 arXiv

  2. [9]

    Hubert: Self-supervised speech rep- resentation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhut- dinov, and A. Mohamed, “Hubert: Self-supervised speech rep- resentation learning by masked prediction of hidden units,” IEEE Transactions on Acoustics, Speech and Signal Processing, vol. 29, p. 3451–3460, 2021

    2021

  3. [10]

    wav2vec 2.0: a framework for self-supervised learning of speech representa- tions,

    A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: a framework for self-supervised learning of speech representa- tions,” in Proceedings of the 34th International Conference on Neural Information Processing Systems (ICML), 2020

    2020

  4. [11]

    Ssast: Self- supervised audio spectrogram transformer,

    Y . Gong, C.-I. Lai, Y .-A. Chung, and J. Glass, “Ssast: Self- supervised audio spectrogram transformer,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, 2022, pp. 10 699–10 709

    2022

  5. [12]

    Masked autoencoders that listen,

    P.-Y . Huanget al., “Masked autoencoders that listen,” inProceed- ings of the 36th International Conference on Neural Information Processing Systems (NIPS), 2022, p. 28708–28720

    2022

  6. [13]

    Efficient self- supervised learning with contextualized target representations for vision, speech and language,

    A. Baevski, A. Babu, W.-N. Hsu, and M. Auli, “Efficient self- supervised learning with contextualized target representations for vision, speech and language,” in Proceedings of the 40th Interna- tional Conference on Machine Learning (ICML), 2023

    2023

  7. [14]

    Eat: Self- supervised pre-training with efficient audio transformer,

    W. Chen, Y . Liang, Z. Ma, Z. Zheng, and X. Chen, “Eat: Self- supervised pre-training with efficient audio transformer,” in Pro- ceedings of the Thirty-Third International Joint Conference on Ar- tificial Intelligence, IJCAI, 2024, pp. 3807–3815

    2024

  8. [16]

    Beats: audio pre-training with acoustic tokeniz- ers,

    S. Chen et al., “Beats: audio pre-training with acoustic tokeniz- ers,” in Proceedings of the 40th International Conference on Ma- chine Learning (ICML), 2023

    2023

  9. [17]

    An image is worth 16x16 words: Transform- ers for image recognition at scale,

    Dosovitskiy et al., “An image is worth 16x16 words: Transform- ers for image recognition at scale,” in Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021

    2021

  10. [18]

    Lost in the middle: How language mod- els use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language mod- els use long contexts,” Transactions of the Association for Com- putational Linguistics, vol. 12, pp. 157–173, 2024

    2024

  11. [19]

    Is attention interpretable?

    S. Serrano and N. A. Smith, “Is attention interpretable?” in Pro- ceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, 2019, pp. 2931–2951

    2019

  12. [20]

    Differential transformer,

    T. Ye, L. Dong, Y . Xia, Y . Sun, Y . Zhu, G. Huang, and F. Wei, “Differential transformer,” arXiv preprint arXiv:2410.05258 , 2024

    2024 arXiv

  13. [21]

    A wideband differential low-noise-amplifier with im3 harmonics and noise canceling,

    H. Lee, T. Chung, H. Seo, I. Choi, and B. Kim, “A wideband differential low-noise-amplifier with im3 harmonics and noise canceling,” IEEE Microwave and Wireless Components Letters , vol. 25, no. 1, pp. 46–48, 2015

    2015

  14. [22]

    A novel concept of smart headphones using active noise cancellation and speech recognition,

    A. B. Roy, A. Halder, R. Sharma, and V . Hegde, “A novel concept of smart headphones using active noise cancellation and speech recognition,” in 2015 International Conference on Smart Tech- nologies and Management for Computing, Communication, Con- trols, Energy and Materials ...

    2015

  15. [23]

    An exponen- tial moving average algorithm,

    D. Haynes, S. Corns, and G. K. Venayagamoorthy, “An exponen- tial moving average algorithm,” in 2012 IEEE Congress on Evo- lutionary Computation, 2012, pp. 1–8

    2012

  16. [24]

    data2vec: A general framework for self-supervised learning in speech, vision and language,

    A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “data2vec: A general framework for self-supervised learning in speech, vision and language,” in Proceedings of the 39th Interna- tional Conference on Machine Learning (PMLR), 2022, pp. 1298– 1312

    2022

  17. [25]

    Gaussian error linear units (gelus),

    D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  18. [26]

    Audio set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke et al., “Audio set: An ontology and human-labeled dataset for audio events,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2017, pp. 776–780

    2017

  19. [27]

    Speech commands: A dataset for limited-vocabulary speech recognition,

    P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” arXiv preprint arXiv:1804.03209, 2018

    2018 arXiv

  20. [28]

    Esc: Dataset for environmental sound classifica- tion,

    K. J. Piczak, “Esc: Dataset for environmental sound classifica- tion,” in Proceedings of the 23rd Annual ACM Conference on Multimedia. ACM Press, 2015, pp. 1015–1018

    2015

  21. [29]

    Ast: Audio spectrogram transformer,

    Y . Gong, Y .-A. Chung, and J. Glass, “Ast: Audio spectrogram transformer,” in Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech) , 2021, pp. 571–575

    2021

  22. [30]

    Attention bottlenecks for multimodal fusion,

    A. Nagrani, S. Yang, A. Arnab, A. Jansen, C. Schmid, and C. Sun, “Attention bottlenecks for multimodal fusion,” in Proceedings of the 35th International Conference on Neural Information Pro- cessing Systems (NIPS), 2021, p. 14200–14213

    2021

  23. [31]

    Effi- cient training of audio transformers with patchout,

    K. Koutini, J. Schl ¨uter, H. Eghbal-zadeh, and G. Widmer, “Effi- cient training of audio transformers with patchout,” in Proceed- ings of the Annual Conference of the International Speech Com- munication Association (Interspeech), 2022, pp. 2753–2757

    2022

  24. [32]

    Conformer-based self-supervised learning for non- speech audio tasks,

    S. Srivastava, Y . Wang, A. Tjandra, A. Kumar, C. Liu, K. Singh, and Y . Saraf, “Conformer-based self-supervised learning for non- speech audio tasks,” in IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) , 2022, pp. 8862– 8866

    2022

  25. [33]

    Adam: A method for stochastic opti- mization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic opti- mization,” in International Conference on Learning Representa- tions (ICLR), 2015, p. 13

    2015

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.