Pith. sign in

REVIEW 4 major objections 5 minor 42 references

On the use of Performer and Agent Attention for Spoken Language Identification

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Linear-complexity attention can replace self-attention in spoken language identification.

desk verdict Useful empirical benchmark undercut by an incorrect Performer formulation and missing reproducibility details; deserves peer review but not acceptance in current form. read the letter →

arxiv 2502.05841 v1 pith:7GRWWUAC submitted 2025-02-09 eess.AS cs.SD

Jitendra Kumar dhiman , Jainag Ambati This is my paper · ORCID
classification eess.AScs.SD
keywords spokenlanguageidentificationattentivestatisticalpoolingperformerattentionagentself-attentionself-supervisedspeechrepresentationslinearcomplexityBEST-RQ
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper makes the case that two linear-complexity attention mechanisms, performer-attention and agent-attention, can take the place of the quadratic self-attention used in attentive statistical pooling for spoken language identification. Using frozen frame-level embeddings from a pre-trained BEST-RQ model, it attaches a small pooling and classifier block and reports that performer-attention at $r=128$ raises average accuracy and macro-F1 across VoxPopuli, FLEURS, and VoxLingua, while agent-attention at $p=4$ stays competitive with self-attention at lower computational cost. The point of the claim is practical: if linear attention works at least as well as self-attention, LID systems can handle longer utterances and stricter latency budgets without giving up accuracy.

What carries the argument

The load-bearing component is attentive statistical pooling with the attention mechanism swapped in the middle. Self-attention computes $C = \operatorname{softmax}(QK^T/\sqrt{d})V$ at $O(N^2 d)$ cost. Performer-attention replaces the softmax with a random-feature kernel approximation, writing $\operatorname{softmax}(QK^T/\sqrt{d}) \approx Q'K'^T/\sqrt{r}$ and computing $C_p = (1/\sqrt{r})Q'(K'^T V)$, which uses the associativity of matrix multiplication to reach $O(N r d)$. Agent-attention introduces an agent matrix $G$ obtained by pooling the queries, computes $V_a = \operatorname{softmax}(GK^T/\sqrt{d})V$ at the agent level and $C_a = \operatorname{softmax}(QG^T/\sqrt{d})V_a$, giving $O(N n d)$. The pooled representation is then the concatenation of the mean and standard deviation of the context matrix across time.

What would settle it

Re-implement the Performer attention exactly as published, with the full algorithm including the normalizing rescaling and the random projection, plug it into the same BEST-RQ plus statistical-pooling LID pipeline on the same VoxPopuli, FLEURS, and VoxLingua splits, and check whether it still beats self-attention; if it does not, the reported advantage belongs to a different attention variant.

Watch

Extended reading notes

Core claim

The authors claim that in a language identification pipeline built on self-supervised speech embeddings, replacing the vanilla self-attention in attentive statistical pooling with performer-attention improves the average accuracy and macro-F1 on all three evaluated datasets, with the largest relative gains on FLEURS (about 18% in both metrics) and on VoxLingua macro-F1 (about 45%). Agent-attention is reported as comparable to or occasionally better than self-attention while using less computation. The mechanism is linear in sequence length: performer-attention approximates the softmax attention map with a low-rank kernel product, and agent-attention first aggregates keys and values through a small agent matrix and then broadcasts back to the queries. The paper treats these results as evidence that efficient attention variants can replace the standard quadratic self-attention in the pooling layer.

Load-bearing premise

The central comparison stands only if the mechanism the paper calls 'performer-attention' is actually the published Performer method, including its normalization and projection steps; the paper's equations leave those steps unspecified.

Editorial extensions

If this is right

  • On the three datasets, performer-attention at $r=128$ raises average accuracy from 74.43% to 79.10% and average macro-F1 from 63.27% to 74.28% compared with self-attention.
  • Agent-attention at $p=4$ reaches average accuracy 77.34% and macro-F1 72.10%, staying close to self-attention while using linear complexity.
  • Because the cost of both replacements is $O(N r d)$ or $O(N n d)$, the pooling layer no longer needs to pay a quadratic price in the number of time frames, which matters for long utterances.
  • The added parameters are small relative to the frozen 440M-parameter backbone, so the efficiency gain does not come from a larger classifier.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the implementation follows Section 3.1.3 exactly, it omits the normalizing denominator of the published Performer estimator, so the reported 'performer-attention' may be a different unnormalized kernel attention; the outperformance claim would then be about that variant, not the original method.
  • The largest accuracy gains appear on FLEURS and VoxLingua, where test speakers are not seen in pre-training, so the benefit may depend on how much domain shift the pooling layer must absorb; re-running with matched versus disjoint speakers would test this.
  • The same complexity argument applies to speaker verification and other utterance-level pooling tasks, which the paper does not run; speaker identification is listed as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies the use of two linear-complexity attention mechanisms, Performer and Agent Attention, as replacements for vanilla self-attention in attentive statistical pooling for spoken language identification (LID). The authors fine-tune a frozen BEST-RQ encoder with a small trainable LID head on three datasets (VoxPopuli, FLEURS, VoxLingua), comparing self-attention with Performer (r = 32, 64, 128, 256) and Agent Attention (p = 2, 4, 6) in terms of accuracy and macro-F1. They report that Performer with r = 128 gives the best average metrics, outperforming self-attention on all three datasets, and that Agent Attention is comparable or slightly better than self-attention while having lower asymptotic complexity. The paper also claims overall relative F1 improvements of 1.74% and 1.4% for Performer and Agent Attention over self-attention, respectively, and states that these mechanisms are computationally less expensive.

Significance. If the results are reliable and the implemented mechanisms indeed correspond to the published Performer and Agent Attention, the paper provides useful empirical evidence that linear-complexity attention can replace quadratic self-attention in LID pooling, which is relevant for real-time applications. The study covers three datasets and a systematic hyperparameter sweep, and the comparison is empirical rather than circular. However, the significance is conditional on resolving the implementation fidelity of the Performer block and on providing statistical support for the claimed improvements, especially because some gains are small (e.g., 0.17 absolute accuracy on VoxPopuli) and no error bars or significance tests are reported.

major comments (4)
  1. [§3.1.3, Eqs. (5)–(6)] The Performer implementation as written deviates from the FAVOR+ estimator in [18]. Eq. (5) states σ(QK^T/√d) ≈ Q'K'^T/√r and Eq. (6) defines C_p = (1/√r) Q'(K'^T V). The published FAVOR+ estimator includes a row-wise normalizer D^{-1}, where D = diag(φ(Q)(φ(K)^T 1_N)), which restores the row-stochastic property of the attention weights. Because this normalizer is omitted and the projection kernel φ is left unspecified, the evaluated mechanism is an unnormalized linear attention variant rather than the Performer from [18]. The reported accuracy/F1 gains therefore cannot be attributed to Performer without either confirming that the implementation actually matches [18] or re-framing the contribution as an evaluation of this specific variant.
  2. [§5, Table 4 (bullet list)] The 'overall relative improvement over self-attention' for F1 is miscalculated. From Table 4, Performer r=128 improves average F1 by (74.28−63.27)/63.27 = 17.4%, not 1.74%; Agent p=4 improves by (72.10−63.27)/63.27 = 14.0%, not 1.4%. The corresponding Acc improvements (6.3% and 3.9%) are correct. These errors appear in the Section 5 bullet list and are reflected in the abstract's summary, and they should be corrected.
  3. [§5, Table 4] All results in Table 4 are from a single training run; no seeds, error bars, or significance tests are reported. This is particularly problematic for the VoxPopuli dataset, where Performer r=128 outperforms self-attention by only 0.17 absolute accuracy (89.78 vs. 89.61) and 0.19 F1. Without multiple runs or a paired significance test, the claim that Performer 'consistently' outperforms self-attention across all three datasets is not statistically supported. Please provide at least the number of runs and the variance, or a statistical test.
  4. [Abstract and §6] The abstract and conclusion state that Agent Attention is 'computationally less expensive' than self-attention, but no runtime, memory, or FLOPs measurements are presented anywhere in the paper. Table 1 only gives asymptotic complexities. If the claim is about asymptotic complexity, it should be stated as such; if it concerns observed computation, an empirical comparison of training/inference time should be added.
minor comments (5)
  1. [Table 4] The column header 'VoxPopulli' is a misspelling; it should be 'VoxPopuli'.
  2. [§4.1, Table 2] The text says 'We use 50 hours and 11 hours of data per language in the train and test/dev sets,' but Table 2 lists VoxPopuli train/dev/test total durations as 1324/267/268 hours. These numbers should be reconciled with the per-language allocation.
  3. [§4.1] The sentence 'The VoxLingua dataset comprises a total of 107 languages in its train set and 33 languages in its test set' appears twice in the same paragraph; remove the duplicate.
  4. [§5, bullet list] The term 'ablation studies' is used for hyperparameter sweeps over r and p; these are parameter sweeps rather than ablations and should be labelled accordingly.
  5. [§6] The phrase 'we showed that interestingly the attention mechanisms having linear complexity can potentially replace...' is grammatically awkward; consider rephrasing.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the attention comparison is an empirical evaluation with externally imported mechanisms and no fitted parameter renamed as a prediction.

full rationale

The paper's central claim is an empirical comparison of three attention mechanisms (self-, performer-, and agent-attention) inside an attentive statistical pooling layer for spoken language identification. The mechanisms are imported from external prior work ([18], [19], [23]); the paper does not derive any theoretical result from a premise that already contains the conclusion. Equations (1)-(6) restate standard definitions from those external sources, and the subsequent accuracy/F1 measurements on VoxPopuli, FLEURS, and VoxLingua are direct experimental outcomes, not quantities recovered from a fitted parameter. Hyper-parameter selection (r=128, p=4) is performed on the dev set and reported as model selection, not as a prediction. There are no self-citations, and no uniqueness theorem or ansatz is imported from the authors' own prior work. A possible concern is that Eq. (5)-(6) omit the D^{-1} normalizer of the FAVOR+ estimator from [18], and phi is left unspecified; however, that is a question of implementation fidelity and reproducibility relative to the cited method, not circularity, because the claim is not forced by construction to equal its inputs. The manuscript is therefore self-contained as an empirical study and exhibits no circular derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical and does not create new theoretical entities. The main burden is concentrated in the unspecified performer kernel (Eq. 5), the hyperparameter sweeps for r and p, and the evaluation restriction to pre-training languages; all of these affect the strength of the 'outperforms' claim.

free parameters (3)
  • r (performer projection dimension) = 128 (best of 32, 64, 128, 256)
    Performer attention uses a low-dimensional random-feature projection of dimension r. The paper sweeps r and reports r=128 as best; this is a hyperparameter tuned on the dev set, not derived from theory.
  • p (agent pooling layers) = 4 (best of 2, 4, 6)
    Agent attention derives its agent matrix by applying p pair-wise pooling layers to the query matrix. The best value is selected from a sweep and affects the agent count n and the accuracy.
  • Number of attention heads and attention dimension = 4 heads, 64 dimension
    The LID module uses 4 heads and an attention dimension of 64 for all variants. These values are chosen without ablation, and they shape the parameter counts and representational capacity of every compared method.
assumptions (4)
  • ad hoc to paper The performer projection kernel φ exists and satisfies the approximation in Eq. (5).
    Section 3.1.3 introduces Q' and K' whose rows are φ(q_i) and φ(k_i), but does not specify φ, its randomness, or the normalization needed by the original Performer estimator. The central comparison depends on this unspecified approximation holding.
  • domain assumption Frozen BEST-RQ embeddings retain sufficient language-discriminative information for LID.
    The whole pipeline keeps BEST-RQ frozen (Section 4.3); if these embeddings are not informative, the pooling-layer comparison is meaningless. No analysis of embedding quality is provided.
  • domain assumption Testing on languages seen during pre-training is a fair evaluation setting.
    Section 4.1 restricts FLEURS and VoxLingua to 23 and 17 languages that overlap with the 23 pre-training languages. The results may not transfer to unseen languages, yet the conclusions are stated without this caveat.
  • domain assumption Standard softmax self-attention is the appropriate baseline for LID pooling.
    The paper compares only against self-attention ASP. Other pooling strategies from the cited LID literature, such as simple temporal averaging or other attentive temporal pooling variants, are not compared, so the baseline set is assumed sufficient.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the use of Performer and Agent Attention for Spoken Language Identification." pith.science (2026). https://pith.science/paper/7GRWWUAC

@misc{pith2026250205841,
  author       = {Pith},
  title        = {Pith review of: On the use of Performer and Agent Attention for Spoken Language Identification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7GRWWUAC}},
  note         = {Machine review of arXiv:2502.05841}
}
read the original abstract

One of the methods for language Identification (LID) involves deriving speech representation from pre-trained models using self-supervised learning, followed by fine-tuning the model for the LID task. State-of-the-art approaches for LID use an attention-based statistical pooling layer to facilitate the aggregation of contextual information across time frames of the embedding vectors extracted from the pre-trained model. In this paper, we delve into exploring recently proposed attention mechanisms, namely performer and agent-attention, in conjunction with the statistical pooling layer. The LID experiments are performed on three datasets: VoxPopuli, FLEURS, and VoxLingua. We compare their performance against vanilla self-attention. Our findings suggest that performer-attention outperforms self-attention and agent-attention exhibits comparable or occasionally superior performance to self-attention, while also being computationally less expensive.

Figures

Figures reproduced from arXiv: 2502.05841 by the authors.

Figure 1
Figure 1. Block diagram for the LID classifier. The BEST-RQ block is pre-trained using self-supervised learning framework followed by fine-tuning of LID classifier. (ASP) has emerged as a valuable technique. In this approach, the frame-level embeddings are combined through attention weights followed by the statistical pooling layer resulting in more emphasis on the most discriminative features while re￾ducing the impacts of l… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 29 canonical work pages

  1. [18]

    Language identifica- tion from short segments of speech,

    J. Balleda, H. A. Murthy, and T. Nagarajan, “Language identifica- tion from short segments of speech,” in Sixth International Con- ference on Spoken Language Processing, 2000

    work page 2000

  2. [1]

    It is often used in applications like speech recogni- tion [1, 2], language translation [3], and content filtering [4]

    Introduction Spoken Language Identification (LID) is the process of auto- matically determining the language spoken in an audio record- ing. It is often used in applications like speech recogni- tion [1, 2], language translation [3], and content filtering [4]. There are several challenges for this task. For instance, different languages exhibit pronunciat...

    work page

  3. [2]

    We analyze the impact of self-attention, performer-attention, and agent-attention on the performance of LID systems

    work page

  4. [3]

    Notations: We denote vectors and matrices by boldface low- ercase and uppercase letters, respectively

    Our findings suggest that performer-attention outperforms self-attention and agent-attention exhibits comparable or oc- casionally superior performance to self-attention, while also being computationally less expensive. Notations: We denote vectors and matrices by boldface low- ercase and uppercase letters, respectively. The operation (·)T denotes the tra...

    work page

  5. [4]

    On the use of Performer and Agent Attention for Spoken Language Identification

    Related Work A straightforward approach to incorporating contextual infor- mation from frame-level embeddings is statistical aggregation across the temporal dimension, such as mean and variance. However, not all the frame-level embeddings contribute equally to the utterance-level representation for Language Identifica- tion. Therefore, relying solely on n...

    work page Pith review arXiv 2025

  6. [5]

    We refer to the output of the pre-trained network (BEST-RQ) as the embedding sequence

    Proposed Framework Fig.-1 depicts the block diagram of our proposed framework. We refer to the output of the pre-trained network (BEST-RQ) as the embedding sequence. Consider embedding sequence repre- sented by X = [x1, x2, . . . ,xN ]T ∈ RN ×d where N and d denote the sequence length and the embedding dimension, re- spectively. We refer the vectorxi(1 ≤ ...

    work page

  7. [6]

    Attention matrix: Attention matrix is given by A = σ QK T √ d ∈ RN ×N (1) where σ(·) is row-wise Softmax operation

    work page

  8. [7]

    The computational complexity for both Eq

    Compute weighted sum: The context matrix is given by C = AV ∈ RN ×d (2) where each row in C represents the weighted sum of the value vectors in V . The computational complexity for both Eq. (1) and Eq. (2) is of the order of O(N 2d), which is quadratic in terms of the input sequence length N, making it computationally expensive for long sequences. 3.1.2. ...

    work page

Show all 42 references
  1. [8]

    Agent aggregation: Consider G ∈ Rn×d, where n ≪ N, the agent aggregation is given by Va = σ GK T √ d V (3) where Va ∈ Rn×d

  2. [9]

    Typically, the agent matrix is given byG = Pooling(Q) (Sec- tion 4.4)

    Agent broadcast: the agent matrix broadcasts the information from Va according to the following equation Ca = σ QGT √ d Va (4) where Ca ∈ RN ×d. Typically, the agent matrix is given byG = Pooling(Q) (Sec- tion 4.4). In addition, we also adopt a depth-wise convolution over V as...

  3. [10]

    Dataset Pre-training Data: We utilized the open-source V oxPopuli dataset [25] for pre-training the BEST-RQ model

    Experimental Setup 4.1. Dataset Pre-training Data: We utilized the open-source V oxPopuli dataset [25] for pre-training the BEST-RQ model. This dataset comprises 400K hours of unlabelled speech data spanning 23 Table 2: Total duration of each of the datasets and the average ut...

  4. [11]

    These results are reported in Table-4

    Results The language identification performance is measured using av- erage accuracy (Acc) and macro-F1 score. These results are reported in Table-4. For each of the datasets, the evaluation was performed on the test set by using the checkpoint that provided the best accuracy ...

  5. [12]

    In particu- lar, we demonstrated the effectiveness of performer- and agent- attention over self-attention for language identification in a self- supervised learning paradigm

    Conclusions and Future Work In this work, we showed that interestingly the attention mecha- nisms having linear complexity can potentially replace the com- putationally expensive self-attention mechanism. In particu- lar, we demonstrated the effectiveness of performer- and age...

  6. [13]

    Streaming end-to-end multilin- gual speech recognition with joint language identification,

    C. Zhang, B. Li, T. N. Sainath, T. Strohman, S. Mavandadi, S. yiin Chang, and P. Haghani, “Streaming end-to-end multilin- gual speech recognition with joint language identification,”ArXiv, vol. abs/2209.06058, 2022

    2022 arXiv

  7. [14]

    A real-time end-to-end multi- lingual speech recognition architecture,

    J. Gonzalez-Dominguez, D. Eustis, I. L ´opez-Moreno, A. W. Se- nior, F. Beaufays, and P. J. Moreno, “A real-time end-to-end multi- lingual speech recognition architecture,”IEEE Journal of Selected Topics in Signal Processing, vol. 9, pp. 749–759, 2015

    2015

  8. [15]

    Transformer-based machine translation for low- resourced languages embedded with language identification,

    T. Sefara, S. Zwane, N. Gama, H. Sibisi, P. N. Senoamadi, and V . Marivate, “Transformer-based machine translation for low- resourced languages embedded with language identification,” in Proc. of Information Communications Technology and Society (ICTAS), pp. 127–132, 2021

    2021

  9. [16]

    Language id in the wild: Unexpected challenges on the path to a thousand- language web text corpus,

    I. Caswell, T. Breiner, D. van Esch, and A. Bapna, “Language id in the wild: Unexpected challenges on the path to a thousand- language web text corpus,” ArXiv, vol. abs/2010.14571, 2020

    2010 arXiv

  10. [17]

    Language and dialect identification: A survey,

    A. Etman and L. Beex, “Language and dialect identification: A survey,” in Proc. of SAI Intelligent Systems Conference (Intel- liSys), pp. 220–231, 2015

    2015

  11. [19]

    Self-supervised learning: Generative or contrastive,

    X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang, “Self-supervised learning: Generative or contrastive,”IEEE trans- actions on knowledge and data engineering , vol. 35, no. 1, pp. 857–876, 2021

    2021

  12. [20]

    Representation learning with contrastive predictive coding,

    A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  13. [21]

    wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,” Advances in neural information processing systems , vol. 33, pp. 12 449–12 460, 2020

    2020

  14. [22]

    W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,

    Y .-A. Chung, Y . Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y . Wu, “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” in IEEE Automatic Speech Recognition and Understanding Work- shop (ASRU), 2021, pp. 244–250

    2021

  15. [23]

    Hubert: Self-supervised speech rep- resentation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhut- dinov, and A. Mohamed, “Hubert: Self-supervised speech rep- resentation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 29, pp. 3451–3460, 2021

    2021

  16. [24]

    Efficient self-supervised learning rep- resentations for spoken language identification,

    H. Liu, L. P. Garc ´ıa-Perera, A. W. H. Khong, E. S. Chng, S. J. Styles, and S. Khudanpur, “Efficient self-supervised learning rep- resentations for spoken language identification,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, pp. 1296–1307, 2022

    2022

  17. [25]

    Asbert: Asr-specific self-supervised learning with self-training,

    H. Y . Kim, B.-Y . Kim, S. W. Yoo, Y . Lim, Y . Lim, and H. Lee, “Asbert: Asr-specific self-supervised learning with self-training,” in IEEE Spoken Language Technology Workshop (SLT), 2023, pp. 9–14

    2023

  18. [26]

    Self-supervised learning with random-projection quantizer for speech recogni- tion,

    C.-C. Chiu, J. Qin, Y . Zhang, J. Yu, and Y . Wu, “Self-supervised learning with random-projection quantizer for speech recogni- tion,” inInternational Conference on Machine Learning. PMLR, 2022, pp. 3915–3924

    2022

  19. [27]

    Attention mechanisms in computer vision: A survey,

    M.-H. Guo, T.-X. Xu, J.-J. Liu, Z.-N. Liu, P.-T. Jiang, T.-J. Mu, S.- H. Zhang, R. R. Martin, M.-M. Cheng, and S.-M. Hu, “Attention mechanisms in computer vision: A survey,”Computational visual media, vol. 8, no. 3, pp. 331–368, 2022

    2022

  20. [28]

    An introductory survey on attention mechanisms in NLP problems,

    D. Hu, “An introductory survey on attention mechanisms in NLP problems,” in Intelligent Systems and Applications: Proceedings of the 2019 Intelligent Systems Conference (IntelliSys) Volume 2. Springer, 2020, pp. 432–448

    2019

  21. [29]

    A survey on efficient training of transformers,

    B. Zhuang, J. Liu, Z. Pan, H. He, Y . Weng, and C. Shen, “A survey on efficient training of transformers,” arXiv preprint arXiv:2302.01107, 2023

    2023 arXiv

  22. [30]

    Rethinking attention with performers,

    K. Choromanski, V . Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarl´os, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, D. Be- langer, L. J. Colwell, and A. Weller, “Rethinking attention with performers,” ArXiv, vol. abs/2009.14794, 2020

    2009 arXiv

  23. [31]

    Agent at- tention: On the integration of softmax and linear attention,

    D. Han, T. Ye, Y . Han, Z. Xia, S. Song, and G. Huang, “Agent at- tention: On the integration of softmax and linear attention,”ArXiv, vol. abs/2312.08874, 2023

    2023 arXiv

  24. [32]

    Attentive statis- tics pooling for deep speaker embedding,

    K. Okabe, T. Koshinaka, and K. Shinoda, “Attentive statis- tics pooling for deep speaker embedding,” ArXiv, vol. abs/1803.10963, 2018

    2018 arXiv

  25. [33]

    On the performance of time-pooling strategies for end-to-end spoken language identifi- cation,

    J. Monteiro, M. J. Alam, and T. H. Falk, “On the performance of time-pooling strategies for end-to-end spoken language identifi- cation,” in International Conference on Language Resources and Evaluation, 2020

    2020

  26. [34]

    Attentive temporal pooling for conformer-based streaming lan- guage identification in long-form speech,

    Q. Wang, Y . Yu, J. W. Pelecanos, Y . Huang, and I. L´opez-Moreno, “Attentive temporal pooling for conformer-based streaming lan- guage identification in long-form speech,” in The Speaker and Language Recognition Workshop, 2022

    2022

  27. [35]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017

    2017

  28. [36]

    Flatten trans- former: Vision transformer using focused linear attention,

    D. Han, X. Pan, Y . Han, S. Song, and G. Huang, “Flatten trans- former: Vision transformer using focused linear attention,” inPro- ceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 5961–5971

    2023

  29. [37]

    V oxpopuli: A large-scale multilingual speech corpus for representation learn- ing, semi-supervised learning and interpretation,

    C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haz- iza, M. Williamson, J. Pino, and E. Dupoux, “V oxpopuli: A large-scale multilingual speech corpus for representation learn- ing, semi-supervised learning and interpretation,” arXiv preprint arXiv:2101.00390, 2021

    2021 arXiv

  30. [38]

    V oxlingua107: a dataset for spoken lan- guage recognition,

    J. Valk and T. Alum ¨ae, “V oxlingua107: a dataset for spoken lan- guage recognition,” in IEEE Spoken Language Technology Work- shop (SLT), 2021, pp. 652–658

    2021

  31. [39]

    Fleurs: Few-shot learning evaluation of universal representations of speech,

    A. Conneau, M. Ma, S. Khanuja, Y . Zhang, V . Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna, “Fleurs: Few-shot learning evaluation of universal representations of speech,” inIEEE Spoken Language Technology Workshop (SLT), 2023, pp. 798–805

    2023

  32. [40]

    Swish: a self-gated activation function,

    P. Ramachandran, B. Zoph, and Q. V . Le, “Swish: a self-gated activation function,” arXiv: Neural and Evolutionary Computing, 2017

    2017

  33. [41]

    Adam: A method for stochastic opti- mization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic opti- mization,” arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  34. [42]

    Bert: Pre- training of deep bidirectional transformers for language under- standing,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- training of deep bidirectional transformers for language under- standing,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.