Pith. sign in

REVIEW 3 major objections 5 minor 44 references

Accurate, fast, cheap: Choose three. Replacing Multi-Head-Attention with Bidirectional Recurrent Attention for Long-Form ASR

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Bidirectional recurrent attention matches multi-head attention for long-form ASR at 44% higher throughput.

desk verdict Solid empirical paper on replacing MHA with bidirectional RWKV; DirDrop is genuinely new, but the headline throughput number is encoder-only, not full-system. read the letter →

arxiv 2506.19761 v1 pith:XQK3FBPN submitted 2025-06-24 cs.CL

Martin Ratajczak , Jean-Philippe Robichaud , Jennifer Drexler Fox This is my paper · ORCID
classification cs.CL
keywords long-formASRrecurrentattentionRWKVMamba-2DirectionDropoutConformer-TransducerlinearcomplexityGigaSpeech
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether linear-complexity recurrent attention can replace quadratic multi-head attention in the encoder of a speech recognizer for long audio. It reports that a bidirectional RWKV recurrent attention layer matches the word error rate of multi-head attention on both short and long-form speech, and after a long-form training and fine-tuning protocol it beats a limited-context attention baseline while processing 44% more audio per second. A second contribution, Direction Dropout, randomly drops one direction of the bidirectional layers during training; the resulting model can be decoded bidirectionally, unidirectionally, or in an alternating-directions mode that nearly matches bidirectional accuracy at unidirectional cost. If these results hold, long-form ASR can avoid the usual trade-off between chunked decoding and full-context accuracy without giving up speed.

What carries the argument

The load-bearing mechanism is the bidirectional recurrent attention layer: at each encoder position the output is computed from a hidden state that accumulates information from all previous frames in the forward pass or all following frames in the backward pass, and the two directions' outputs are averaged, giving linear time and memory in sequence length. The paper replaces the multi-head attention block of each Conformer encoder layer with the time-mixing component of RWKV v6, and for comparison with the Mamba-2 state-space model layer, keeping all other layers identical. Direction Dropout is the second mechanism: during training each Conformer block randomly drops one of the two recurrent directions with probability 20%, which both regularizes the bidirectional model and lets a single checkpoint be decoded with any subset of layers bidirectional, or in an alternating-direction mode that interleaves directions across layers.

What would settle it

Decode a natural multi-hour recording, with speaker turns and topic shifts crossing 400-second boundaries, using the long-form-fine-tuned bidirectional RWKV and limited-context attention models under the paper's own 40k-frame chunking, and compare full-file WER and minutes-of-audio-per-second; if bi-RWKV does not improve on the baseline's WER while maintaining 44% higher throughput, the central claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that the attention blocks of a Conformer-Transducer encoder can be swapped for bidirectional recurrent attention layers, specifically the RWKV v6 time-mixing component, without losing accuracy, while cutting attention cost from quadratic to linear in sequence length. On short-form segments, bidirectional RWKV reaches 11.0/11.0 WER against 10.9/10.9 for multi-head attention. On long-form decoding of full files, after one epoch of fine-tuning on 75-85-second utterances, bidirectional RWKV reaches 12.1/12.9 WER at 40k-frame chunks versus 12.3/13.1 for limited-context attention with global tokens, at 26 versus 18 minutes of audio per second; unidirectional RWKV reaches 31 MPS at accuracy comparable to the baseline. Direction Dropout trains the bidirectional model to also run left-to-right, right-to-left, or alternating-direction decoding, with alternating decoding nearly matching full bidirectional accuracy and the bidirectional model matching multi-head attention accuracy.

Load-bearing premise

The long-form claims assume that training on concatenated 10-to-85-second utterances and decoding in fixed chunks up to 400 seconds captures what makes multi-hour audio difficult, so the reported accuracy and speed gains carry over to natural recordings of hours.

Editorial extensions

If this is right

  • At matched accuracy, replacing multi-head attention with bidirectional RWKV in the encoder raises long-form throughput from 18 to 26 minutes of audio per second, a 44% gain, and unidirectional RWKV reaches 31 MPS with accuracy comparable to the limited-context baseline.
  • Recurrent attention generalizes across sequence lengths: the RWKV model trained only on short segments keeps WER near 12.8/13.6 at 40k-frame chunks, while multi-head attention degrades to 20.6/22.4.
  • Direction Dropout makes alternating-direction decoding nearly as accurate as full bidirectional decoding, so most of the benefit of future context can be obtained at unidirectional cost.
  • A single DirDrop-trained bidirectional model can be decoded in bidirectional, left-to-right, right-to-left, or alternating modes, letting one checkpoint serve multiple latency and accuracy regimes.
  • Long-form fine-tuning that updates only the attention blocks improves all attention types, but helps RWKV more than limited-context attention, closing or reversing the accuracy gap after one epoch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Alternating-direction decoding performing near bidirectional accuracy implies that future and past context do not need to be merged inside every layer; a natural extension is to schedule directions per layer or per segment at runtime, turning direction choice into a compute-quality knob.
  • The short-form-trained RWKV model's long-form generalization, if replicated, suggests recurrent attention layers could reduce dependence on long-utterance training data; a direct test is training on 4-second segments and measuring WER at 40k-frame chunks against a multi-head attention model.
  • Extending recurrent attention to the transducer decoder, which the paper lists as future work, could make the encoder less of a throughput bottleneck, so end-to-end speedups may exceed the reported encoder-only 44%.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies replacing multi-head attention (MHA) in a Conformer-Transducer ASR encoder with bidirectional recurrent attention (RA) layers—RWKV v6 and Mamba-2—for long-form speech recognition. The authors train Conformer-Transducer models on GigaSpeech XL with identical architecture except for the attention block, evaluate on short-form and long-form (chunked) decoding, and report WER and throughput. They introduce Direction Dropout (DirDrop), which randomly drops one direction of a bidirectional RA layer during training, enabling unidirectional, alternating-direction, or fully bidirectional decoding from a single model. The main empirical claims are: (i) bidirectional RWKV matches MHA accuracy on short-form segments; (ii) after long-form training and light fine-tuning on 75-85 s concatenated utterances, a bi-RWKV encoder is slightly more accurate than a limited-context attention (LCA+GT) baseline (e.g., 12.2/13.0 vs 12.3/13.1 at 20k-frame chunks) at 26 vs 18 minutes-of-audio-per-second encoder throughput; and (iii) DirDrop-trained models can decode in unidirectional mode at near-bidirectional accuracy, with alternating-direction decoding near bidirectional accuracy at higher throughput. The paper presents controlled experiments with a fixed decoder and training pipeline, and all WER numbers are on the external GigaSpeech dev/test partitions.

Significance. The result, if confirmed, is practically valuable: it would demonstrate that linear-complexity recurrent attention layers can replace quadratic MHA in a production-style transducer ASR model without sacrificing accuracy on a 10,000-hour benchmark, and that a single model can be configured for different latency/throughput regimes. The strength of the paper is its controlled ablation structure—same encoder depth, same decoder, same data, only the attention computation is varied—and the interesting empirical finding that RWKV generalizes better than MHA from short to long inputs (Table 2). The DirDrop result, especially the alternating-direction decoding phenomenon, is a novel contribution that may be of independent interest. However, the headline 'more accurate and 44% faster' is currently supported only by sub-decimal WER differences without uncertainty quantification and by encoder-only throughput measurements, so the breadth of the claim exceeds the evidence in its present form.

major comments (3)
  1. [Abstract; §4.3; Table 3] The '44% higher throughput' claim is measured with the transducer removed. Section 4.3 states that the transducer portion was removed for throughput evaluation, while all WER numbers are produced by the full Conformer-Transducer with CTC projection, SAD, and LSTM transducer decoded with joint CTC+RNN-T. An encoder-only speedup of 26 vs 18 MPS does not translate to the same speedup for the full system, because the shared decoder and joint decoding cost dilute the relative gain. Please report end-to-end throughput (or at least an estimate including decoder cost) for the models used in the WER tables, or explicitly restrict the throughput claim to the encoder. As written, the abstract's 'more accurate... with 44% higher throughput' overstates what is measured.
  2. [§5.2; Table 3] The 'more accurate than LCA' claim rests on WER differences of 0.1-0.2 points absolute (e.g., bi-RWKV 12.2/13.0 vs LCA 12.3/13.1 at 20k chunks, and 12.1/12.9 vs 12.3/13.1 at 40k). No confidence intervals, number of evaluation utterances, or significance tests are reported, and the paper does not state whether the dev/test sets are scored in a way that would make such small differences reliable. Because the claimed advantage is within the range of typical run-to-run variation, please provide uncertainty estimates or repeated decoding runs, or soften the wording from 'more accurate' to 'comparable'.
  3. [§4.2; §1] The long-form training and evaluation protocol does not yet support the 'multiple hours' framing in the introduction. Long-form training data are concatenated neighboring GigaSpeech utterances of 10-15 s or 75-85 s, and long-form inference splits files into fixed non-overlapping chunks up to 40k frames (400 s) without modeling cross-chunk context. This protocol demonstrates behavior on 400-second segments, not on multi-hour audio as a whole. Please either add an evaluation on naturally long audio (e.g., full-file decoding of a long-form benchmark such as Earnings-21, or reporting WER for full multi-hour files with cross-chunk context handling) or revise the scope claims in the introduction and abstract to match the evaluated chunked setting.
minor comments (5)
  1. [§4.2; References] Reference [31] is incorrect: GigaSpeech is not Longformer. Please cite the actual GigaSpeech paper (e.g., Chen et al., Interspeech 2021) and move the Longformer reference to wherever it is actually discussed, if needed.
  2. [Table 3; §5.2] Throughput values are rounded to the nearest MPS, and the 44% figure is computed from rounded values (26 vs 18). Report unrounded values or exact relative differences to avoid giving a false impression of precision.
  3. [§4.3] Throughput is measured on a single audio file and a single batch size. State the characteristics of that file (duration, number of speakers, channel conditions) and, ideally, report results on several files, since MPS can vary with audio content. At minimum, acknowledge this limitation in the text.
  4. [§3.4; §3.5] Clarify whether the forward and backward outputs are averaged with fixed equal weights or learned weights, and describe what happens under DirDrop when one direction is dropped (is the output simply the retained direction?).
  5. [Table 4] The 'N' and 'Y' row labels in Table 1 and the 'Bi' row in Table 4 are not self-explanatory. Define the row labels explicitly in the captions (e.g., N = unidirectional, Y = bidirectional).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all central claims are empirical measurements against the external GigaSpeech benchmark, and the self-citations are not load-bearing.

full rationale

The paper's central claims are that bidirectional recurrent attention (RWKV) can match the accuracy of multi-head attention in a Conformer encoder, that long-form training improves accuracy, that Direction Dropout enables flexible decoding modes, and that throughput is higher. Every one of these claims is supported by measured word error rates and measured throughput on the external GigaSpeech dev/test partitions. There is no derivation chain in which an output quantity is defined in terms of an input quantity, and no parameter is fitted to a subset of data and then relabeled as a prediction. The LCA+GT baseline is taken from an external implementation and prior work; the RWKV and Mamba layers are standard published architectures; the DirDrop rate is a fixed hyperparameter. The self-citations in the paper are not load-bearing. Reference [16], co-authored by one of the present authors, is cited to motivate the transducer architecture and to explain how GigaSpeech dev/test files can be reconstructed by concatenating segments, but the benchmark itself is external and the accuracy results are measured, not imported from that citation. Reference [20], also by a present author, is cited only as an inspiration for structured dropout and does not supply any result used to derive the paper's conclusions. The throughput measurement in Section 4.3 is encoder-only ('we removed the transducer portion of the model for this part of the evaluation'), which is a legitimate scope limitation for a fair comparison, not a circular step, because the throughput numbers are still direct measurements of the components being compared. Overall, the paper is self-contained against an external benchmark, and no circularity is present.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The empirical claims rest on public benchmark data and standard ASR assumptions rather than on a derivation. The only hand-chosen quantity that directly shapes the headline trade-off is the Direction Dropout rate. The long-form training and chunked-decode protocols are assumptions not validated against true continuous recordings. No new physical or architectural entities are introduced beyond the training procedure itself.

free parameters (1)
  • Direction Dropout rate = 0.2 (20%)
    The DirDrop probability is set to 20% without a reported ablation. The accuracy and throughput values in Table 5 depend on this choice, though the qualitative conclusions likely hold across a range.
assumptions (4)
  • domain assumption GigaSpeech dev/test WER with the provided filtering rules is a valid and sufficient measure of ASR accuracy for the comparisons.
    Used for all tables; no confidence intervals, significance tests, or external benchmark validation are provided.
  • domain assumption Bidirectional recurrent attention obtained by averaging forward and backward passes captures the same context information as full multi-head attention.
    Section 3.4 states outputs of the backward pass are reversed and averaged with forward outputs; this merge function is not ablated.
  • domain assumption Chunked decoding with fixed non-overlapping segments approximates true long-form inference.
    Sections 4.2 and 4.3 assemble full-file WER from chunk outputs, so within-chunk context is limited to the chunk size, up to 400 seconds.
  • domain assumption Concatenating neighboring utterances reproduces the properties of natural long-form speech for training.
    Sections 3.3 and 4.2; this proxy may differ from real multi-hour audio in discourse structure and acoustic continuity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Accurate, fast, cheap: Choose three. Replacing Multi-Head-Attention with Bidirectional Recurrent Attention for Long-Form ASR." pith.science (2026). https://pith.science/paper/XQK3FBPN

@misc{pith2026250619761,
  author       = {Pith},
  title        = {Pith review of: Accurate, fast, cheap: Choose three. Replacing Multi-Head-Attention with Bidirectional Recurrent Attention for Long-Form ASR},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XQK3FBPN}},
  note         = {Machine review of arXiv:2506.19761}
}
read the original abstract

Long-form speech recognition is an application area of increasing research focus. ASR models based on multi-head attention (MHA) are ill-suited to long-form ASR because of their quadratic complexity in sequence length. We build on recent work that has investigated linear complexity recurrent attention (RA) layers for ASR. We find that bidirectional RA layers can match the accuracy of MHA for both short- and long-form applications. We present a strong limited-context attention (LCA) baseline, and show that RA layers are just as accurate while being more efficient. We develop a long-form training paradigm which further improves RA performance, leading to better accuracy than LCA with 44% higher throughput. We also present Direction Dropout, a novel regularization method that improves accuracy, provides fine-grained control of the accuracy/throughput trade-off of bidirectional RA, and enables a new alternating directions decoding mode with even higher throughput.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 28 canonical work pages

  1. [1]

    However, Transformers, especially their multi-head attention (MHA) component, are ill-suited for long- form ASR due to quadratic time/memory complexity in se- quence length

    Introduction Improved ASR accuracy enables long-form audio applications (e.g., multiple hours). However, Transformers, especially their multi-head attention (MHA) component, are ill-suited for long- form ASR due to quadratic time/memory complexity in se- quence length. Decoding long audio in chunks (e.g., Whisper

    work page

  2. [2]

    Recently, several layer types have been introduced that mimic the properties of MHA while having linear time and memory complexity in sequence length

    uses 30s segments) loses crucial full-audio context. Recently, several layer types have been introduced that mimic the properties of MHA while having linear time and memory complexity in sequence length. These layers achieve linear complexity through the use of recurrent-style computa- tions that depend only on a hidden state containing the accumu- lated ...

    work page

  3. [3]

    Bidi- rectional RWKV-Conformer is more efficient than standard Conformer and limited-context attention with global tokens

    Bidirectional RA layers (Mamba-2, RWKV v6) are com- petitive with MHA in Conformer encoders for ASR. Bidi- rectional RWKV-Conformer is more efficient than standard Conformer and limited-context attention with global tokens

    work page

  4. [4]

    Finally, Alternating direction decoding with DirDrop nears bidirectional decoding accuracy at unidirectional cost. This offers new insights into bidirectional information propaga- tion: it not only shows the importance of merging informa- tion from the past and future, but also reveals that it is not necessary to process both directions within each layer

    work page

  5. [5]

    With additional long-form fine-tuning, the bidirectional RWKV-Conformer is more accurate than limited-context MHA with 44% higher throughput, while the unidirectional RWKV-Conformer is as accurate as limited-context MHA with 72% higher throughput

    work page

  6. [6]

    for long-form ASR at similar accuracy

    work page

  7. [7]

    DirDrop regularization closes the bidirectional RA vs. MHA gap. DirDrop-trained bidirectional models achieve unidirec- tional inference accuracy similar to unidirectional models, unlike standard bidirectional models which fail catastrophi- cally in unidirectional inference. In this way, our approach enables one model to be used flexibly for both offline a...

    work page

  8. [8]

    Accurate, fast, cheap: Choose three. Replacing Multi-Head-Attention with Bidirectional Recurrent Attention for Long-Form ASR

    Related Work Different types of recurrent attention (RA) layers have been in- troduced. These broadly fall into two classes: RWKV [2] and state-space models (SSM), which include S4 [7], H3 [8], and Mamba [4]. For this paper, we compare RWKV v6 [3] and Mamba-2 [5]. For more details, see Section 3.1. RA layers have previously been applied to ASR, with promi...

    work page Pith review arXiv 2025

Show all 44 references
  1. [9]

    For all models in this paper, we maintain the same overall architecture, the same Conformer layer structure, and the same parameters for all convolutional and linear layers

    Methods Our models use the Conformer-Transducer architecture that consists of a Conformer encoder [25] and a Transducer decoder [26]. For all models in this paper, we maintain the same overall architecture, the same Conformer layer structure, and the same parameters for all co...

  2. [10]

    SF” (short-form), we performed training on the stan- dard segments released with the dataset, which have an average length of 4.4 seconds. For long-form training (“LF

    Experiment Details 4.1. Model Architecture Our experiments are performed using the WeNet framework (see [29], [30]). All ASR models used in this paper have the same basic architecture: a Conformer encoder, a CTC projec- tion layer at the encoder output, a bidirectional self-at...

  3. [11]

    Short-Form ASR Table 1: MHA vs

    Results and Discussion 5.1. Short-Form ASR Table 1: MHA vs. RA. WER (%) on short-form segments. Bidir . MHA Mamba-2 RWKV N - 11.2/11.4 11.2/11.3 Y 10.9/10.9 11.0/11.0 11.0/11.0 Table 1 shows results for short-form decoding. When used unidirectionally, Mamba and RWKV underperfo...

  4. [12]

    Our bi-RWKV- Conformer matches or exceeds MHA and limited-context MHA accuracy, while processing more audio per second

    Conclusions and Future Work In this paper, we showed that RA layers can effectively replace MHA in long-form ASR models, achieving competitive accu- racy with significant throughput advantages. Our bi-RWKV- Conformer matches or exceeds MHA and limited-context MHA accuracy, whi...

  5. [13]

    Robust speech recognition via large- scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large- scale weak supervision,” 2022. [Online]. Available: https: //arxiv.org/abs/2212.04356

    2022 arXiv

  6. [14]

    RWKV: Reinventing RNNs for the transformer era,

    B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, M. Grella et al. , “RWKV: Reinventing RNNs for the transformer era,” arXiv preprint arXiv:2305.13048, 2023

    2023 arXiv

  7. [15]

    Eagle and Finch: RWKV with matrix-valued states and dynamic recur- rence,

    B. Peng, D. Goldstein, Q. Anthony, A. Albalak, E. Alcaide, S. Bi- derman, E. Cheah, X. Du, T. Ferdinan, H. Hou et al. , “Eagle and Finch: RWKV with matrix-valued states and dynamic recur- rence,” arXiv preprint arXiv:2404.05892, 2024

    2024 arXiv

  8. [16]

    Mamba: Linear-time sequence modeling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  9. [17]

    Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality,

    T. Dao and A. Gu, “Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality,” arXiv preprint arXiv:2405.21060, 2024

    2024 arXiv

  10. [18]

    Investigating end-to-end ASR architectures for long form audio transcription,

    N. R. Koluguri, S. Kriman, G. Zelenfroind, S. Majumdar, D. Rekesh, V . Noroozi, J. Balam, and B. Ginsburg, “Investigating end-to-end ASR architectures for long form audio transcription,” in ICASSP 2024-2024 IEEE International Conference on Acous- tics, Speech and Signal Proces...

    2024

  11. [19]

    Efficiently modeling long sequences with structured state spaces,

    A. Gu, K. Goel, and C. R ´e, “Efficiently modeling long sequences with structured state spaces,” arXiv preprint arXiv:2111.00396 , 2021

    2021 arXiv

  12. [20]

    Hungry hungry hippos: Towards language modeling with state space models,

    D. Y . Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. R´e, “Hungry hungry hippos: Towards language modeling with state space models,” arXiv preprint arXiv:2212.14052, 2022

    2022 arXiv

  13. [21]

    Multi- head state space model for speech recognition,

    Y . Fathullah, C. Wu, Y . Shangguan, J. Jia, W. Xiong, J. Ma- hadeokar, C. Liu, Y . Shi, O. Kalinli, M. Seltzer et al. , “Multi- head state space model for speech recognition,” arXiv preprint arXiv:2305.12498, 2023

    2023 arXiv

  14. [22]

    Mamba-based decoder-only approach with bidirectional speech modeling for speech recognition,

    Y . Masuyama, K. Miyazaki, and M. Murata, “Mamba-based decoder-only approach with bidirectional speech modeling for speech recognition,” arXiv preprint arXiv:2411.06968, 2024

    2024 arXiv

  15. [23]

    Structured state space decoder for speech recognition and synthesis,

    K. Miyazaki, M. Murata, and T. Koriyama, “Structured state space decoder for speech recognition and synthesis,” in ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  16. [24]

    Exploring RWKV for memory efficient and low latency streaming ASR,

    K. An and S. Zhang, “Exploring RWKV for memory efficient and low latency streaming ASR,” arXiv preprint arXiv:2309.14758 , 2023

    2023 arXiv

  17. [25]

    Augmenting conformers with structured state-space sequence models for online speech recognition,

    H. Shan, A. Gu, Z. Meng, W. Wang, K. Choromanski, and T. Sainath, “Augmenting conformers with structured state-space sequence models for online speech recognition,” in ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, p...

    2024

  18. [26]

    Efficient and robust long-form speech recognition with hybrid h3-conformer,

    T. Honda, S. Sakai, and T. Kawahara, “Efficient and robust long-form speech recognition with hybrid h3-conformer,” arXiv preprint arXiv:2410.04159, 2024

    2024 arXiv

  19. [27]

    Exploring the ca- pability of Mamba in speech applications,

    K. Miyazaki, Y . Masuyama, and M. Murata, “Exploring the ca- pability of Mamba in speech applications,” in Interspeech 2024, 2024, pp. 237–241

    2024

  20. [28]

    Updated corpora and benchmarks for long-form speech recognition,

    J. D. Fox, D. Raj, N. Delworth, Q. McNamara, C. Miller, and M. Jett´e, “Updated corpora and benchmarks for long-form speech recognition,” in ICASSP 2024-2024 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 13 246–13 250

    2024

  21. [29]

    Learning with marginalized corrupted features,

    L. Van Der Maaten, M. Chen, S. Tyree, and K. Q. Weinberger, “Learning with marginalized corrupted features,” in Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28 , ser. ICML’13. JMLR.org, 2013, p. I–410–I–418

    2013

  22. [30]

    Dropout training as adaptive regularization,

    S. Wager, S. Wang, and P. Liang, “Dropout training as adaptive regularization,” in Proceedings of the 27th International Confer- ence on Neural Information Processing Systems - Volume 1 , ser. NIPS’13. Red Hook, NY , USA: Curran Associates Inc., 2013, p. 351–359

    2013

  23. [31]

    Learning with pseudo- ensembles,

    P. Bachman, O. Alsharif, and D. Precup, “Learning with pseudo- ensembles,” in Proceedings of the 28th International Confer- ence on Neural Information Processing Systems - Volume 2 , ser. NIPS’14. Cambridge, MA, USA: MIT Press, 2014, p. 3365–3373

    2014

  24. [32]

    Structured reg- ularizer for neural higher-order sequence models,

    M. Ratajczak, S. Tschiatschek, and F. Pernkopf, “Structured reg- ularizer for neural higher-order sequence models,” in Machine Learning and Knowledge Discovery in Databases , A. Appice, P. P. Rodrigues, V . Santos Costa, C. Soares, J. Gama, and A. Jorge, Eds. Cham: Springer In...

    2015

  25. [33]

    Improving neural networks by preventing co-adaptation of feature detectors,

    G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, “Improving neural networks by preventing co-adaptation of feature detectors,” 2012. [Online]. Available: https://arxiv.org/abs/1207.0580

    2012 arXiv

  26. [34]

    Regularization of neural networks using dropconnect,

    L. Wan, M. Zeiler, S. Zhang, Y . Le Cun, and R. Fergus, “Regularization of neural networks using dropconnect,” in Proceedings of the 30th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, S. Dasgupta and D. McAllester, Eds., vol. 28, ...

    2013

  27. [35]

    Reducing transformer depth on demand with structured dropout,

    A. Fan, E. Grave, and A. Joulin, “Reducing transformer depth on demand with structured dropout,” in International Conference on Learning Representations

  28. [36]

    Dynamic encoder transducer: A flexible solution for trading off accuracy for latency,

    Y . Shi, V . Nagaraja, C. Wu, J. Mahadeokar, D. Le, R. Prab- havalkar, A. Xiao, C.-F. Yeh, J. Chan, C. Fuegenet al., “Dynamic encoder transducer: A flexible solution for trading off accuracy for latency,”arXiv preprint arXiv:2104.02176, 2021

    2021 arXiv

  29. [37]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in Interspeech 2020, 2020, pp. 5036–5040

    2020

  30. [38]

    Sequence transduction with recurrent neural networks,

    A. Graves, “Sequence transduction with recurrent neural networks,” 2012. [Online]. Available: https://arxiv.org/abs/1211. 3711

    2012

  31. [39]

    Fast conformer with linearly scalable attention for efficient speech recognition,

    D. Rekesh, N. R. Koluguri, S. Kriman, S. Majumdar, V . Noroozi, H. Huang, O. Hrinchuk, K. Puvvada, A. Kumar, J. Balam et al., “Fast conformer with linearly scalable attention for efficient speech recognition,” in2023 IEEE Automatic Speech Recognition and Understanding Workshop...

    2023

  32. [40]

    Bidirectional recurrent neu- ral networks,

    M. Schuster and K. K. Paliwal, “Bidirectional recurrent neu- ral networks,” IEEE transactions on Signal Processing , vol. 45, no. 11, pp. 2673–2681, 1997

    1997

  33. [41]

    WeNet: Production oriented stream- ing and non-streaming end-to-end speech recognition toolkit,

    Z. Yao, D. Wu, X. Wang, B. Zhang, F. Yu, C. Yang, Z. Peng, X. Chen, L. Xie, and X. Lei, “WeNet: Production oriented stream- ing and non-streaming end-to-end speech recognition toolkit,” in Proc. Interspeech. Brno, Czech Republic: IEEE, 2021

    2021

  34. [42]

    WeNet 2.0: More productive end-to- end speech recognition toolkit,

    B. Zhang, D. Wu, Z. Peng, X. Song, Z. Yao, H. Lv, L. Xie, C. Yang, F. Pan, and J. Niu, “WeNet 2.0: More productive end-to- end speech recognition toolkit,”arXiv preprint arXiv:2203.15455, 2022

    2022 arXiv

  35. [43]

    Longformer: The long-document transformer,

    I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” CoRR, vol. abs/2004.05150, 2020. [Online]. Available: https://arxiv.org/abs/2004.05150

    2004 arXiv

  36. [44]

    Earnings-21: A practical benchmark for ASR in the wild,

    M. Del Rio, N. Delworth, R. Westerman, M. Huang, N. Bhandari, J. Palakapilly, Q. McNamara, J. Dong, P. ˙Zelasko, and M. Jett ´e, “Earnings-21: A practical benchmark for ASR in the wild,” in Interspeech 2021, 2021, p. 3465–3469

    2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.