Pith. sign in

REVIEW 4 major objections 5 minor 63 references

Mixture of LoRA Experts with Multi-Modal and Multi-Granularity LLM Generative Error Correction for Accented Speech Recognition

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Per-accent LLM error-correction experts combined by hierarchical routing cut the average word error rate on nine English accents from 6.34% to 2.07%, a 67.35% relative drop over vanilla Whisper-large-v3.

desk verdict A sensible, well-ablated ASR error-correction pipeline whose headline 67.35% WER gain is unverifiable until the authors disclose the train/test split. read the letter →

arxiv 2507.09116 v3 pith:HN35ZJRP submitted 2025-07-12 cs.SD eess.AS

Bingshen Mu , Kun Wei , Pengcheng Guo , Lei Xie This is my paper
classification cs.SDeess.AS
keywords generativeerrorcorrectionaccentedspeechrecognitionLoRAexpertsmixtureofhierarchicalroutingdynamicthresholdsmulti-granularityphonemehypotheses
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to fix one of the weak spots of generative error correction (GER) for speech: accented speech, where an LLM that only sees the ASR's word hypotheses lacks the pronunciation cues that carry the accent's deviation from standard speech. It argues that two kinds of information must reach the LLM, speech-derived pronunciation and word-level semantics, and that one model trained on all accents at once is weaker than per-accent specialists combined by a routing layer. Its proposed pipeline does this in three stages: multi-modal GER feeds Whisper's speech embeddings plus the 1-best hypothesis to a LoRA-tuned LLM, per-accent versions of that model are merged by HDMoLE (hierarchical routing plus dynamic thresholds over nine frozen mono-accent LoRA experts), and a two-pass multi-granularity GER refines the output using N-best word and IPA phoneme hypotheses. On a combined dataset of nine accented-English varieties, the full pipeline lowers average WER from 6.34% (vanilla Whisper-large-v3) to 2.07%, a relative reduction of 67.35%. The paper also shows each stage earns its keep in ablations, and that word-level hypotheses matter more than phoneme-level ones, with the best results from combining five of each.

What carries the argument

The load-bearing machinery is the HDMoLE layer, an equation for linearly combining frozen LoRA experts: $H^{out} = W_0 H^{in} + \frac{\alpha}{r} \sum_{j=1}^N P_a^j B_j A_j H^{in} + b$, where $P_a^j$ is the adapted weight of the $j$-th mono-accent LoRA expert, equal to the sum of a global weight (from the frozen accent-recognition router, thresholded by $\tau_g$) and a local weight (from the layer's trainable linear router, thresholded by $\tau_l$), with both thresholds trainable and initialized at $1/N$. Around this layer sit the other two objects: multi-modal GER, which supplies the input representation (speech embeddings, 1-best hypothesis, and prompt embeddings concatenated for the LLM) and the training scheme that produces the mono-accent experts; and multi-granularity GER, which supplies the refined input (N-best word hypotheses plus their IPA phoneme transcriptions, concatenated) to the second LLM. Together they let a frozen Whisper encoder and frozen LLM adapt to accent diversity by re-weighting nine LoRA experts per layer instead of fine-tuning the whole model.

What would settle it

Re-run the HDMoLE plus multi-granularity pipeline on a strictly speaker-disjoint subset of the same nine accents (train mono-accent experts on one set of speakers, test on held-out speakers, and keep the test utterances out of every training stage); if the average WER lands well above 2.07%, say above 3.5%, then the 67.35% relative reduction does not hold for unseen speakers and the reported number reflects training/test overlap rather than accent-aware generalization.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a hypothesis-to-transcription LLM corrector can be made accent-aware by feeding it pronunciation signals at two granularities and by treating accents as domains for a mixture of experts. The multi-modal GER model takes the Whisper encoder's speech embeddings (projected by a trainable connector) together with the 1-best Whisper hypothesis and, through LoRA fine-tuning of LLaMA-3.2-3B, learns to predict the correct transcription; training it separately on each of nine accents yields nine mono-accent LoRA experts. HDMoLE then plugs these frozen experts into a single GER model, using a pre-trained accent-recognition model as a global router and a trainable linear layer in each LLM layer as a local router, with two trainable thresholds per layer that replace the static Top-K selection; the adapted weights combine the global and local scores. Finally, multi-granularity GER beam-decodes the HDMoLE model, converts the N-best word hypotheses to IPA phonemes with the phonemizer tool, and feeds both word and phoneme hypotheses to a LoRA-tuned LLaMA-3.1-8B to produce the final transcription. The result is an average WER of 2.07% across the nine accents versus 6.34% for vanilla Whisper-large-v3 (a 67.35% relative reduction), with intermediate averages of 3.20% for multi-modal GER, 2.92% with mono-accent experts, and 2.49% for HDMoLE before the final multi-granularity stage.

Load-bearing premise

The reported 2.07% average WER assumes that the recorded speech used to test each accent comes from speakers and utterances the mono-accent LoRA experts never saw during training, and the paper does not describe how the nine-accent corpus was split into training and test sets, so speaker or utterance overlap would inflate the gains.

Editorial extensions

If this is right

  • The same frozen Whisper-large-v3 plus a 3B/8B LLM reaches 2.07% average WER on nine accents, below large audio-language models Qwen2-Audio (5.81%) and FireRedASR-LLM-L (7.00%) on the same test set.
  • Only a small number of parameters are trained: HDMoLE adds about 6.7M trainable parameters on top of the frozen LoRA experts, keeping the per-accent experts intact, so the mixture can be applied without retraining the experts.
  • Static Top-K selection is a bottleneck: per-layer dynamic thresholds let early LLM layers activate more experts (about 7 to 9) and deeper layers fewer, a flexibility that static Top-K cannot reproduce.
  • Word-level hypotheses carry the semantic information the LLM needs most: they outperform phoneme-level hypotheses at every N, and combining five word-level and five phoneme-level hypotheses gives the best result (2.07%).
  • The gains are additive across the three stages: 3.20% (multi-modal GER), 2.92% (mono-accent experts), 2.49% (HDMoLE), and 2.07% (multi-granularity GER), so each proposed component has an observable, positive effect.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the headline 67.35% hides wide per-accent spread (US at 0.54%, JP at 2.71%, ZA at 8.62%); for deployment, the useful question is not the average but whether the pipeline transfers to a new, unseen accent, which the global router, trained only on the nine labeled accents, may not do.
  • Editorial inference: the phoneme-level inputs come from the phonemizer grapheme-to-phoneme tool applied to the ASR's word hypotheses; any tool failures on accented prosody (elision, substitution, linking) would cap the multi-granularity stage's benefit, so a testable extension is to compare against forced-alignment phonemes from the speech itself.
  • Editorial inference: at inference the pipeline runs an LLM on every utterance (13.70 G FLOPs per frame or token for HDMoLE, 15.35 G for multi-granularity GER, versus 8.29 G for the Whisper baseline), so real-time cost is roughly double the baseline plus the second LLM pass; distilling the pipeline into a single small model is the natural pressure-test for deployment.
  • Editorial inference: the central claim is conditional on the evaluation being speaker-independent and utterance-disjoint, since mono-accent LoRA experts fine-tuned on the same speakers that later appear in the test set could memorize them; the paper does not describe the train/test split, so this condition is unverified.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a three-stage pipeline for accented English ASR: multi-modal GER (Whisper speech embeddings plus 1-best hypotheses) to train mono-accent LoRA experts; HDMoLE to combine those experts via hierarchical routing with a pre-trained accent-recognition global router and dynamic thresholds; and multi-granularity GER (N-best word-level and phoneme-level hypotheses) to produce final transcriptions. On a combined 9-accent English dataset, the full pipeline is reported to reduce average WER from 6.34% for vanilla Whisper-large-v3 to 2.07%, a 67.35% relative reduction. The LoRA and MoE formulations are standard, and the contribution is primarily empirical.

Significance. If the headline results survive a proper speaker-disjoint evaluation, this would be a useful demonstration that GER with accent-specialized LoRA experts and multi-granularity inputs can substantially improve accented ASR. The paper includes extensive ablations, parameter/FLOP accounting, and comparisons against several MoLE variants. The main value is therefore empirical rather than theoretical. However, as written, the evaluation protocol is not established: no train/dev/test split is described, the 'Vanilla GER' baseline is defined using fully fine-tuned Whisper hypotheses rather than vanilla Whisper hypotheses, and the reported AVG WER values are internally inconsistent with the per-accent columns. These issues directly affect the 67.35% headline claim and must be resolved before the results can be verified.

major comments (4)
  1. [§III.A, §IV.B, §V (Tables II–VII)] No train/dev/test partition is described. Stage 1 and Stage 2 are said to train the connector and LoRA on 'all accented speech data,' and §IV.B states that 1-best hypotheses are produced by vanilla Whisper on 'all speech data in our combined dataset,' while results are reported on 'our combined multi-accent English test sets.' Because there is no statement that test speakers or utterances are disjoint from training, the mono-accent LoRA experts and the later HDMoLE and multi-granularity stages could be evaluated on data they were trained on, which would make the reported 6.34%→2.07% WER reduction (67.35% relative) reflect memorization rather than generalization to unseen accented speech. The authors must specify the split, report the number of speakers/utterances per split, and confirm that there is no speaker or utterance overlap.
  2. [§V.A, Table II] The 'Vanilla GER' baseline is not faithful to the definition of GER. The text says the authors 'reproduce GER by fine-tuning the LLM with LoRA to learn the mapping between the N-best hypotheses generated by the fully fine-tuned Whisper and the ground truth transcriptions.' A true vanilla GER baseline should use N-best hypotheses from the vanilla Whisper-large-v3 model, not from a fully fine-tuned model, since the latter already incorporates in-domain training data and makes the baseline stronger. Without this distinction, the comparison in Table II cannot attribute the observed gains to the proposed multi-modal/multi-granularity GER inputs rather than to the stronger ASR hypotheses. Please add a vanilla GER run with vanilla Whisper N-best hypotheses and, separately, a controlled variant with fully fine-tuned Whisper N-best hypotheses.
  3. [Table II] The reported AVG WER does not match the arithmetic mean of the nine per-accent columns. For example, in the final HDMoLE + multi-granularity row the per-accent values (1.22, 0.92, 3.44, 1.11, 2.91, 2.71, 0.54, 2.23, 8.62) average to 2.63, not the stated 2.07; similarly, the fully fine-tuned Whisper row averages to 4.21, not 3.29. Because the headline 67.35% relative reduction is computed from AVG, this inconsistency must be resolved, e.g., by correcting column alignment or by clearly stating the weighting used to compute AVG.
  4. [Tables III, V, VI, VII] No error bars, repeated-seed results, or significance tests are reported. Several central comparisons are small (e.g., Table III: utterance-level 2.57 vs. frame-level 2.49; Table V: Top-1 3.24 vs. Top-3 3.23), and single-run WER differences of this size cannot be distinguished from run-to-run variance. Please report at least 2–3 seeds with standard deviations, or a significance test, for the main comparisons and ablations.
minor comments (5)
  1. [§IV.B] The pre-trained AR model used as the global router is described only by architecture and accuracy; its training data, split, label set, and training objective should be specified, since Table VI shows that global routing accuracy is load-bearing for HDMoLE.
  2. [§III.A, Eq. (10)] The concatenation order is written as Concat(H_hyp, H_pro, H_enc), but the prose describes 'speech embeddings, 1-best hypothesis embeddings, and prompt embeddings'; please clarify the intended input order.
  3. [§III.C] 'impressed' should be 'expressed' in Eqs. (17)–(18), and 'thethresholdτl' is missing a space in the sentence following Eq. (18).
  4. [§II.C] 'during trainin' should be 'during training' in the paragraph after Eq. (4).
  5. [Overall] The manuscript does not state whether code or data will be released; given the evaluation-protocol concerns, releasing the exact split and evaluation scripts would materially improve verifiability.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation-level circularity found: the WER claims are supervised empirical results, not quantities derived from the model's own definitions; the main caveats are minor self-citations and an undescribed train/test split, which is a protocol risk rather than a circular reduction.

full rationale

The paper's central claim, a 67.35% relative WER reduction from 6.34% to 2.07%, is an experimentally measured benchmark outcome rather than a quantity derived from the model equations. Each proposed component is specified with explicit equations: multi-modal GER (Eqs. 7-13), HDMoLE hierarchical routing and dynamic thresholds (Eqs. 14-21), and multi-granularity GER (Eqs. 22-27). These components are validated through comparisons with external baselines and ablations in Tables II-VII, so the central empirical claim does not reduce to a definition, a fitted parameter, or a self-citation chain. HDMoLE is attributed to the authors' prior work [38], but the present paper re-specifies the routing and threshold equations, trains the experts and routers on the described data, and compares against MOELoRA, MoRAL, MoLE, LoRAMoE, and MoA; the self-citation is therefore descriptive rather than load-bearing. Similarly, MMGER [7] is contrasted rather than relied on as proof. No 'uniqueness theorem' is invoked, and no known result is renamed as a new contribution. The principal caveat is external validity, not circularity: Section IV.A and IV.B generate 1-best and N-best hypotheses on 'all speech data in our combined dataset' and never state a speaker- or utterance-disjoint train/test partition, so if the test utterances overlap training, the measured gains could partly reflect memorization. That is a contamination and reproducibility risk that should be weighed in assessing the empirical claim, but it does not make the derivation circular. The score of 2 reflects the minor self-citation of HDMoLE as the origin of a central component plus the unverified evaluation protocol, not an identified circular step.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on standard supervised-learning assumptions plus the domain-specific assumption that the phonemizer and the accent-recognition router are accurate. The absence of a described data split is the most consequential unstated assumption, because the headline WER numbers are only meaningful if the test utterances are speaker-disjoint from training. No new physical or learned entities are introduced beyond the HDMoLE gating mechanism, which is an algorithmic design rather than a postulated entity.

free parameters (3)
  • Number of accents/experts N=9 = 9
    Chosen by dataset composition; the HDMoLE architecture has one LoRA expert per accent.
  • Initial dynamic thresholds 1/N = 1/9
    Hand-initialized to guarantee at least one selected expert; thresholds are then learned during HDMoLE training.
  • LoRA ranks and alphas = r=32/a=8 for multi-modal GER; r=64/a=16 for multi-granularity GER
    Standard LoRA hyperparameters chosen without a described tuning procedure.
assumptions (4)
  • domain assumption Phonemizer toolkit produces accurate IPA phoneme hypotheses for accented English.
    Multi-granularity GER relies on phoneme-level hypotheses generated by phonemizer; errors in phonemization would propagate to the LLM input.
  • domain assumption The pre-trained accent recognition model provides reliable global routing weights.
    Ablation (Table VI) shows routing degrades with low AR accuracy; the final system assumes a 90.51% AR model, whose training data is not specified.
  • domain assumption The three-stage training preserves mono-accent specialization in LoRA experts.
    Stage 3 fine-tunes each expert on one accent; the paper provides no analysis of catastrophic forgetting or expert contamination.
  • domain assumption The combined dataset has no speaker overlap between training and test partitions.
    No train/dev/test split is described; the central WER numbers assume the test set measures generalization to unseen speakers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mixture of LoRA Experts with Multi-Modal and Multi-Granularity LLM Generative Error Correction for Accented Speech Recognition." pith.science (2026). https://pith.science/paper/HN35ZJRP

@misc{pith2026250709116,
  author       = {Pith},
  title        = {Pith review of: Mixture of LoRA Experts with Multi-Modal and Multi-Granularity LLM Generative Error Correction for Accented Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HN35ZJRP}},
  note         = {Machine review of arXiv:2507.09116}
}
read the original abstract

Despite improvements in automatic speech recognition, performance drops with accented speech. Generative error correction (GER) leverages the linguistic knowledge of large language models (LLMs), outperforming typical language model methods. However, it lacks specificity in accented speech scenarios. Accents represent deviations from standard pronunciation, making multi-granularity pronunciation and semantic information essential for accented speech recognition. Moreover, accents exhibit considerable diversity, with each accent possessing distinct characteristics. In this study, we leverage GER to improve transcription accuracy by addressing the two primary features. We propose the multi-modal GER, which integrates pronunciation information from the speech modality, and the multi-granularity GER, which incorporates fine-grained phoneme-level pronunciation information. These methods enable the LLM to utilize the pronunciation information of accented speech and the semantic information from word-level hypotheses for accurate transcription predictions through low-rank adaptation (LoRA) fine-tuning. We employ a three-stage strategy to train separate multi-modal GER models for each accent to obtain mono-accent LoRA experts. By adopting our proposed HDMoLE method, which incorporates hierarchical routing and dynamic thresholds within the mixture of LoRA experts, we effectively merge mono-accent LoRA experts within a single multi-modal GER to overcome accent diversity challenges. Furthermore, multi-granularity GER leverages N-best word-level and phoneme-level hypotheses from the HDMoLE model to predict final transcriptions. Experiments on a multi-accent English dataset show that our methods reduce word error rate by 67.35% compared to the baseline vanilla Whisper-large-v3 model.

Figures

Figures reproduced from arXiv: 2507.09116 by the authors.

Figure 1
Figure 1. Overview of vanilla GER and our proposed multi-modal GER. In this [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Overview of the three-stage training strategy for obtaining mono-accent LoRA experts from multi-modal GER. In this figure, “Encoder” means [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overview of the proposed HDMoLE for LoRA experts combination. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Overview of the multi-granularity GER. The N-best word-level and [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The average number of LoRA experts selected in the seven modules [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: T-SNE visualization of the local weights with or without global [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 49 canonical work pages

  1. [1]

    Sequence Transduction with Recurrent Neural Networks,

    Alex Graves, “Sequence Transduction with Recurrent Neural Networks,” arXiv preprint arXiv:1211.3711 , 2012

    arXiv 2012

  2. [2]

    End-to-end attention-based large vocabulary speech recognition,

    Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio, “End-to-end attention-based large vocabulary speech recognition,” in Proc. ICASSP, 2016, pp. 4945–4949

    work page 2016

  3. [3]

    Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,

    William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in Proc. ICASSP, 2016, pp. 4960–4964

    work page 2016

  4. [4]

    CIF: Continuous Integrate-And-Fire for End- To-End Speech Recognition,

    Linhao Dong and Bo Xu, “CIF: Continuous Integrate-And-Fire for End- To-End Speech Recognition,” in Proc. ICASSP, 2020, pp. 6079–6083

    work page 2020

  5. [5]

    Hybrid CTC/Attention Architecture for End-to- End Speech Recognition,

    Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi, “Hybrid CTC/Attention Architecture for End-to- End Speech Recognition,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017

    work page 2017

  6. [6]

    Automatic Channel Selection and Spatial Feature Integration for Multi-Channel Speech Recognition Across Various Array Topologies,

    Bingshen Mu, Pengcheng Guo, Dake Guo, Pan Zhou, Wei Chen, and Lei Xie, “Automatic Channel Selection and Spatial Feature Integration for Multi-Channel Speech Recognition Across Various Array Topologies,” in Proc. ICASSP, 2024, pp. 11396–11400

    work page 2024

  7. [7]

    MMGER: Multi-Modal and Multi-Granularity Generative Error Correction With LLM for Joint Accent and Speech Recognition,

    Bingshen Mu, Xucheng Wan, Naijun Zheng, Huan Zhou, and Lei Xie, “MMGER: Multi-Modal and Multi-Granularity Generative Error Correction With LLM for Joint Accent and Speech Recognition,” IEEE Signal Processing Letters , vol. 31, pp. 1940–1944, 2024

    work page 1940

  8. [8]

    An Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model,

    Anjuli Kannan, Yonghui Wu, Patrick Nguyen, Tara N Sainath, Zhijeng Chen, and Rohit Prabhavalkar, “An Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model,” in Proc. ICASSP, 2018, pp. 5824–5828

    work page 2018

Show all 63 references
  1. [9]

    De- liberation Model Based Two-Pass End-To-End Speech Recognition,

    Ke Hu, Tara N Sainath, Ruoming Pang, and Rohit Prabhavalkar, “De- liberation Model Based Two-Pass End-To-End Speech Recognition,” in Proc. ICASSP, 2020, pp. 7799–7803

    2020

  2. [10]

    Component Fusion: Learning Replaceable Language Model Component for End-to-end Speech Recognition System,

    Changhao Shan, Chao Weng, Guangsen Wang, Dan Su, Min Luo, Dong Yu, and Lei Xie, “Component Fusion: Learning Replaceable Language Model Component for End-to-end Speech Recognition System,” in Proc. ICASSP, 2019, pp. 5361–5635

    2019

  3. [11]

    Cold fusion: Training Seq2seq Models Together with Language Mod- els,

    Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates, “Cold fusion: Training Seq2seq Models Together with Language Mod- els,” in Proc. ICLR, 2018

    2018

  4. [12]

    FastCor- rect: Fast Error Correction with Edit Alignment for Automatic Speech Recognition,

    Yichong Leng, Xu Tan, Linchen Zhu, Jin Xu, Renqian Luo, Linquan Liu, Tao Qin, Xiangyang Li, Edward Lin, and Tie-Yan Liu, “FastCor- rect: Fast Error Correction with Edit Alignment for Automatic Speech Recognition,” in Proc. NeurIPS, 2021, pp. 21708–21719

    2021

  5. [13]

    ASR Error Correction and Domain Adaptation Using Machine Translation,

    Anirudh Mani, Shruti Palaskar, Nimshi Venkat Meripo, Sandeep Konam, and Florian Metze, “ASR Error Correction and Domain Adaptation Using Machine Translation,” in Proc. ICASSP, 2020, pp. 6344–6348

    2020

  6. [14]

    Softcorrect: Error correction with soft detection for automatic speech recognition,

    Yichong Leng, Xu Tan, Wenjie Liu, Kaitao Song, Rui Wang, Xiang-Yang Li, Tao Qin, Ed Lin, and Tie-Yan Liu, “Softcorrect: Error correction with soft detection for automatic speech recognition,” in Proc. AAAI, 2023, pp. 13034–13042

    2023

  7. [15]

    N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space,

    Rao Ma, Mark JF Gales, Kate M Knill, and Mengjie Qian, “N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space,” in Proc. Interspeech, 2023, pp. 3267– 3271

    2023

  8. [16]

    GPT-4 Technical Report,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al., “GPT-4 Technical Report,” arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  9. [17]

    PaLM: Scaling Language Modeling with Pathways,

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al., “PaLM: Scaling Language Modeling with Pathways,” Journal of Machine Learning Research , vol. 24, no. 240, pp....

    2023

  10. [18]

    PaLM 2 Technical Report,

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al., “PaLM 2 Technical Report,” arXiv preprint arXiv:2305.10403, 2023

    2023 arXiv

  11. [19]

    LLaMA: Open and Efficient Foundation Language Models,

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, et al., “LLaMA: Open and Efficient Foundation Language Models,” arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  12. [20]

    Llama 2: Open Foundation and Fine-Tuned Chat Models,

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Alma- hairi, et al., “Llama 2: Open Foundation and Fine-Tuned Chat Models,” arXiv preprint arXiv:2307.09288 , 2023

    2023 arXiv

  13. [21]

    HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models,

    Chen Chen, Yuchen Hu, Chao-Han Huck Yang, Sabato Marco Sinis- calchi, Pin-Yu Chen, and Eng-Siong Chng, “HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models,” in Proc. NeurIPS, 2024

    2024

  14. [22]

    Large Language Models are Efficient Learners of Noise-Robust Speech Recognition,

    Yuchen Hu, Chen Chen, Chao-Han Huck Yang, Ruizhe Li, Chao Zhang, Pin-Yu Chen, and EnSiong Chng, “Large Language Models are Efficient Learners of Noise-Robust Speech Recognition,” in Proc. ICLR, 2024

    2024

  15. [23]

    Everyone has an accent,

    Nina Markl and Catherine Lai, “Everyone has an accent,” in Proc. Interspeech, 2023, pp. 4424–4427

    2023

  16. [24]

    Decoupling and Interacting Multi-Task Learning Network for Joint JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13 Speech and Accent Recognition,

    Qijie Shao, Pengcheng Guo, Jinghao Yan, Pengfei Hu, and Lei Xie, “Decoupling and Interacting Multi-Task Learning Network for Joint JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13 Speech and Accent Recognition,” IEEE/ACM Transactions on Audio, Speech, and Language ...

    2021

  17. [25]

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer,

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V . Le, Geoffrey E. Hinton, and Jeff Dean, “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer,” in Proc. ICLR, 2017

    2017

  18. [26]

    A Review of Sparse Expert Models in Deep Learning,

    William Fedus, Jeff Dean, and Barret Zoph, “A Review of Sparse Expert Models in Deep Learning,” arXiv preprint arXiv:2209.01667 , 2022

    2022 arXiv

  19. [27]

    ST-MoE: Design- ing Stable and Transferable Sparse Expert Models,

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus, “ST-MoE: Design- ing Stable and Transferable Sparse Expert Models,” arXiv preprint arXiv:2202.08906, 2022

    2022 arXiv

  20. [28]

    MoEC: Mixture of Expert Clusters,

    Yuan Xie, Shaohan Huang, Tianyu Chen, and Furu Wei, “MoEC: Mixture of Expert Clusters,” in Proc. AAAI, 2023, pp. 13807–13815

    2023

  21. [29]

    U2++ MoE: Scaling 4.7 x parameters with minimal impact on RTF,

    Xingchen Song, Di Wu, Binbin Zhang, Dinghao Zhou, Zhendong Peng, Bo Dang, Fuping Pan, and Chao Yang, “U2++ MoE: Scaling 4.7 x parameters with minimal impact on RTF,” arXiv preprint arXiv:2404.16407, 2024

    2024 arXiv

  22. [30]

    MoLE : Mixture Of Language Experts For Multi-Lingual Automatic Speech Recognition,

    Yoohwan Kwon and Soo-Whan Chung, “MoLE : Mixture Of Language Experts For Multi-Lingual Automatic Speech Recognition,” in Proc. ICASSP, 2023, pp. 1–5

    2023

  23. [31]

    LoRA: Low-Rank Adaptation of Large Language Models,

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” in Proc. ICLR, 2022

    2022

  24. [32]

    MOELoRA: An MOE-based Parameter Efficient Fine-Tuning Method for Multi-task Medical Applications,

    Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng, “MOELoRA: An MOE-based Parameter Efficient Fine-Tuning Method for Multi-task Medical Applications,” arXiv preprint arXiv:2310.18339 , 2023

    arXiv

  25. [33]

    Mixture-of-LoRAs: An Efficient Multitask Tuning Method for Large Language Models,

    Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang, “Mixture-of-LoRAs: An Efficient Multitask Tuning Method for Large Language Models,” in Proc. LREC-COLING, 2024, pp. 11371–11380

    2024

  26. [34]

    SiRA: Sparse Mixture of Low Rank Adaptation,

    Yun Zhu, Nevan Wichers, Chu-Cheng Lin, Xinyi Wang, Tianlong Chen, Lei Shu, Han Lu, Canoee Liu, Liangchen Luo, Jindong Chen, et al., “SiRA: Sparse Mixture of Low Rank Adaptation,” arXiv preprint arXiv:2311.09179, 2023

    2023 arXiv

  27. [35]

    MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA based Mixture of Experts,

    Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang, “MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA based Mixture of Experts,” arXiv preprint arXiv:2404.15159 , 2024

    2024 arXiv

  28. [36]

    MoRAL: MoE Augmented LoRA for LLMs’ Lifelong Learning,

    Shu Yang, Muhammad Asif Ali, Cheng-Long Wang, Lijie Hu, and Di Wang, “MoRAL: MoE Augmented LoRA for LLMs’ Lifelong Learning,” arXiv preprint arXiv:2402.11260 , 2024

    2024 arXiv

  29. [37]

    Lo- RAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin,

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang, “Lo- RAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-St...

    2024

  30. [38]

    HDMoLE: Mixture of LoRA Experts with Hierarchical Routing and Dynamic Thresholds for Fine-Tuning LLM-based ASR Models,

    Bingshen Mu, Kun Wei, Qijie Shao, Yong Xu, and Lei Xie, “HDMoLE: Mixture of LoRA Experts with Hierarchical Routing and Dynamic Thresholds for Fine-Tuning LLM-based ASR Models,” inProc. ICASSP, 2025

    2025

  31. [39]

    Mixture of LoRA Experts,

    Xun Wu, Shaohan Huang, and Furu Wei, “Mixture of LoRA Experts,” in Proc. ICLR, 2024

    2024

  32. [40]

    AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts,

    Zefang Liu and Jiahua Luo, “AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts,” arXiv preprint arXiv:2405.00361, 2024

    2024 arXiv

  33. [41]

    Robust Speech Recognition via Large- Scale Weak Supervision,

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever, “Robust Speech Recognition via Large- Scale Weak Supervision,” in Proc. ICML, 2023, pp. 28492–28518

    2023

  34. [42]

    Towards Better Decoding and Language Model Integration in Sequence to Sequence Models,

    Jan Chorowski and Navdeep Jaitly, “Towards Better Decoding and Language Model Integration in Sequence to Sequence Models,” in Proc. Interspeech, 2017, pp. 523–527

    2017

  35. [43]

    Improved Training of End-to-end Attention Models for Speech Recognition,

    Albert Zeyer, Kazuki Irie, Ralf Schl ¨uter, and Hermann Ney, “Improved Training of End-to-end Attention Models for Speech Recognition,” in Proc. Interspeech, 2018, pp. 7–11

    2018

  36. [44]

    A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition,

    Shubham Toshniwal, Anjuli Kannan, Chung-Cheng Chiu, Yonghui Wu, Tara N Sainath, and Karen Livescu, “A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition,” in Proc. SLT, 2018, pp. 369–375

    2018

  37. [45]

    Deliberation Networks: Sequence Generation Beyond One-Pass Decoding,

    Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu, “Deliberation Networks: Sequence Generation Beyond One-Pass Decoding,” in Proc. NeurIPS, 2017, pp. 1784–1794

    2017

  38. [46]

    Trans- former Based Deliberation for Two-Pass Speech Recognition,

    Ke Hu, Ruoming Pang, Tara N Sainath, and Trevor Strohman, “Trans- former Based Deliberation for Two-Pass Speech Recognition,” in Proc. SLT, 2021, pp. 68–74

    2021

  39. [47]

    Scaling Up Deliberation For Multilingual ASR,

    Ke Hu, Bo Li, and Tara N Sainath, “Scaling Up Deliberation For Multilingual ASR,” in Proc. SLT, 2023, pp. 771–776

    2023

  40. [48]

    Language Models are Unsupervised Multitask Learners,

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al., “Language Models are Unsupervised Multitask Learners,” OpenAI blog, 2019

    2019

  41. [49]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proc. NAACL-HLT, 2019, pp. 4171–4186

    2019

  42. [50]

    WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing,

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al., “WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing,” IEEE Journal of Selected Topics in Signal Processing, vol...

    2022

  43. [51]

    On Online Attention-Based Speech Recognition and Joint Mandarin Character-Pinyin Training,

    William Chan and Ian R Lane, “On Online Attention-Based Speech Recognition and Joint Mandarin Character-Pinyin Training,” in Proc. Interspeech, 2016, pp. 3404–3408

    2016

  44. [52]

    A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinese,

    Shiyu Zhou, Linhao Dong, Shuang Xu, and Bo Xu, “A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinese,” in Proc. ICONIP , 2018, pp. 210–220

    2018

  45. [53]

    On Modular Training of Neural Acoustics-to-Word Model for LVCSR,

    Zhehuai Chen, Qi Liu, Hao Li, and Kai Yu, “On Modular Training of Neural Acoustics-to-Word Model for LVCSR,” in Proc. ICASSP, 2018, pp. 4754–4758

    2018

  46. [54]

    Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese,

    Shiyu Zhou, Linhao Dong, Shuang Xu, and Bo Xu, “Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese,” in Proc. Interspeech, 2018, pp. 791–795

    2018

  47. [55]

    Decoupling Recognition and Transcription in Mandarin ASR,

    Jiahong Yuan, Xingyu Cai, Dongji Gao, Renjie Zheng, Liang Huang, and Kenneth Church, “Decoupling Recognition and Transcription in Mandarin ASR,” in Proc. ASRU, 2021, pp. 1019–1025

    2021

  48. [56]

    Cascade RNN- Transducer: Syllable Based Streaming On-Device Mandarin Speech Recognition with a Syllable-To-Character Converter,

    Xiong Wang, Zhuoyuan Yao, Xian Shi, and Lei Xie, “Cascade RNN- Transducer: Syllable Based Streaming On-Device Mandarin Speech Recognition with a Syllable-To-Character Converter,” in Proc. SLT , 2021, pp. 15–21

    2021

  49. [57]

    Decoupling Pronunciation and Language for End-to- End Code-Switching Automatic Speech Recognition,

    Shuai Zhang, Jiangyan Yi, Zhengkun Tian, Ye Bai, Jianhua Tao, and Zhengqi Wen, “Decoupling Pronunciation and Language for End-to- End Code-Switching Automatic Speech Recognition,” in Proc. ICASSP, 2021, pp. 6249–6253

    2021

  50. [58]

    Multi-Level Modeling Units for End-to-End Mandarin Speech Recognition,

    Yuting Yang, Binbin Du, and Yuke Li, “Multi-Level Modeling Units for End-to-End Mandarin Speech Recognition,” in Proc. ISCSLP, 2022, pp. 175–179

    2022

  51. [59]

    On the Properties of Neural Machine Translation: Encoder- Decoder Approaches,

    Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio, “On the Properties of Neural Machine Translation: Encoder- Decoder Approaches,” in Proc. SSST@EMNLP, 2014, pp. 103–111

    2014

  52. [60]

    Common V oice: A Massively-Multilingual Speech Corpus,

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber, “Common V oice: A Massively-Multilingual Speech Corpus,” in Proc. LREC, 2020, pp. 4218–4222

    2020

  53. [61]

    The Accented English Speech Recognition Challenge 2020: Open Datasets, Tracks, Baselines, Results and Methods,

    Xian Shi, Fan Yu, Yizhou Lu, Yuhao Liang, Qiangze Feng, Daliang Wang, Yanmin Qian, and Lei Xie, “The Accented English Speech Recognition Challenge 2020: Open Datasets, Tracks, Baselines, Results and Methods,” in Proc. ICASSP, 2021, pp. 6918–6922

    2020

  54. [62]

    Qwen2- audio Technical Report,

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al., “Qwen2- audio Technical Report,” arXiv preprint arXiv:2407.10759 , 2024

    2024 arXiv

  55. [63]

    FireRedASR: Open-Source Industrial-Grade Mandarin Speech Recognition Mod- els from Encoder-Decoder to LLM Integration,

    Kai-Tuo Xu, Feng-Long Xie, Xu Tang, and Yao Hu, “FireRedASR: Open-Source Industrial-Grade Mandarin Speech Recognition Mod- els from Encoder-Decoder to LLM Integration,” arXiv preprint arXiv:2501.14350, 2025

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.