Pith. sign in

REVIEW 4 major objections 5 minor 31 references

Beyond Hard Sharing: Efficient Multi-Task Speech-to-Text Modeling with Supervised Mixture of Experts

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Supervised Mixture of Experts routes speech-to-text by task and bandwidth, reporting a 6.35% relative WER improvement over hard parameter sharing with no added inference cost.

desk verdict The paper's core idea is simple and the setup is controlled, but the headline WER improvements don't recompute from the tables and the encoder S-MoE's contribution is confounded by initialization from DecS-MoE. read the letter →

arxiv 2508.10009 v1 pith:VVAZ7AKG submitted 2025-08-05 cs.CL cs.AIcs.SDeess.AS

Hojun Jin , Eunsoo Hong , Ziwon Hyung , Sungjun Lim , Seungjin Lee , Keunseok Cho This is my paper
classification cs.CLcs.AIcs.SDeess.AS
keywords mixtureofexpertsmulti-tasklearningspeech-to-textautomaticspeechrecognitiontranslationsupervisedroutingbandwidth-invariantASRtransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes S-MoE, a variant of Mixture of Experts in which routing is decided by explicit guiding tokens and known input conditions rather than by a learned gating network. The aim is to let a single Transformer-based speech-to-text model handle two tasks—Korean automatic speech recognition and Korean-to-English speech translation—and two audio bandwidths without the task interference typical of hard parameter sharing. The authors show that assigning separate feedforward experts to each bandwidth in the encoder and to each task in the decoder improves WER and BLEU over the shared baseline, and that this comes with no additional inference cost since only one expert runs per sample. If the result holds, it offers a simple recipe for multi-task, multi-condition speech models on resource-constrained devices.

What carries the argument

The Supervised Mixture of Experts (S-MoE) layer: a standard Transformer feedforward block is replaced by two expert feedforward networks selected by a predefined gating function $G'(x)$ that outputs a one-hot vector based on the input audio bandwidth (encoder) or the task tag in the target text (decoder). No learned gate is trained; the route is known from the data itself. This gives the parameter increase of MoE without its routing computation or routing instability.

What would settle it

Recompute the 6.35% relative WER improvement claimed for the narrowband condition in Section 4.2: averaging the three narrowband rows (NB, G.711 NB, AMR NB) across the three test sets in Table 2 gives roughly 5.97%, not 6.35%. Publishing exact unrounded per-condition averages, or running the comparison across multiple random seeds with confidence intervals, would settle the central claim.

Watch

Extended reading notes

Core claim

The paper claims that replacing the learned gating network of a Mixture of Experts layer with a fixed, condition-dependent router—called Supervised Mixture of Experts (S-MoE)—reduces task interference in a single Transformer speech-to-text model. The router is driven by guiding tokens prepended to the target text (<transcribe> vs <translate>) and by the known bandwidth of the input audio (8 kHz narrowband vs 16 kHz wideband). In the encoder, each bandwidth activates its own feedforward expert; in the decoder, each task activates its own feedforward expert. Because exactly one expert is active per layer per input, the model's active parameter count equals that of the base shared model, so the

Load-bearing premise

The claimed gains rest on single-run WER and BLEU differences measured on 1,000-utterance in-house test sets with no significance testing or confidence intervals, and on ST training targets partly produced by an unmeasured in-house machine translation model; if those numbers are noise or the translation targets are skewed, the central quantitative claim does not hold.

Editorial extensions

If this is right

  • If S-MoE works as reported, a single deployed model can serve ASR and ST on both 8 kHz and 16 kHz audio, and can output both transcription and translation in one inference step by batching the two task tags.
  • At matched trainable parameters, the decoder S-MoE outperforms a doubled-FFN decoder, so the benefit comes from task-specialized parameters rather than raw capacity.
  • Since only one expert is active per input per layer, the active parameter count and inference compute match the base model, making the method compatible with on-device constraints.
  • The same predefined-routing idea can be applied to any known input condition, since the router needs a label already present in the data, not a learned predictor.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the router is keyed to observable tags, the same design could separate experts by speaker, noise condition, or language within a single model, where each condition can be rendered as a guiding token.
  • The architecture supports more than two experts with no inference-cost penalty, so scaling to per-domain or per-codec experts is a natural stress test of the method's limits.
  • A head-to-head comparison against soft-gated MoE, adapters, or task-specific normalization at identical active-parameter budgets would separate the benefit of hard, tag-based routing from the benefit of extra specialized parameters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Supervised Mixture of Experts (S-MoE) for multi-task speech-to-text modeling. Instead of a learned gating network, S-MoE uses fixed routing: in the encoder, the gating function selects an FFN expert according to whether the input is narrowband (8 kHz) or wideband (16 kHz); in the decoder, it selects an FFN expert according to whether the target task is ASR or ST. The method is applied to a 12-layer encoder, 6-layer decoder Transformer trained on 40,000 hours of Korean speech for Korean ASR and Korean-to-English ST. Experiments report WER and BLEU improvements over a hard-parameter-sharing baseline, including a 6.35% relative WER improvement when S-MoE is applied to both encoder and decoder, with no increase in active parameters during single-task inference. The paper also compares against parameter-matched DecFFNx2 and Whisper baselines.

Significance. If the reported results hold, the contribution is practical and simple: it removes the need to train a gating network, preserves the computational efficiency of hard sharing, and shows modest but consistent gains over a parameter-matched hard-sharing baseline on public Fleurs data. The fixed-gating design also avoids the circularity concern that would arise if the router were fitted to the evaluation set. However, the quantitative claims are undermined by arithmetic errors in the relative-improvement calculations and by the absence of a key ablation isolating the encoder S-MoE; these must be corrected before the central claim is credible.

major comments (4)
  1. [§4.2, Table 2, and Abstract] The headline 6.35% relative WER improvement does not recompute from Table 2. Averaging the three NB rows gives Base WER 63.84/9 = 7.09% and EncDecS-MoE WER 60.03/9 = 6.67%; the correct relative improvement is (7.09−6.67)/7.09 = 5.97%, not 6.35%. The reported number equals (7.09−6.67)/6.67, i.e., it divides by the new model's average. The same denominator error appears in §4.1: from Table 1, Base vs DecS-MoE WER averages of 4.92 vs 4.55 imply 7.5% relative, not 8.06%; DecFFNx2 vs DecS-MoE implies 6.83%, not 7.33%; and the Whisper-large comparison implies about 18% relative WER improvement, not 21.98%. Please recalculate all relative percentages and correct the abstract.
  2. [§3.2 and §4.2] The encoder-specific contribution is not isolated. Section 3.2 states that EncDecS-MoE is initialized from the DecS-MoE model, and Section 4.1 shows DecS-MoE already improves WER over Base-Model on the same wideband evaluation sets. Table 2 compares EncDecS-MoE only to Base-Model; no DecS-MoE-only model fine-tuned on the same NB/WB subset is reported. The NB/WB gains in Table 2 could therefore be inherited entirely from the decoder S-MoE. The claim that applying S-MoE to both encoder and decoder is responsible for the improvement requires a fine-tuned DecS-MoE control in Table 2, and ideally a fine-tuned DecFFNx2 control as well.
  3. [§4.1–§4.2, Tables 1–2] All comparisons are single runs on in-house test sets of 1,000 utterances, with no confidence intervals, significance tests, or repeated-seed variation. Key differences are about 0.3–0.4% absolute WER, which is within typical run-to-run variation for this architecture class. Without at least a few seeds or a paired bootstrap for the headline comparisons, the reported effect sizes cannot be distinguished from noise. Please provide variance estimates or additional runs for the central WER/BLEU claims.
  4. [§3.1.1] The ST training signal is partially generated by an in-house MT model whose quality is never measured. The paper does not state what fraction of the 40,000 hours uses MT-generated translations, nor does it report any automatic or human evaluation of the MT system. Since ST target quality affects the shared encoder and is part of the multi-task training, and since BLEU is a reported outcome, this is a missing control. Please quantify the proportion of synthesized targets and report the MT model's quality, or replace it with a public fixed system.
minor comments (5)
  1. [Figure 2 caption vs §4.2] Figure 2 states that 'both feedforward blocks within the decoder are utilized,' while §4.2 claims the same number of active parameters as Base-Model. Clarify whether simultaneous ASR/ST inference activates both decoder experts and how active parameters are counted in that mode.
  2. [Table 2] Table 2 should include the same Param/Active columns as Table 1. Currently trainable and active parameters for EncDecS-MoE are given only in prose, making the efficiency comparison harder to verify.
  3. [§3.2 and §4.2] The sentence 'All models in Table 2 are fine-tuned on the NB/WB data' is underspecified. State the exact fine-tuning subset size, number of epochs, learning rate, and whether Base-Model, DecS-MoE, and EncDecS-MoE all receive identical fine-tuning procedures.
  4. [§3.1.1] The description of NB/WB conversion is ambiguous: '15% of the training data' with 'simple downsampling' and 'codec-based processing using AMR-WB, AMR-NB, and G.711-NB' could mean duplicated variants or replacements. Clarify the actual composition of the NB/WB fine-tuning subset.
  5. [§4.1] Phrases such as 'a 0.37% in WER' and 'a 0.53 in BLEU' should specify absolute percentage-point changes and BLEU units, respectively, to avoid confusion with relative percentages.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: S-MoE routing is a fixed, label-based indicator function and all reported gains are direct measurements against held-out test sets.

full rationale

The paper's central claim is empirical: replacing hard-shared FFNs with two fixed-routed experts (encoder routed by bandwidth, decoder by task) improves WER/BLEU. Equations 2 and 3 define G'_enc and G'_dec as indicator functions of pre-labeled bandwidth/task, not learned from or fit to the evaluation data; no parameter is tuned to reproduce the reported WER/BLEU numbers. The EncDecS-MoE model is initialized from DecS-MoE, but this is a training initialization, not a fitted input renamed as prediction; the Table 2 numbers are direct measurements on held-out in-house and Fleurs test sets. No load-bearing self-citation: all references are external datasets, codecs, Whisper, or standard MoE/Transformer works; the authors' prior work is not invoked as evidence. The abstract's 6.35% relative WER figure does not exactly recompute from the Table 2 row averages (which give about 5.97% using the baseline as denominator), but an arithmetic/reporting inconsistency is a correctness-risk issue, not circularity. Similarly, the absence of error bars and the confounding of encoder/decoder contributions (no DecS-MoE-only fine-tuned control) weaken evidential strength but do not make the derivation equivalent to its inputs. The architecture is self-contained against external benchmarks (Whisper, Fleurs), and no 'prediction' is forced by construction. Therefore score 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper makes no theoretical derivation; its claim is carried by empirical comparisons. The ledger therefore captures the manual architecture choices and unvalidated data assumptions, not fitted constants.

free parameters (3)
  • Number of experts n per S-MoE layer = 2
    Hand-chosen; no sensitivity analysis, all results are with n=2.
  • Expert-to-condition assignment (encoder NB->E1, WB->E0; decoder ASR->E1, ST->E0) = Fixed mapping
    Hand-specified routing, though symmetric relabeling would be equivalent, so it is a naming choice rather than a fitted quantity.
  • NB/WB augmentation mix (15% of training data with AMR-WB, AMR-NB, G.711-NB) = 15%
    Chosen by hand for the fine-tuning subset; no ablation on its size or composition.
assumptions (3)
  • standard math Standard Transformer seq2seq training with cross-entropy loss converges to a usable ASR/ST model.
    Invoked throughout Section 3.2 without proof; the entire empirical evaluation assumes supervised training works.
  • domain assumption The in-house MT model produces adequate English translations for ST training data when human translations are unavailable.
    Section 3.1.1 states ST targets are generated by an in-house MT model; quality is never measured, and this directly shapes ST training.
  • domain assumption Bandwidth and task labels used for routing are correct and available at inference.
    Section 2.1 defines gating based on these labels; at deployment one must know whether input is NB or WB and which task is requested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Hard Sharing: Efficient Multi-Task Speech-to-Text Modeling with Supervised Mixture of Experts." pith.science (2026). https://pith.science/paper/VVAZ7AKG

@misc{pith2026250810009,
  author       = {Pith},
  title        = {Pith review of: Beyond Hard Sharing: Efficient Multi-Task Speech-to-Text Modeling with Supervised Mixture of Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VVAZ7AKG}},
  note         = {Machine review of arXiv:2508.10009}
}
read the original abstract

Hard-parameter sharing is a common strategy to train a single model jointly across diverse tasks. However, this often leads to task interference, impeding overall model performance. To address the issue, we propose a simple yet effective Supervised Mixture of Experts (S-MoE). Unlike traditional Mixture of Experts models, S-MoE eliminates the need for training gating functions by utilizing special guiding tokens to route each task to its designated expert. By assigning each task to a separate feedforward network, S-MoE overcomes the limitations of hard-parameter sharing. We further apply S-MoE to a speech-to-text model, enabling the model to process mixed-bandwidth input while jointly performing automatic speech recognition (ASR) and speech translation (ST). Experimental results demonstrate the effectiveness of the proposed S-MoE, achieving a 6.35% relative improvement in Word Error Rate (WER) when applied to both the encoder and decoder.

Figures

Figures reproduced from arXiv: 2508.10009 by the authors.

Figure 1
Figure 1. The S-MoE layer embedded within a STT model. The S-MoE in the encoder directs inputs to expert networks based on the speech bandwidth (NB or WB), while the S-MoE in the decoder routes inputs according to the task (ASR or ST). ficient multi-task STT training. Our approach is inspired by the MoE principle but tailored for scenarios where both input and output conditions are clearly known. We focus on structured, multi… view at source ↗
Figure 2
Figure 2. Embedding flow of S-MoE Transformer. During training/inference, a single feedforward block within the encoder is chosen, whereas both feedforward blocks within the decoder are utilized. 3.1.2. Evaluation datasets To evaluate the general performance of ASR and ST, we use two in-house test sets along with the publicly available Fleurs [21] dataset. Our in-house test sets consist of 1,000 samples of male and female spe… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 23 canonical work pages

  1. [1]

    Beyond Hard Sharing: Efficient Multi-Task Speech-to-Text Modeling with Supervised Mixture of Experts

    Introduction Speech-to-text (STT) models are typically trained on wideband (WB) audio, as it provides richer acoustic features. However, narrowband (NB) audio remains essential in mobile environ- ments, particularly for telephony applications. Due to the spec- tral differences resulting from the sampling rate gap, NB and WB models are often trained separa...

    work page Pith review arXiv 2025

  2. [2]

    Further details are presented in the following subsections

    Approach In this section, we propose an architecture as shown in Figure 1. Further details are presented in the following subsections. 2.1. Supervised Mixture of Experts (S-MoE) By slightly adapting the conventional MoE [5] structure, the output y of the S-MoE module is expressed as Equation 1. Sim- ilar to the standard MoE layer, Ei(x) represents the out...

    work page

  3. [3]

    Datasets 3.1.1

    Experimental Setup 3.1. Datasets 3.1.1. Training datasets Our training dataset consists of 40,000 hours of Korean speech data from the publicly available AIHub corpus [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18]. For ASR, we use the provided transcripts as target text. For ST, if a corresponding English translation is not available, we generate the target t...

    work page

  4. [4]

    Decoder S-MoE To validate the effectiveness of S-MoE applied to the decoder blocks, we conducted experiments on two tasks: Korean-to- English (ko2en) ST and Korean ASR (ko-ASR)

    Experimental Results 4.1. Decoder S-MoE To validate the effectiveness of S-MoE applied to the decoder blocks, we conducted experiments on two tasks: Korean-to- English (ko2en) ST and Korean ASR (ko-ASR). The results are summarized in Table 1. The model described in Section 3.2 serves as the primary baseline (Base-Model) for comparison. The trainable param...

    work page

  5. [5]

    By us- ing guiding tokens instead of dynamic gating functions, S-MoE ensures efficient training and inference while improving per- formance across various tasks

    Conclusion We propose the S-MoE architecture, which mitigates task in- terference in multi-task learning for STT applications. By us- ing guiding tokens instead of dynamic gating functions, S-MoE ensures efficient training and inference while improving per- formance across various tasks. Our approach enables a single model to handle ASR and ST tasks with ...

    work page

  6. [6]

    Sources of degradation of speech recognition in the telephone network,

    P. J. Moreno and R. M. Stern, “Sources of degradation of speech recognition in the telephone network,” in Proceedings of ICASSP’94. IEEE International Conference on Acoustics, Speech and Signal Processing, vol. 1. IEEE, 1994, pp. I–109

    work page 1994

  7. [7]

    Training wideband acoustic mod- els using mixed-bandwidth training data for speech recognition,

    M. L. Seltzer and A. Acero, “Training wideband acoustic mod- els using mixed-bandwidth training data for speech recognition,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 15, no. 1, pp. 235–245, 2006

    work page 2006

  8. [8]

    Multi-task learning with deep neural networks: A survey,

    M. Crawshaw, “Multi-task learning with deep neural networks: A survey,”arXiv preprint arXiv:2009.09796, 2020

    arXiv 2009

Show all 31 references
  1. [9]

    An overview of multi-task learning in deep neural net- works,

    S. Ruder, “An overview of multi-task learning in deep neural net- works,” arXiv preprint arXiv:1706.05098, 2017

    2017 arXiv

  2. [10]

    A survey on multi-task learning ieee transactions on knowledge and data engineering,

    Y . Zhang and Q. Yang, “A survey on multi-task learning ieee transactions on knowledge and data engineering,” 2021

    2021

  3. [11]

    Sparsely activated mixture-of-experts are robust multi-task learners,

    S. Gupta, S. Mukherjee, K. Subudhi, E. Gonzalez, D. Jose, A. H. Awadallah, and J. Gao, “Sparsely activated mixture-of-experts are robust multi-task learners,” arXiv preprint arXiv:2204.07689, 2022

    2022 arXiv

  4. [12]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  5. [13]

    [Online]

    (2020) Conference voice data. [Online]. Avail- able: https://aihub.or.kr/aihubdata/data/view.do?currMenu=115& topMenu=100&aihubDataSe=data&dataSetSn=132

    2020

  6. [14]

    [Online]

    (2022) Broadcasting contents korean-european interpre- tation and translation voice data. [Online]. Avail- able: https://aihub.or.kr/aihubdata/data/view.do?currMenu=115& topMenu=100&aihubDataSe=data&dataSetSn=71384

    2022

  7. [15]

    [Online]

    (2022) Multilingual translation and translation reading data. [Online]. Available: https://aihub.or.kr/aihubdata/data/view.do? currMenu=115&topMenu=100&aihubDataSe=data&dataSetSn= 71524

    2022

  8. [16]

    [Online]

    (2021) Korean-english mixed recognition data. [Online]. Avail- able: https://aihub.or.kr/aihubdata/data/view.do?currMenu=115& topMenu=100&aihubDataSe=data&dataSetSn=71260

    2021

  9. [17]

    [Online]

    (2018) Korean voice data. [Online]. Avail- able: https://aihub.or.kr/aihubdata/data/view.do?currMenu=115& topMenu=100&aihubDataSe=data&dataSetSn=123

    2018

  10. [18]

    [Online]

    (2021) Broadcast content dialogue voice recognition data. [Online]. Available: https://aihub.or.kr/aihubdata/data/view.do? currMenu=115&topMenu=100&aihubDataSe=data&dataSetSn= 463

    2021

  11. [19]

    [Online]

    (2022) Korean language college lecture data. [Online]. Avail- able: https://aihub.or.kr/aihubdata/data/view.do?currMenu=115& topMenu=100&aihubDataSe=data&dataSetSn=71627

    2022

  12. [20]

    Pulse code modulation (pcm) of voice fre- quencies,

    C. Recommendation, “Pulse code modulation (pcm) of voice fre- quencies,” in ITU, 1988

    1988

  13. [21]

    Our in-house test sets consist of 1,000 samples of male and female speech from daily conversations, with refer- ence translations curated by professional translators

    dataset. Our in-house test sets consist of 1,000 samples of male and female speech from daily conversations, with refer- ence translations curated by professional translators. Addition- ally, to assess the model under different bandwidth conditions, we generate NB/WB variants ...

  14. [22]

    [Online]

    (2022) Broadcasting contents korean-english interpre- tation and translation voice data. [Online]. Avail- able: https://aihub.or.kr/aihubdata/data/view.do?currMenu=115& topMenu=100&aihubDataSe=data&dataSetSn=71379

    2022

  15. [23]

    [Online]

    (2021) Conference voice recognition data by major area. [Online]. Available: https://aihub.or.kr/aihubdata/data/view.do?currMenu= 115&topMenu=100&aihubDataSe=data&dataSetSn=464

    2021

  16. [24]

    Whisper is a widely used mul- tilingual speech model trained with diverse language pairs

    models in our experiments. Whisper is a widely used mul- tilingual speech model trained with diverse language pairs. Our models are optimized for a single translation direction so a di- rect comparison may not be entirely fair, but Whisper provides a strong reference point for...

  17. [25]

    [Online]

    (2021) Noise environment voice recognition data. [Online]. Available: https://aihub.or.kr/aihubdata/data/view.do?currMenu= 115&topMenu=100&aihubDataSe=data&dataSetSn=568

    2021

  18. [26]

    [Online]

    (2022) News script and anchor voice data. [Online]. Avail- able: https://aihub.or.kr/aihubdata/data/view.do?currMenu=115& topMenu=100&aihubDataSe=data&dataSetSn=71557

    2022

  19. [27]

    The adaptive multirate wideband speech codec (amr-wb),

    B. Bessette, R. Salami, R. Lefebvre, M. Jelinek, J. Rotola-Pukkila, J. Vainio, H. Mikkola, and K. Jarvinen, “The adaptive multirate wideband speech codec (amr-wb),” IEEE transactions on speech and audio processing, vol. 10, no. 8, pp. 620–636, 2002

    2002

  20. [28]

    Fleurs: Few-shot learning evaluation of universal representations of speech,

    A. Conneau, M. Ma, S. Khanuja, Y . Zhang, V . Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna, “Fleurs: Few-shot learning evaluation of universal representations of speech,” in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 798–805

    2022

  21. [29]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural In- formation Processing Systems, 2017

    2017

  22. [30]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318

    2002

  23. [31]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.