Optimizing Speech Multi-View Feature Fusion through Conditional Computation

Bei Li; Hao Yang; Jingbo Zhu; Min Zhang; Tong Xiao; Weiqiao Shan; Xiaofeng Zhao; Yuang Li; Yuchen Han; Yuhao Zhang

REVIEW 4 major objections 6 minor 31 references

Optimizing Speech Multi-View Feature Fusion through Conditional Computation

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that fusing SSL units and FBanks through a gradient-sensitive gate resolves their update conflict, matching BLEU while training about 1.24 times faster.

desk verdict Useful fusion recipe with a mechanism story that overreaches: the BLEU result is plausible, but Eq. (7) does not hold for the trained network. read the letter →

arxiv 2501.08057 v1 pith:VHZIE4TI submitted 2025-01-14 eess.AS cs.AIcs.CLcs.SD

Weiqiao Shan , Yuhao Zhang , Yuchen Han , Bei Li , Xiaofeng Zhao , Yuang Li , Min Zhang , Hao Yang

show 2 more authors

Tong Xiao Jingbo Zhu

This is my paper · ORCID

classification eess.AScs.AIcs.CLcs.SD

keywords speechtranslationconditionalcomputationfeaturefusionself-supervisedfeaturesgradientconflictgatingnetworkmulti-stagedropoutMuST-C

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Self-supervised speech features converge fast but underperform spectral features in speech translation; the paper argues the reason is that the two feature types push model parameters in conflicting directions. It proposes conditional-computation fusion: a gating network that weights each feature's contribution according to gradient alignment, plus a multi-stage dropout that alternates between the two single views and the fused view during training. The claimed result is that the conflict is resolved, so the fused model matches or slightly exceeds FBanks-only BLEU on MuST-C En-De, En-Fr, and En-Es while reaching that quality in 30 epochs instead of 35-40, an average 1.24 times training speedup. The claim matters because it offers a recipe for combining cheap, in-context SSL representations with reliable spectral features in downstream speech tasks without paying a quality penalty.

What carries the argument

The load-bearing mechanism is the gradient-sensitive gating network (GSGN) together with a multi-stage dropout. GSGN computes a per-dimension gate vector for each branch, $g_{\text{fbank}}$ and $g_{\text{unit}}$, fusing inputs as $x_{\text{fusion}} = g_{\text{fbank}} \circ x_{\text{fbank}} + g_{\text{unit}} \circ x_{\text{unit}}$. The paper shows, under a linear residual approximation, that the fused branch's gradient is the Hadamard-weighted sum of the two separate-feature gradients (Eq. 7), so an auxiliary loss (Eq. 11) can supervise $g_{\text{fbank}}$ to realize the gradient-surgery deconfliction rule of Eq. 10. The multi-stage dropout (Eq. 13) samples the FBanks, unit, or fused input with stage-dependent thresholds, which the authors argue counters the drifting gradient alignment over training.

What would settle it

Measure, on the trained model with a batch of real inputs, the cosine similarity between the actual gradient of the fused branch and the right-hand side of Eq. 7 using the trained gate values; if the similarity is not close to 1 where the gate loss is applied, or if removing the gate loss does not raise the conflict fraction above the reported 32%, the claimed deconfliction mechanism is not operating as stated.

Watch

Extended reading notes

Core claim

The central discovery is a measurable, fixable cause of fusion failure: the gradients produced by FBanks and by HuBERT-unit features contain conflicting components at the parameter level, up to 32% of them in the authors' measurements, and the conflict worsens as training advances. The paper's GSGN derives per-dimension gating weights and an auxiliary loss that steers the FBanks-branch gate weight to match the deconfliction coefficient from gradient surgery, canceling the S-feature gradient's opposing component while keeping the S-feature branch fully active. Multi-stage dropout then samples FBanks, unit, or fused input per training stage to counter gradient-direction instability. With both mechanisms, the full model reaches 25.26, 36.24, and 30.24 BLEU on En-De, En-Fr, and En-Es in 30 epochs, versus 25.08, 36.22, and 30.04 for the FBanks-only baseline in 40, 35, and 37 epochs respectively.

Load-bearing premise

The load-bearing premise is that the gradient of the fused branch is exactly the Hadamard-weighted sum of the two features' separate gradients, which holds only under the paper's linear residual approximation with frozen parameters.

Editorial extensions

If this is right

On MuST-C En-De, En-Fr, and En-Es, GSGN+Drop matches or improves FBanks-only BLEU while converging in 30 epochs versus 35-40, an average 1.24 times training speedup.
The same recipe applies to pre-trained ST models: on En-De it yields 28.19 BLEU versus 27.98 for FBanks-only, with a 1.22 times speedup and 89.53% of FBanks gate weights exceeding 1, indicating active conflict correction.
GSGN alone outperforms a simple concatenation-based gating network (24.18 vs. 23.8 BLEU on En-De, 28 vs. 32 epochs), showing that the gradient-sensitive weighting, not just extra capacity, is what resolves the conflict.
The random-noise ablation indicates the S-feature's convergence advantage is genuine: replacing it with noise degrades the speedup, while adding noise to FBanks-only improves robustness but not convergence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The deconfliction target assumes the gradient decomposition of Eq. 7 holds; in a real nonlinear network the true fused gradient will deviate, so the gate loss may be optimizing an approximation. A direct gradient-similarity audit on the trained model would show how large the mismatch is.
Because gradient conflict is measured per parameter element and the gate is also per dimensional, the method could be extended to per-layer or per-head gating, which might capture where conflicts concentrate.
The same conditional-computation recipe is testable in other multi-view settings (audio-visual, multilingual, or multimodal speech) where feature update directions diverge; the 1.24x speedup is measured in epochs, and wall-clock gains depend on the added gate and dropout cost.
The stage thresholds $\delta_{\text{fbank}}$ and $\delta_{\text{unit}}$ are chosen empirically; a schedule learned from the measured $\cos(\theta)$ trajectory could replace the hand-set stages.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a real engineering contribution that deserves referee time, but the headline mechanism is shakier than the paper lets on. If you read it for the recipe — a gradient-sensitive gate plus staged dropout for fusing FBanks and HuBERT units — the reported result is genuinely useful. GSGN+Drop matches FBanks-only BLEU on En-De, En-Fr, and En-Es while cutting training from 35–40 epochs down to 30. Code is linked and hyperparameters are concrete enough to re-implement. The noise-replacement experiment in Table II is also a thoughtful check that the S-feature carries signal, not just regularization. The main soft spot is exactly what the stress-test note flags. Eq. (7) is not valid for the trained network. The derivation assumes a linear residual layer and frozen parameters; in the real network the loss Jacobian depends on which feature produced the forward pass, and the gates are learned functions of both inputs. So Eq. (11) supervises the gate toward a target derived from a decomposition that does not hold. That weakens the mechanistic claim, not the BLEU comparison. The paper gives no direct measurement that Lgate actually moves the fused gradient toward the deconflicted target. Sec. IV-D's weight analysis is partly self-confirming: the gate is trained to hit that target, so seeing gate values near it does not independently validate deconfliction. Smaller issues: all results are single runs with no error bars, and the BLEU gaps of 0.18, 0.02, and 0.20 are within typical seed variance. The performance match is the honest claim; the small gains are not. Speedup is epoch-based, not wall-clock, and the staged thresholds are hand-tuned. These are fixable with additional seeds, timing, and a clarified loss formulation. For a practitioner working on speech translation feature fusion, this is a solid recipe paper to read. The empirical result is not circular and stands independent of the gradient-surgery story. I would not cite the mechanism as established, but I would cite the empirical recipe with a caveat. Send it to review; the right outcome is conditional acceptance after adding seeds, wall-clock timing, and either a corrected derivation or an explicitly labeled approximation.

Referee Report

4 major / 6 minor

Summary. The paper proposes a conditional-computation framework for fusing spectral (FBanks) and self-supervised (HuBERT discrete-unit) features in end-to-end speech translation. The main components are a gradient-sensitive gating network (GSGN) that learns per-dimension weights for the two feature streams, and a multi-stage dropout that randomly samples FBanks, S-feature, or their fusion during training. The central empirical claim is that GSGN+Drop matches a FBanks-only baseline in BLEU (+0.18, +0.02, +0.20 on En-De, En-Fr, En-Es) while reducing training epochs by factors of 1.17-1.33. The paper derives the gating loss from a claimed gradient decomposition and presents analyses of gradient angles and gate weights to support the conflict-resolution mechanism.

Significance. If the empirical claims hold, the contribution is practically useful: it offers a lightweight way to accelerate convergence with SSL features without sacrificing translation quality, and the code is publicly released. The ablations (concat-gating baseline, random-noise control, pre-trained initialization) are well chosen and move in the right direction. However, the significance is limited in its current form by two issues. First, all BLEU numbers are single runs, and the reported gains are smaller than the effect of random noise in the paper's own control experiment, so the headline result is not statistically established. Second, the theoretical derivation of the gating loss relies on a gradient decomposition that is not valid for the trained network, so the mechanistic explanation of the gains is not currently supported by the evidence.

major comments (4)

[II-B, Eq. (7)] Eq. (7) is presented as a weighted summation of the two separate-feature gradients under frozen parameters. In the actual model this decomposition does not hold: the factor labeled 'input-independent' in Eq. (6) is the product of the loss Jacobian and (w_j^T+1), and dL/dh_out depends on which input was fed because the hidden states differ. Moreover, g_fbank and g_unit in Eq. (4) are outputs of Eq. (3), i.e., learned functions of both inputs, not constants. Therefore the Lgate target in Eq. (11) is not guaranteed to realize Eq. (10), and the claim that GSGN 'eliminates' conflicting gradient components is not proven. The manuscript should either measure the actual fused gradient direction before and after applying the gate, or substantially weaken the mechanism claim.
[Table I and Section III-B] All primary results are single runs with no seed variation, error bars, or significance test; the deltas over FBanks-only are +0.18, +0.02, and +0.20, while the paper's own 'Sum' noise baseline in Table II improves BLEU by +0.29 over FBanks-only, so these differences are within run-to-run variation. The claimed 1.24x speedup is also based only on the number of epochs at which the best 10 checkpoints were averaged, not on wall-clock time, and no budget-matched comparison with multiple seeds is reported.
[IV-D and Eq. (11)] The weight analysis in Section IV-D is self-confirming because gfbank is explicitly trained with Lgate toward the deconfliction target. The observation that 89.53% of gfbank elements exceed 1 in the pretrained model is therefore a property of the loss, not independent evidence that the fused gradient is deconflicted. The authors should report the gradient angle cos(theta) or the projection of the actual fused gradient onto Eq. (10)'s target before and after applying the gate, or compare with a model trained without Lgate.
[Section III-B, inference paragraph] The sentence 'we use the same setting as the third stage in multi-stage dropout to sample from three branches' is ambiguous, since Eq. (13) with delta_fbank=0.3 and delta_unit=0 still randomly chooses xfbank with probability 0.3. If inference is stochastic, the reported test BLEU values depend on the random draw and need to be averaged over multiple samples; if inference uses the fusion branch deterministically, the text must say so. Either way, the current description is insufficient to reproduce Table I.

minor comments (6)

[Section IV-A, Table II] The row labels in Table II are confusing; in particular, the 'Replace' rows do not identify the base model being modified, and the prose description does not match the table structure.
[Section II-B, Eqs. (8)-(10)] The vectors a and b and the norm operations in Eqs. (8)-(10) are not formally defined, which makes the target in Eq. (11) ambiguous for readers.
[Figure 1 and Section I] The definition of 'conflicting components' and the fraction 'up to 32%' are not specified, and the axes of Fig. 1(b) are not clearly labeled.
[Section II-C] The multi-stage dropout thresholds and stage boundaries are chosen empirically; a sensitivity analysis would help establish robustness.
[Section III-A] The S-feature extraction uses 'Hubert' but the model name is commonly abbreviated as 'HuBERT'; please standardize the spelling and provide the exact checkpoint version used.
[Throughout] Several typos and grammatical issues remain, e.g., 'carry on the experiences in MuST-C' and 'we provide the last epoch at the average 10 checkpoints'; the manuscript needs a careful language pass.

Circularity Check

1 steps flagged · score 3.0 of 10

Sec. IV-D's gate-weight 'evidence' is self-confirming: Eq. (11) trains gfbank to equal the deconfliction coefficient, so gfbank>1 in conflict cases is the fitted target by construction; BLEU/speed claims remain independently measured.

self definitional [Sec. IV-D (Weight Analysis for GSGN) and Eq. (11) in Sec. II-B]
"Lgate = MSE(gfbank, 1) if cos(θ) >= 0, MSE(gfbank, (1 − ∥∂Lunit/∂wi∥ cos(θ)/∥∂Lfbank/∂wi∥)) else (11) ... For the pre-trained model, we observe that GSGN assigns a higher weight to gfbank. This is because, when gradient conflicts occur, the cos(θ) is less than 0, and the term (1 − ∥b∥ cos(θ)/∥a∥) in Eq. 10, or equivalently the mean of gfbank in Eq. 4 would be greater than 1. We find that 89.53% of the elements in gfbank exceed 1 in the pre-trained model. This indicates that the conflict gradient in the pre-trained model is severe."

Eq. (11) defines the gate training loss as MSE(gfbank, 1 − ∥∂Lunit/∂wi∥ cosθ/∥∂Lfbank/∂wi∥) whenever cosθ < 0, so the optimization explicitly drives gfbank toward exactly that deconfliction coefficient. The Section IV-D observation that 89.53% of gfbank elements exceed 1 is therefore a read-back of the supervised target, not an independent measurement that the conflict gradient is severe or that GSGN has handled it. The later statement that this demonstrates GSGN 'effectively addresses the gradient-related issues' partly rests on this self-confirming evidence. The central BLEU/speed claims in Tables I and IV are still externally measured and not entailed by Eq. (11), so the circularity is partial rather than total.

full rationale

The only concrete circular step is the Section IV-D weight analysis. Because Eq. (11) supervises gfbank to match exactly the deconfliction scalar from Eq. (10), reporting gfbank>1 where cosθ<0 is observing the loss target by construction. This makes the mechanistic claim that GSGN resolves gradient conflict partly self-confirming. However, the paper's main empirical claims are independent of this circular step: Table I reports BLEU scores on the MuST-C test sets with training epochs, Tables III and IV compare against concatenation-based gating and pre-trained baselines, and Fig. 3 measures gradient conflict statistics directly. Those results are not forced by Eq. (11). The approximation in Eq. (7) (linear residual, frozen parameters) is a correctness or robustness concern, not circularity. The paper's self-citations to the authors' earlier ST architectures and pretraining recipes are setup choices, not load-bearing derivations, so they do not raise the score. Overall, one self-confirming analysis step with an independently measured central result warrants a score of 3.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's claimed effectiveness depends on hand-tuned dropout thresholds, an implicit gate loss weight, and stage boundary epochs, plus five stated or unstated modeling assumptions about linear residual approximations, gradient decomposition, gradient surgery, frozen-parameter measurement, and staged feature importance. No new physical or conceptual entities are introduced; the S-feature is existing HuBERT output. The deconfliction target in Lgate is computed from the model's own gradients, which explains why the gate analysis in Section IV-D is partly self-confirming.

free parameters (4)

δfbank thresholds = 0.3 (epoch<10), 0.5 (10≤epoch<25), 0.3 (epoch≥25)
Hand-tuned to balance the three branches across training stages; part of the multi-stage dropout schedule.
δunit thresholds = 0 (epoch<10), 0.3 (10≤epoch<25), 0 (epoch≥25)
Hand-tuned in parallel with δfbank to control when the unit feature branch is sampled.
Stage boundary epochs = 10 and 25
The switch points between the three training stages are chosen empirically, not derived from the analysis.
Gate loss weight = 1.0 (implicit)
Lfinal = L + Lgate has no reported coefficient for Lgate; the relative scale of the gate loss is a free choice that affects training behavior.

assumptions (5)

ad hoc to paper Residual network layer can be approximated as linear: h_{i+1} ≈ w_i h_i + b_i + h_i (Eq. 5).
The derivation of Eq. 6 and Eq. 7 relies on this approximation, which omits nonlinearities and layer normalization.
ad hoc to paper Gradient of the fused input equals a Hadamard-weighted sum of separate-feature gradients (Eq. 7).
This holds only when parameters are frozen and layers are linear; the paper uses it as the foundation for Lgate without validating it in the joint-training setting.
domain assumption Gradient deconfliction via Eq. 8, borrowed from multi-task gradient surgery [26], is an appropriate balance for FBanks and S-feature gradients.
The paper adopts the multitask-learning deconfliction rule without justifying that the two feature branches should be treated as tasks with one designated main feature.
domain assumption Cosine similarity between gradients measured by sequentially feeding each feature through a frozen model reflects actual training conflict.
The motivation in Fig. 1(b) and the analysis in Fig. 3 use frozen-parameter measurements, but the model updates all parameters jointly during real training.
domain assumption Feature roles change over training in three discrete stages, justifying the multi-stage dropout schedule.
The staged threshold schedule is motivated by observed instability of cos(θ), but the specific staging and thresholds are not derived from a model of training dynamics.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Optimizing Speech Multi-View Feature Fusion through Conditional Computation." pith.science (2026). https://pith.science/paper/VHZIE4TI

@misc{pith2026250108057,
  author       = {Pith},
  title        = {Pith review of: Optimizing Speech Multi-View Feature Fusion through Conditional Computation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VHZIE4TI}},
  note         = {Machine review of arXiv:2501.08057}
}

read the original abstract

Recent advancements have highlighted the efficacy of self-supervised learning (SSL) features in various speech-related tasks, providing lightweight and versatile multi-view speech representations. However, our study reveals that while SSL features expedite model convergence, they conflict with traditional spectral features like FBanks in terms of update directions. In response, we propose a novel generalized feature fusion framework grounded in conditional computation, featuring a gradient-sensitive gating network and a multi-stage dropout strategy. This framework mitigates feature conflicts and bolsters model robustness to multi-view input features. By integrating SSL and spectral features, our approach accelerates convergence and maintains performance on par with spectral models across multiple speech translation tasks on the MUSTC dataset.

Figures

Figures reproduced from arXiv: 2501.08057 by the authors.

**Figure 2.** ST model architecture. encoder (T-enc), and a decoder (dec). The forward process of the network can be denoted as follows: hA = A-enc(x) hT = T-enc(hA) hout = dec(hT) (1) During the training phase, the model generates the final prediction based on the decoder’s output and obtains the final loss via the subsequent cross-entropy function: L = CrossEntropy(wouthout, ˆy) (2) where ˆy is the target text. Conventional ST … view at source ↗

**Figure 3.** The cos(θ) between two gradients generated by each feature. We find Instability(left): The cos(θ) becomes smaller and smaller with training for cos(θ) > 0, and Conflicting(right): Percent of conflicting gradients(cos(θ) < 0) among all gradients in an input batch. 5k 10k 15k 20k 0.6 0.65 0.7 0.75 0.8 Training Step Mean (a) Scratch Model. 5k 10k 15k 20k 1 1.1 1.2 1.3 1.4 Training Step Mean (b) Pretrained Model [PITH_… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: The mean value of gfbank in the model training from scratch (left) and the pre-trained model (right). only the S-feature. Our fusion method successfully leverages the strengths of both features, as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: The final result of our method. V. CONCLUSION AND FUTURE WORK Recent studies have demonstrated the effectiveness of S-features across various speech-related tasks, offering lightweight and generalpurpose multi-view speech representations. However, we observed that whi…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 18 canonical work pages

[1]

Introduction to digital speech processing,

Lawrence R Rabiner, Ronald W Schafer, et al., “Introduction to digital speech processing,” Foundations and Trends® in Signal Processing, vol. 1, no. 1–2, pp. 1–194, 2007

work page 2007
[2]

Learning robust features using deep learning for automatic seizure detection,

Pierre Thodoroff, Joelle Pineau, and Andrew Lim, “Learning robust features using deep learning for automatic seizure detection,” in Machine learning for healthcare conference . PMLR, 2016, pp. 178–190

work page 2016
[3]

Listen, attend and spell,

William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals, “Listen, attend and spell,” arXiv preprint arXiv:1508.01211 , 2015

arXiv 2015
[4]

wav2vec: Unsupervised pre-training for speech recognition,

Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli, “wav2vec: Unsupervised pre-training for speech recognition,” arXiv preprint arXiv:1904.05862, 2019

arXiv 1904
[5]

Data2vec: A general framework for self-supervised learning in speech, vision and language,

Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli, “Data2vec: A general framework for self-supervised learning in speech, vision and language,” in International Conference on Machine Learning . PMLR, 2022, pp. 1298–1312

work page 2022
[6]

wav2vec 2.0: A framework for self-supervised learning of speech representations,

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems , vol. 33, pp. 12449–12460, 2020

work page 2020
[7]

Hubert: Self- supervised speech representation learning by masked prediction of hidden units,

Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakho- tia, Ruslan Salakhutdinov, and Abdelrahman Mohamed, “Hubert: Self- supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

work page 2021
[8]

Exploration on HuBERT with Multiple Resolu- tions,

Jiatong Shi, Yun Tang, Hirofumi Inaguma, Hongyu Gong, Juan Pino, and Shinji Watanabe, “Exploration on HuBERT with Multiple Resolu- tions,” in Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH , 2023, vol. 2023, pp. 3287–3291

work page 2023

Show all 31 references

[9]

Multi-resolution HuBERT: Multi-resolution speech self-supervised learning with masked unit prediction,

Jiatong Shi, Hirofumi Inaguma, Xutai Ma, Ilia Kulikov, and Anna Sun, “Multi-resolution HuBERT: Multi-resolution speech self-supervised learning with masked unit prediction,” arXiv preprint arXiv:2310.02720, 2023

2023 arXiv
[10]

Unit- speech: Speaker-adaptive speech synthesis with untranscribed data,

Heeseung Kim, Sungwon Kim, Jiheum Yeom, and Sungroh Yoon, “Unit- speech: Speaker-adaptive speech synthesis with untranscribed data,” arXiv preprint arXiv:2306.16083 , 2023

2023 arXiv
[11]

Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities,

Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu, “Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities,” arXiv preprint arXiv:2305.11000, 2023

2023 arXiv
[12]

Multi- view information-bottleneck representation learning,

Zhibin Wan, Changqing Zhang, Pengfei Zhu, and Qinghua Hu, “Multi- view information-bottleneck representation learning,” in Proceedings of the AAAI conference on artificial intelligence , 2021, vol. 35, pp. 10085– 10092

2021
[13]

Deep multi-view learning methods: A review,

Xiaoqiang Yan, Shizhe Hu, Yiqiao Mao, Yangdong Ye, and Hui Yu, “Deep multi-view learning methods: A review,” Neurocomputing, vol. 448, pp. 106–129, 2021

2021
[14]

The platonic representation hypothesis,

Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola, “The platonic representation hypothesis,” arXiv preprint arXiv:2405.07987 , 2024

2024 arXiv
[15]

Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translation,

Dan Berrebbi, Jiatong Shi, Brian Yan, Osbel L ´opez-Francisco, Jonathan Amith, and Shinji Watanabe, “Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translation,” Inter- speech 2022, 2022

2022
[16]

Improving speech emotion recognition by fusing self-supervised learn- ing and spectral features via mixture of experts,

Jonghwan Hyeon, Yung-Hwan Oh, Young-Jun Lee, and Ho-Jin Choi, “Improving speech emotion recognition by fusing self-supervised learn- ing and spectral features via mixture of experts,” Data & Knowledge Engineering, vol. 150, pp. 102262, 2024

2024
[17]

Deep learning of representations: Looking forward,

Yoshua Bengio, “Deep learning of representations: Looking forward,” in International conference on statistical language and speech processing . Springer, 2013, pp. 1–37

2013
[18]

Depth- adaptive transformer,

Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli, “Depth- adaptive transformer,” arXiv preprint arXiv:1910.10073 , 2019

1910 arXiv
[19]

Layer skip: Enabling early exit inference and self-speculative decoding,

Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agar- wal, Ahmed Roman, et al., “Layer skip: Enabling early exit inference and self-speculative decoding,” arXiv preprint arXiv:2404.16710, 2024

2024 arXiv
[20]

Modeling task relationships in multi-task learning with multi- gate mixture-of-experts,

Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi, “Modeling task relationships in multi-task learning with multi- gate mixture-of-experts,” in Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , 2018, pp. 1930–1939

2018
[21]

Sparsely activated mixture-of-experts are robust multi-task learners,

Shashank Gupta, Subhabrata Mukherjee, Krishan Subudhi, Eduardo Gonzalez, Damien Jose, Ahmed H Awadallah, and Jianfeng Gao, “Sparsely activated mixture-of-experts are robust multi-task learners,” arXiv preprint arXiv:2204.07689 , 2022

2022 arXiv
[22]

Stacked acoustic-and-textual encoding: Integrating the pre-trained models into speech translation encoders,

Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Qi Ju, Tong Xiao, Jingbo Zhu, et al., “Stacked acoustic-and-textual encoding: Integrating the pre-trained models into speech translation encoders,” arXiv preprint arXiv:2105.05752, 2021

2021 arXiv
[23]

Soft Alignment of Modality Space for End-to- End Speech Translation,

Yuhao Zhang, Kaiqi Kou, Bei Li, Chen Xu, Chunliang Zhang, Tong Xiao, and Jingbo Zhu, “Soft Alignment of Modality Space for End-to- End Speech Translation,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 1...

2024
[24]

Deep residual learning for image recognition,

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770– 778

2016
[25]

Progres- sive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations,

Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong, “Progres- sive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations,” in Proceedings of the 14th ACM Conference on Recommender Systems , 2020, pp. 269–278

2020
[26]

Gradient surgery for multi-task learning,

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn, “Gradient surgery for multi-task learning,” Advances in Neural Information Processing Systems , vol. 33, pp. 5824– 5836, 2020

2020
[27]

Must-c: a multilingual speech translation corpus,

Mattia A Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi, “Must-c: a multilingual speech translation corpus,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno...

2019
[28]

Librispeech: an asr corpus based on public domain audio books,

Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

2015
[29]

Modality adaption or regularization? a case study on end-to-end speech translation,

Yuchen Han, Chen Xu, Tong Xiao, and Jingbo Zhu, “Modality adaption or regularization? a case study on end-to-end speech translation,” arXiv preprint arXiv:2306.07650, 2023

2023 arXiv
[30]

Attention is all you need,

Ashish Vaswani, “Attention is all you need,” arXiv preprint arXiv:1706.03762, 2017

2017 arXiv
[31]

A call for clarity in reporting BLEU scores,

Matt Post, “A call for clarity in reporting BLEU scores,” arXiv preprint arXiv:1804.08771, 2018

2018 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Introduction to digital speech processing,

Lawrence R Rabiner, Ronald W Schafer, et al., “Introduction to digital speech processing,” Foundations and Trends® in Signal Processing, vol. 1, no. 1–2, pp. 1–194, 2007

work page 2007

[2] [2]

Learning robust features using deep learning for automatic seizure detection,

Pierre Thodoroff, Joelle Pineau, and Andrew Lim, “Learning robust features using deep learning for automatic seizure detection,” in Machine learning for healthcare conference . PMLR, 2016, pp. 178–190

work page 2016

[3] [3]

Listen, attend and spell,

William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals, “Listen, attend and spell,” arXiv preprint arXiv:1508.01211 , 2015

arXiv 2015

[4] [4]

wav2vec: Unsupervised pre-training for speech recognition,

Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli, “wav2vec: Unsupervised pre-training for speech recognition,” arXiv preprint arXiv:1904.05862, 2019

arXiv 1904

[5] [5]

Data2vec: A general framework for self-supervised learning in speech, vision and language,

Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli, “Data2vec: A general framework for self-supervised learning in speech, vision and language,” in International Conference on Machine Learning . PMLR, 2022, pp. 1298–1312

work page 2022

[6] [6]

wav2vec 2.0: A framework for self-supervised learning of speech representations,

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems , vol. 33, pp. 12449–12460, 2020

work page 2020

[7] [7]

Hubert: Self- supervised speech representation learning by masked prediction of hidden units,

Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakho- tia, Ruslan Salakhutdinov, and Abdelrahman Mohamed, “Hubert: Self- supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

work page 2021

[8] [8]

Exploration on HuBERT with Multiple Resolu- tions,

Jiatong Shi, Yun Tang, Hirofumi Inaguma, Hongyu Gong, Juan Pino, and Shinji Watanabe, “Exploration on HuBERT with Multiple Resolu- tions,” in Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH , 2023, vol. 2023, pp. 3287–3291

work page 2023