Pith. sign in

REVIEW 3 major objections 5 minor 53 references

Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One-pass pruning cuts speech models 60–65% with no WER loss

desk verdict A simple, cheap one-pass pruning method that looks genuinely effective on LibriSpeech, but the 'lossless' claim rests on a significance test the paper doesn't document. read the letter →

arxiv 2505.22608 v1 pith:ZDKAUXKP submitted 2025-05-28 cs.SD cs.AIeess.AS

Haoning Xu , Zhaoqing Li , Youjun Chen , Huimeng Wang , Guinan Li , Mengzhe Geng , Chengxi Deng , Xunying Liu This is my paper
classification cs.SDcs.AIeess.AS
keywords speechfoundationmodelsmodelpruningone-passcompressionsparsity-awaregatingself-pinchinggateswav2vec2HuBERTlossless
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that speech foundation models can be pruned and fine-tuned in one pass, using one learnable threshold per layer to decide which parameters to keep. It reports that on the LibriSpeech 100-hour corpus, this method removes 65% of wav2vec2-base parameters and 60% of HuBERT-large parameters while causing no statistically significant word error rate increase on the test-clean set. Because pruning and parameter update happen together, the method avoids the separate post-pruning fine-tuning stage used by many earlier approaches. The claim matters because lossless compression at high sparsity would make large speech models cheaper to deploy without incurring a multi-stage compression pipeline.

What carries the argument

The sparsity-aware self-pinching gate is a single learnable threshold $t_l$ per layer. Each weight is converted into a pruning probability by a sigmoid over $(W_{i,j}^l)^2 - (t_l)^2$ scaled by a temperature, and a straight-through estimator rounds the mask for the forward pass while retaining the soft gradient for the backward pass. The threshold is shared across the layer, so the added parameters are only 72 for wav2vec2-base and 144 for HuBERT-large. The gate couples pruning with weight update and lets each layer learn its own sparsity level, which is what the argument relies on.

What would settle it

Train wav2vec2-base at 65% sparsity with the same one-pass gate on a different dataset, such as LibriSpeech test-other or TED-LIUM, and run the paper's statistical significance test; if a significant WER increase appears, the lossless claim does not generalise beyond the tested condition. Alternatively, freeze the thresholds at their initial values during fine-tuning and compare the final WER to the full method; if there is no significant difference, the learned thresholds are not the source of the benefit.

Watch

Extended reading notes

Core claim

The central discovery is that fine-grained magnitude pruning can be lossless at high sparsity when the pruning decision is learned jointly with the weights through a per-layer threshold. Each layer's threshold is converted by a sigmoid into soft pruning probabilities, and a straight-through estimator binarizes the mask during the forward pass while keeping a soft gradient in the backward pass. This lets each layer discover its own sensitivity-based sparsity, with an overhead of only one scalar per layer. The paper reports lossless compression at 65% sparsity for wav2vec2-base and 60% for HuBERT-large on the test-clean subset, and a best WER of 7.05% for WavLM-base+ under a 23-million-parameter constraint.

Load-bearing premise

The method assumes that a single per-layer scalar threshold applied to parameter magnitudes is a reliable enough importance criterion, and that the straight-through estimator gives a trustworthy gradient for learning that threshold and the weights together.

Editorial extensions

If this is right

  • If correct, lossless compression of SSL speech models at 60–65% sparsity can be achieved in a single fine-tuning pass, removing the need for post-pruning fine-tuning or iterative pruning.
  • The per-layer thresholds provide an automatic, data-driven allocation of sparsity across layers, which outperforms uniform magnitude pruning and channel-wise NAS pruning at sparsity levels above 50%.
  • With only one additional scalar per layer, the pruning overhead is negligible under 1e-6 of the model size, so the method scales to larger transformer models without an architecture search stage.
  • The method generalises to other SSL speech models such as WavLM-base+, achieving the lowest reported test-clean WER of 7.05% under a 23M-parameter limit among the methods compared in the paper.
  • Because compression time is at least 25% lower than previous methods on the same task, the approach shortens development cycles for on-device ASR systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The learned per-layer threshold could be read as a sensitivity measure for each layer, which might be reused to guide other compression decisions such as mixed-precision quantization.
  • The same one-pass gating principle could be tested on other transformer-based modalities where magnitude pruning and straight-through estimators are already common, provided the threshold initialisation is adapted.
  • The final binary mask is unstructured, so real inference speedups would require hardware or software support for arbitrary sparsity; without that, the reported gains are primarily in parameter count and training time, not latency.
  • Because the paper's statistical significance test is applied mainly to the test-clean subset, the lossless claim may not automatically extend to dev-other or test-other subsets at the same sparsity; the tables show WER increases there that are not marked as insignificant.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes Sparsity-aware Self-pinching Gates, a one-pass pruning method for speech self-supervised models. A single learnable threshold per linear layer is trained jointly with the model; the forward pass uses a hard binary mask while the backward pass uses a soft mask via a straight-through estimator. Experiments on LibriSpeech-100h report lossless compression at 65% sparsity for wav2vec2-base and 60% for HuBERT-large on the test-clean subset, with compression times of 11 and 13 GPU hours and at least 25% fewer fine-tuning epochs than SparseWav. The paper also compares against uniform magnitude pruning, NAS-based channel pruning, and prior compression methods, and reports a best test-clean WER of 7.05% at a 4.26x compression ratio on WavLM-base+.

Significance. The core idea is simple and parameter-efficient: a per-layer scalar threshold, tied to weight magnitudes, requires only 72 or 144 additional parameters for wav2vec2-base and HuBERT-large. If the lossless claim holds, the method is practically attractive for on-device ASR because pruning and fine-tuning are integrated into a single stage, avoiding post-hoc iterative pruning. The systematic comparison with uniform magnitude pruning and NAS-based channel pruning is also useful and supports the claim that joint training of thresholds and weights helps. However, the central 'lossless' claim rests on a statistical test that is not documented in sufficient detail, and the cross-method comparisons in Table 2 are confounded by different training data sizes and backbones, so the headline significance is not yet established.

major comments (3)
  1. [§5.2, Table 1, footnote 2] The definition of 'lossless' as 'no statistically significant WER increase' makes the MAPSSWE significance test the load-bearing evidence for the paper's main claim, but the manuscript reports only dagger markers with no p-values, test statistics, matched-segment counts, variance estimates, or multiple-comparison correction across the many sparsity levels and subsets in Table 1. A non-significant difference such as test-clean 6.12 vs. 6.06 at 65% sparsity (row 5) or 3.54 vs. 3.44 at 60% (row 12) is not evidence of equivalence; low power can produce non-significance even when a real degradation exists. The authors should report full test details, p-values, a pre-specified equivalence or non-inferiority margin, and confidence intervals, and should address the multiple-comparison issue arising from testing many sparsity levels.
  2. [§5.3, Table 2] The headline comparison with prior compression methods is not apples-to-apples. Most prior systems in rows 5–16 are trained on LibriSpeech-960h, whereas the proposed systems are trained on the 100h subset, and the backbones and decoders also differ across systems. Consequently, the claims of 'lowest WER of 7.05% under a comparable compression ratio' and 'at least 25% less fine-tuning epochs' are not supported as controlled comparisons. The authors should either compare against methods trained on the same 100h data with the same backbone and decoder, or clearly state the confounds and avoid presenting the comparison as a head-to-head improvement.
  3. [§5.2.1, Table 1 rows 1–5] The 'lossless' claim is subset-specific, but the abstract and conclusion state 'lossless compression' without consistently noting this restriction. At 65% sparsity for wav2vec2-base, test-other WER increases from 13.52 (row 1) to 16.89 (row 5), a 3.37-point absolute increase, while test-clean changes by only 0.06. The paper should explicitly state which subsets are used for the lossless claim and justify why a single subset (test-clean) is sufficient to support the general claim of lossless compression, especially when other subsets show large degradations.
minor comments (5)
  1. [Table 1] The column header row for Table 1 is garbled ('dev testclean other clean other'), making it hard to map the numeric columns to subsets; please reformat the table with clear column labels.
  2. [§4, Eq. (4) and Eq. (6)] The threshold t^l is described as a learnable scalar, but the manuscript does not specify whether it is constrained to be non-negative or how it is initialized and updated relative to the weight magnitudes; adding these details would improve reproducibility.
  3. [§4, Eq. (7)] The text says η is set to 0 'if the desired sparsity is achieved', but the criterion for 'achieved' during training is not defined; please clarify how the target sparsity is monitored and enforced.
  4. [§5.1] No random seeds, number of runs, or variance estimates are reported for the main results, and no code is provided; this limits the ability to assess the stability of the reported WER numbers.
  5. [§5.3] The 'at least 25% less compression time' statement is based on comparing fine-tuning epochs (30 vs. >40), but the actual wall-clock training hours for SparseWav are not given; please clarify whether the comparison is in epochs or hours and whether other computational costs are included.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the compression results are empirical, and the 'lossless' label is a defined statistical criterion, not a prediction derived from the method's own inputs.

full rationale

The paper's central claim is an experimental result, not a derivation whose conclusion is contained in its assumptions. The sparsity-aware self-pinching gate is explicitly defined in Eqs. (4)-(6), and Eq. (7) is a regularized training loss; the paper then measures WER on LibriSpeech against external baselines. The 'lossless' criterion is defined in footnote 2 as 'no statistically significant WER increase' using MAPSSWE at α=0.05, which is a decision rule for evaluating measured outcomes rather than a quantity fitted from the data. The reported maximum lossless sparsity levels of 65% and 60% are observed results across multiple sparsity settings, not predictions that reduce to the manually set η values or learned thresholds. The two self-citations, [7] and [26], appear only in the literature overview and are not load-bearing for the method or its evaluation. Concerns about the statistical power of MAPSSWE and post-hoc selection across sparsity levels are validity issues, not circularity, and no equation in the paper reduces to itself by construction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim relies on three key assumptions: the magnitude-based importance criterion, the validity of STE gradients, and the statistical power of the significance test. The freely set hyperparameters (eta, tau, target sparsity) are fitted to achieve the desired sparsity, not derived from first principles.

free parameters (3)
  • eta (sparsity control coefficient) = 2e-5, 1e-6, 3e-5, 2e-6 depending on model and sparsity range
    Controls overall sparsity via the L0 penalty; manually set per model and target sparsity range (see Footnote 6).
  • tau (temperature) = Cosine-annealed from 0.5 to 0.01
    Temperature in the sigmoid mask; hand-tuned schedule.
  • Target sparsity levels = 50%, 60%, 65%, 70%, 75%, 80%, 85%, 90%
    The reported sparsity values are selected targets, not emergent predictions; the paper scans multiple levels and reports selected results.
assumptions (3)
  • domain assumption Weight magnitude is a valid importance criterion for pruning
    The gate prunes parameters with magnitude below the threshold; this assumes small-magnitude weights contribute little to ASR accuracy.
  • domain assumption Straight-through estimator provides a valid gradient for the binary mask
    The paper uses STE to backpropagate through the binarized mask, which is an approximation with no theoretical guarantee of convergence.
  • domain assumption MAPSSWE test with alpha=0.05 has sufficient power to detect meaningful WER differences
    The claim of 'no statistically significant WER increase' rests on this test, which is not described in detail.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates." pith.science (2026). https://pith.science/paper/ZDKAUXKP

@misc{pith2026250522608,
  author       = {Pith},
  title        = {Pith review of: Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZDKAUXKP}},
  note         = {Machine review of arXiv:2505.22608}
}
read the original abstract

This paper presents a novel approach for speech foundation models compression that tightly integrates model pruning and parameter update into a single stage. Highly compact layer-level tied self-pinching gates each containing only a single learnable threshold are jointly trained with uncompressed models and used in fine-grained neuron level pruning. Experiments conducted on the LibriSpeech-100hr corpus suggest that our approach reduces the number of parameters of wav2vec2.0-base and HuBERT-large models by 65% and 60% respectively, while incurring no statistically significant word error rate (WER) increase on the test-clean dataset. Compared to previously published methods on the same task, our approach not only achieves the lowest WER of 7.05% on the test-clean dataset under a comparable model compression ratio of 4.26x, but also operates with at least 25% less model compression time.

Figures

Figures reproduced from arXiv: 2505.22608 by the authors.

Figure 1
Figure 1. Comparison between Uniform Magnitude-based Pruning (UMP), NAS-based Channel-wise Pruning (NAS-CP) and Sparsity￾aware Self-pinching Gates (ours). For the l-th layer, (a) UMP directly prunes the same proportion of parameters by magnitude across all layers; (b) NAS-CP introduces architecture-dependent parameters proportional to architecture candidates, which are pre-selected before NAS search; (c) Ours utilizes the wei… view at source ↗
Figure 2
Figure 2. The ASR performance of the pruned wav2vec2-base-100h on the (1) dev and (3) test subsets, as well as the pruned hubert￾large on the (2) dev and (4) test subsets with different sparsity using different methods. Abbreviations are the same as those in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 49 canonical work pages

  1. [1]

    structured

    Introduction In recent years, advancements in self-supervised learning (SSL) for speech technologies, particularly through foundation mod- els like wav2vec2.0 [1], HuBERT [2] and WavLM [3], have greatly improved their utility in applications like automatic speech recognition (ASR). Despite these advancements, the widespread adoption of these models in res...

    work page

  2. [2]

    Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates

    Our method reduces the fragility to pruning by using the 1Compared to the uncompressed wav2vec2.0- base and HuBERT- large models, the additional pruning-required parameters account for only 8e-7 and 5e-7 of the total model size, respectively. 2“lossless” in this paper refers to no statistically significant WER increase against the uncompressed baseline. a...

    work page Pith review arXiv 2025

  3. [3]

    In contrast, previous fine-grained ap- proach [21] performed pruning and fine-tuning separately

    Our method ensures consistency between model pruning and parameter update by integrating them into a single stage for SSL-based ASR systems. In contrast, previous fine-grained ap- proach [21] performed pruning and fine-tuning separately. This one-pass compression stage also enables different layers to be pruned in varying sparsity based on their sensitivi...

    work page

  4. [4]

    Our method demonstrates efficiency in terms of com- pression time, as it eliminates the need for additional operations such as post-pruning fine-tuning [21] or iterative pruning [32] after the one-pass compression stage

    work page

  5. [5]

    Our method guarantees the compactness of pruning- required parameters by introducing a single threshold as an ad- ditional component for each layer. In contrast, the number of additional pruning-required parameters in previous methods is based on i) the design of teacher-student model [22, 29], ii) the number of candidates [10, 25] or iii) the layer size [35]

    work page

  6. [6]

    For example, HuBERT consists of a CNN encoder, a Transformer encoder, a projection layer and a code embedding layer

    wav2vec2.0 and HuBERT Models Speech SSL models such as wav2vec2.0 [1], HuBERT [2], and WavLM [3] share similar Transformer backbones. For example, HuBERT consists of a CNN encoder, a Transformer encoder, a projection layer and a code embedding layer. Transformer en- coder accounts for over 90% of the total number of parameters, where each encoder block co...

    work page

  7. [7]

    Previous works 3.1. Uniform Magnitude-based Pruning The Uniform Magnitude-based Pruning (UMP) is inspired by the principle that parameters with smaller magnitudes (abso- lute values) have less influence on the output. As shown in Fig. 1 (a), in UMP, the parameters of a specific layer are sorted by magnitude, and the same proportion of parameters with rel-...

    work page

  8. [8]

    Sparsity-aware Self-pinching Gates The core concept of Sparsity-aware Self-pinching Gates is to leverage the weights already being learned to construct the mask using only one additional learnable threshold per layer. Our approach facilitates flexible allocation of layer-wise sparsity across different layers based on their sensitivities, ultimately en- ab...

    work page

Show all 53 references
  1. [9]

    The training loss is given by: L = Lctc + η X l ∥M l ∥0, (7) where η is a constant coefficient to control the overall sparsity

    During backward propagation, the gradient is accumulated using the actual values in Ml. The training loss is given by: L = Lctc + η X l ∥M l ∥0, (7) where η is a constant coefficient to control the overall sparsity. η is set to 0 if the desired sparsity is achieved; otherwise,...

  2. [10]

    (” and “)

    Experiments 5.1. Experimental setup Uncompressed baselines and data. For wav2vec2.0-base, the wav2vec2-base-100h is downloaded from Huggingface3 as our baseline. For HuBERT- large, we fine-tuned HuBERT-large- ll60k4 for 20 epochs as our baseline, with other setups con- sistent...

  3. [11]

    Our method shows superior performance under the same model-size constraint while reduc- ing the fine-tuning time compared to the previous works

    Conclusion We introduced a cutting-edge one-pass compression method that simultaneously prunes and trains SSL speech foundation models using a threshold per layer. Our method shows superior performance under the same model-size constraint while reduc- ing the fine-tuning time ...

  4. [12]

    14200220, 14200021, 14200324 and Innovation Technology Fund grant No

    Acknowledgements This research is supported by Hong Kong RGC GRF grant No. 14200220, 14200021, 14200324 and Innovation Technology Fund grant No. ITS/218/21

  5. [13]

    wav2vec 2.0: A frame- work for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed et al., “wav2vec 2.0: A frame- work for self-supervised learning of speech representations,” in NeurIPS, 2020

    2020

  6. [14]

    HuBERT: Self- supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai et al. , “HuBERT: Self- supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM T-ASLP, vol. 29, pp. 3451–3460, 2021

    2021

  7. [15]

    WavLM: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen et al. , “WavLM: Large-scale self- supervised pre-training for full stack speech processing,” IEEE J-STSP, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  8. [16]

    2-bit conformer quantization for automatic speech recognition,

    O. Rybakov, P. Meadowlark, S. Ding et al. , “2-bit conformer quantization for automatic speech recognition,” in Interspeech, 2023

    2023

  9. [17]

    4-bit conformer with na- tive quantization aware training for speech recognition,

    S. Ding, P. Meadowlark, Y . He et al., “4-bit conformer with na- tive quantization aware training for speech recognition,” in Inter- speech, 2022

    2022

  10. [18]

    I-bert: Integer-only bert quantization,

    S. Kim, A. Gholami, Z. Yao et al. , “I-bert: Integer-only bert quantization,” in International conference on machine learning . PMLR, 2021, pp. 5506–5518

    2021

  11. [19]

    Effective and efficient mixed preci- sion quantization of speech foundation models,

    H. Xu, Z. Li, Z. Jin et al., “Effective and efficient mixed preci- sion quantization of speech foundation models,” arXiv preprint arXiv:2501.03643, 2025

    2025 arXiv

  12. [20]

    A model for every user and budget: Label-free and personalized mixed-precision quantiza- tion,

    E. Fish, U. Michieli, and M. Ozay, “A model for every user and budget: Label-free and personalized mixed-precision quantiza- tion,” in Interspeech, 2023

    2023

  13. [21]

    Efficient conformer-based speech recognition with linear attention,

    S. Li, M. Xu, and X.-L. Zhang, “Efficient conformer-based speech recognition with linear attention,” in APSIPA ASC, 2021

    2021

  14. [22]

    Lossless 4-bit quantization of architecture compressed conformer asr systems on the 300-hr switchboard corpus,

    Z. Li, T. Wang, J. Deng et al. , “Lossless 4-bit quantization of architecture compressed conformer asr systems on the 300-hr switchboard corpus,” in Interspeech, 2023

    2023

  15. [23]

    Unstructured pruning and low rank factorisation of self-supervised pre-trained speech models,

    H. Wang and W.-Q. Zhang, “Unstructured pruning and low rank factorisation of self-supervised pre-trained speech models,” IEEE Journal of Selected Topics in Signal Processing, 2024

    2024

  16. [24]

    Multi-stage progres- sive compression of conformer transducer for on-device speech recognition,

    J. Rathod, N. Dawalatabad, S. Singh et al., “Multi-stage progres- sive compression of conformer transducer for on-device speech recognition,” in Interspeech, 2022

    2022

  17. [25]

    Conformer-based on-device stream- ing speech recognition with KD compression and two-pass archi- tecture,

    J. Park, S. Jin, J. Park et al., “Conformer-based on-device stream- ing speech recognition with KD compression and two-pass archi- tecture,” in IEEE SLT, 2023

    2023

  18. [26]

    DistillW2V2: A small and streaming wav2vec 2.0 based asr model,

    Y . Fu, Y . Kang, S. Cao et al. , “DistillW2V2: A small and streaming wav2vec 2.0 based asr model,” arXiv preprint arXiv:2303.09278, 2023

    2023 arXiv

  19. [27]

    DistilHuBERT: Speech representation learning by layer-wise distillation of hidden-unit bert,

    H.-J. Chang, S.-w. Yang, and H.-y. Lee, “DistilHuBERT: Speech representation learning by layer-wise distillation of hidden-unit bert,” in ICASSP, 2022

    2022

  20. [28]

    LightHuBERT: Lightweight and configurable speech representation learning with once-for-all hidden-unit bert,

    R. Wang, Q. Bai, J. Ao et al. , “LightHuBERT: Lightweight and configurable speech representation learning with once-for-all hidden-unit bert,” in Interspeech, 2022

    2022

  21. [29]

    Deep versus wide: An analysis of student architectures for task-agnostic knowledge dis- tillation of self-supervised speech models,

    T. Ashihara, T. Moriya, K. Matsuura et al., “Deep versus wide: An analysis of student architectures for task-agnostic knowledge dis- tillation of self-supervised speech models,” in Interspeech, 2022

    2022

  22. [30]

    FitHuBERT: Going thinner and deeper for knowledge distillation of speech self-supervised learn- ing,

    Y . Lee, K. Jang, J. Goo et al., “FitHuBERT: Going thinner and deeper for knowledge distillation of speech self-supervised learn- ing,” in Interspeech, 2022

    2022

  23. [31]

    Dynamic sparsity neural net- works for automatic speech recognition,

    Z. Wu, D. Zhao, Q. Liang et al., “Dynamic sparsity neural net- works for automatic speech recognition,” in ICASSP, 2021

    2021

  24. [32]

    Layer pruning on demand with intermediate ctc,

    J. Lee, J. Kang, and S. Watanabe, “Layer pruning on demand with intermediate ctc,” in Interspeech, 2021

    2021

  25. [33]

    Sparsewav: Fast and accurate one- shot unstructured pruning for large speech foundation models,

    T. Gu, B. Liu, H. Shao et al., “Sparsewav: Fast and accurate one- shot unstructured pruning for large speech foundation models,” in Proc. Interspeech 2024, 2024, pp. 4498–4502

    2024

  26. [34]

    Task-agnostic structured pruning of speech representation models,

    H. Wang, S. Wang, W.-Q. Zhang et al., “Task-agnostic structured pruning of speech representation models,” in Interspeech 2023, 2023, pp. 231–235

    2023

  27. [35]

    Accurate and structured prun- ing for efficient automatic speech recognition,

    H. Jiang, L. L. Zhang, Y . Li et al., “Accurate and structured prun- ing for efficient automatic speech recognition,” in Interspeech, 2023

    2023

  28. [36]

    PADA: Pruning assisted domain adaptation for self-supervised speech representations,

    V . S. Lodagala, S. Ghosh, and S. Umesh, “PADA: Pruning assisted domain adaptation for self-supervised speech representations,” in IEEE SLT, 2023

    2023

  29. [37]

    Structured pruning of self- supervised pre-trained models for speech recognition and under- standing,

    Y . Peng, K. Kim, F. Wu et al. , “Structured pruning of self- supervised pre-trained models for speech recognition and under- standing,” in ICASSP, 2023

    2023

  30. [38]

    One-pass multiple conformer and foundation speech systems compression and quantization using an all-in-one neural model,

    Z. Li, H. Xu, T. Wang et al., “One-pass multiple conformer and foundation speech systems compression and quantization using an all-in-one neural model,” inInterspeech 2024, 2024, pp. 4503– 4507

    2024

  31. [39]

    Deep compres- sion of pre-trained transformer models,

    N. Wang, C.-C. C. Liu, S. Venkataramani et al., “Deep compres- sion of pre-trained transformer models,” Advances in Neural In- formation Processing Systems, vol. 35, pp. 14 140–14 154, 2022

    2022

  32. [40]

    USM-Lite: Quantization and sparsity aware fine-tuning for speech recognition with universal speech models,

    S. Ding, Q. David, D. Rim et al., “USM-Lite: Quantization and sparsity aware fine-tuning for speech recognition with universal speech models,” in ICASSP, 2024

    2024

  33. [41]

    DPHuBERT: Joint dis- tillation and pruning of self-supervised speech models,

    Y . Peng, Y . Sudo, S. Muhammad et al., “DPHuBERT: Joint dis- tillation and pruning of self-supervised speech models,” in Inter- speech, 2023

    2023

  34. [42]

    Skill: Similarity- aware knowledge distillation for speech self-supervised learning,

    L. Zampierin, G. B. Hacene, B. Nguyen et al., “Skill: Similarity- aware knowledge distillation for speech self-supervised learning,” in 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), 2024, pp. 675–679

    2024

  35. [43]

    Some statistical issues in the comparison of speech recognition algorithms,

    L. Gillick and S. J. Cox, “Some statistical issues in the comparison of speech recognition algorithms,” in ICASSP, 1989

    1989

  36. [44]

    Parp: Prune, adjust and re- prune for self-supervised speech recognition,

    C.-I. J. Lai, Y . Zhang, A. H. Liuet al., “Parp: Prune, adjust and re- prune for self-supervised speech recognition,” Advances in Neu- ral Information Processing Systems , vol. 34, pp. 21 256–21 272, 2021

    2021

  37. [45]

    Learning asr pathways: A sparse multilingual asr model,

    M. Yang, A. Tjandra, C. Liu et al., “Learning asr pathways: A sparse multilingual asr model,” inICASSP 2023-2023 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  38. [46]

    Learnable sparsity structured pruning for acoustic pre-trained models,

    S. Wang, H. Wang, J. Li et al. , “Learnable sparsity structured pruning for acoustic pre-trained models,” in Proceedings of the 2023 6th International Conference on Signal Processing and Ma- chine Learning, 2023, pp. 76–81

    2023

  39. [47]

    Losses can be blessings: Routing self-supervised speech representations towards efficient multilin- gual and multitask speech processing,

    Y . Fu, Y . Zhang, K. Qianet al., “Losses can be blessings: Routing self-supervised speech representations towards efficient multilin- gual and multitask speech processing,” Advances in Neural Infor- mation Processing Systems, vol. 35, pp. 20 902–20 920, 2022

    2022

  40. [48]

    Darts: Differentiable archi- tecture search,

    H. Liu, K. Simonyan, and Y . Yang, “Darts: Differentiable archi- tecture search,” in International Conference on Learning Repre- sentations, 2018

    2018

  41. [49]

    The concrete distri- bution: A continuous relaxation of discrete random variables,

    C. J. Maddison, A. Mnih, and Y . W. Teh, “The concrete distri- bution: A continuous relaxation of discrete random variables,” in International Conference on Learning Representations, 2022

    2022

  42. [50]

    Estimating or propa- gating gradients through stochastic neurons for conditional com- putation,

    Y . Bengio, N. L ´eonard, and A. Courville, “Estimating or propa- gating gradients through stochastic neurons for conditional com- putation,” arXiv preprint arXiv:1308.3432, 2013

    2013 arXiv

  43. [51]

    LibriSpeech: an asr cor- pus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey et al., “LibriSpeech: an asr cor- pus based on public domain audio books,” in ICASSP, 2015

    2015

  44. [52]

    Superb: Speech pro- cessing universal performance benchmark,

    S. wen Yang, P.-H. Chi, Y .-S. Chuanget al., “Superb: Speech pro- cessing universal performance benchmark,” in Interspeech 2021, 2021, pp. 1194–1198

    2021

  45. [53]

    Distilling HuBERT with LSTMs via decoupled knowledge distillation,

    D. de Oliveira and T. Gerkmann, “Distilling HuBERT with LSTMs via decoupled knowledge distillation,” in ICASSP, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.