Pith. sign in

REVIEW 4 major objections 6 minor 48 references

A correlation-permutation approach for speech-music encoders model merging

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A correlation-permutation scheme merges two independently trained audio encoders into one unified model with no retraining.

desk verdict A real downstream extension of Git Re-Basin to audio transformers, but the headline gain is misreported and the winning configuration was selected on the evaluation benchmark itself. read the letter →

arxiv 2506.11403 v1 pith:WE2PJXJA submitted 2025-06-13 cs.SD cs.AIeess.AS

Fabian Ritter-Gutierrez , Yi-Cheng Lin , Jeremy H.M Wong , Hung-yi Lee , Eng Siong Chng , Nancy F. Chen This is my paper · ORCID
classification cs.SDcs.AIeess.AS
keywords modelmergingpermutationalignmentcross-correlationself-supervisedspeechmodelsMERTHuBERTunifiedaudioencodertransformerlayer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that two audio encoders trained independently on different data and objectives—HuBERT for speech, MERT for music—can be fused into one encoder by reordering the music model's internal channels to match the speech model's, layer by layer, before averaging the weights. The authors report that the merged model keeps nearly all of HuBERT's speech ability (speech score 997.72 versus 1000) while gaining a large music boost (music score 917.58 versus 885.63), for an average score of 957.65. The gain is achieved without any pre-training, only a calibration pass over 10,000 audio clips. If correct, the approach offers a cheap way to build unified audio models from existing specialist encoders instead of training from scratch.

What carries the argument

The load-bearing object is a per-layer permutation matrix $\pi^*_l$, one for each chosen activation point, computed by solving an objective that maximizes the summed cross-correlation across feature channels between MERT's and HuBERT's activations, with temporal and batch dimensions collapsed into samples. The assignment is solved with the Jonker-Volgenant algorithm, and the inverse permutation is applied at each layer before the next activation is computed, so each layer sees clean statistics. For transformer blocks, the permutation is hierarchical: first reorder the $h$ attention heads, then permute features inside each matched head, and permute only $W_2$ of the feed-forward block; this preserves the functional pairing of query, key, and value within a head and avoids destructive averaging.

What would settle it

Take the proposed pipeline but choose the set of permuted components on a held-out validation split of the SUPERB/MARBLE tasks (or hold out several tasks entirely), then compare the final average score against the 0.9 HuBERT + 0.1 MERT linear-interpolation baseline; if the advantage disappears or reverses, the reported gain is selection-dependent rather than a property of correlation-permutation alignment.

Watch

Extended reading notes

Core claim

The central claim is that maximizing feature-wise cross-correlation between the activations of two independently trained encoders, layer by layer, finds a permutation that places the music encoder (MERT) into the same loss basin as the speech encoder (HuBERT), after which a simple linear interpolation at 0.9/0.1 weights produces a merged encoder that outperforms either specialist on average. The best configuration permutes the CNN channels, the ordering of attention heads, the channels within each reordered head, and the second feed-forward matrix $W_2$, while leaving query/key/value matrices unpermuted. The authors report an average SUPERB/MARBLE score of 957.65, with speech 997.72 and music 917.58; by the table's numbers, this is 14.83 points above the HuBERT baseline and 19.54 points above the weighted linear-interpolation baseline (the abstract credits the 14.83-point figure to the comparison against linear interpolation).

Load-bearing premise

The choice of which layers and components to permute (the 'CNN + fnn+attn' configuration) was made after looking at results on the same downstream benchmark used to report the final scores, so the claimed edge over simple weighted interpolation could depend on that choice rather than on the permutation principle itself.

Editorial extensions

If this is right

  • A unified speech-music encoder can be assembled from two publicly available self-supervised encoders with no gradient updates.
  • Music understanding tasks improve substantially (e.g., SingID 70.25 to 72.74, VocID 61.32 to 63.98, ESC50 70.20 to 71.69) while speech tasks remain near HuBERT levels.
  • Permuting all transformer parameters is worse than the targeted subset, so the recommended recipe is the 'CNN + fnn+attn' configuration.
  • The framework is stated to extend to other same-architecture encoder pairs, such as Whisper variants.
  • The first CNN layer needs far fewer reorderings (30.86% of channels) than deeper layers, suggesting the two models share low-level acoustic features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 14.83-point gain reported in the abstract is actually the gain over HuBERT, not over weighted linear interpolation, which per Table I is 19.54 points; readers comparing methods should use a consistent baseline.
  • The choice of which components to permute was made after examining results on the same downstream benchmark used for the final scores, so the advantage over linear interpolation may shrink if that choice is made on held-out tasks.
  • The 10,000-clip calibration set appears sufficient for this pair, but its sensitivity to size and domain mix is untested; a natural extension is to measure how the merged score varies with calibration data.
  • The method is demonstrated only for HuBERT and MERT, so claims about other same-architecture pairs remain extrapolations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a correlation-permutation method for merging two independently trained speech and music encoders, HuBERT and MERT, which share the same architecture but were trained on different data and objectives. The method computes per-layer permutation matrices that maximize channel-wise cross-correlation between the two models' feature activations on a calibration set, then linearly interpolates the permuted MERT weights with HuBERT weights at a 0.1/0.9 ratio. The merged model is evaluated on SUPERB speech tasks and MARBLE music tasks, plus ESC50. The best configuration, 'CNN + fnn+attn' (ID 5 in Table I), yields an average score of 957.65, compared with 938.11 for the 0.9 HuBERT + 0.1 MERT linear interpolation baseline and 942.82 for HuBERT alone.

Significance. If the result holds, the paper makes a useful practical contribution: it extends permutation-based model merging beyond same-initialization fine-tuned models to independently pretrained transformer encoders, and it evaluates downstream task performance rather than only loss barriers. The paper also provides component-wise ablations, a layer-wise analysis of permutation rates, and statistical significance tests on the downstream tasks. The use of a calibration set that is separate from the downstream task labels is a positive feature of the experimental design. However, the central claim that the proposed method beats linear interpolation is weakened by the fact that the winning component configuration was selected using the same benchmark on which the final scores are reported, and by an arithmetic error in the abstract.

major comments (4)
  1. [Abstract and Table I] The abstract states that the method achieves 'an improvement of 14.83 points in average score compared to linear interpolation model merging.' This is not what Table I shows. The average score of the proposed method (ID 5) is 957.65, while the linear interpolation baseline (ID 4) is 938.11, so the improvement over linear interpolation is 19.54 points. The value 14.83 is the improvement over the HuBERT baseline (942.82 to 957.65). The abstract should be corrected to name the correct comparison base.
  2. [Section V.B and Table I] The choice of the permutation configuration S is not independent of the reported evaluation. The best configuration, 'CNN + fnn+attn' (ID 5), was selected after comparing ablations on the same SUPERB/MARBLE benchmark used for the headline scores. Table I shows that the alternative 'CNN + all' configuration (ID 8) scores 928.49, which is below the 938.11 linear interpolation baseline (ID 4). With at least five configurations explored (IDs 5-9), the best one is expected to look better than average even if the true advantage over linear interpolation were small or zero. The claim that correlation-permutation merging beats linear interpolation therefore needs support from a held-out validation split or an explicit multiple-comparison correction. In addition, Section III.C states that excluding W1 was 'based on preliminary experiments,' but those experiments are not reported; they should be described or moved into the ablation study.
  3. [Section III.A, Eq. (1)] Equation (1) is written as an objective over the weight parameters θA_l and θB_l, but the text immediately following it and the entire method description in Sections III.B and III.C say that the objective is computed on feature activations. The notation is therefore inconsistent: the equation as printed is not the objective that is actually optimized. The authors should rewrite Eq. (1) using activation functions f_l(x; θA) and f_l(x; θB), or clearly define θA_l and θB_l as the feature activations at layer l, not the weight matrices.
  4. [Table I and Section V.B] The aggregate speech, music, and average scores are reported as single point estimates with no confidence intervals or variance across runs. Given the custom normalized scoring in Eq. (6), small differences such as the speech score of 997.72 versus HuBERT's 1000 may be within evaluation noise. The McNemar tests are informative for per-task comparisons, but they do not account for the fact that the winning configuration was selected after inspecting the same table. Reporting error bars, or at least explicitly stating that the scores are single-run point estimates, would make the strength of the claimed improvement clearer.
minor comments (6)
  1. [Abstract] The phrase 'an unified audio model' should be 'a unified audio model.'
  2. [Section III.C, step 2] The description of 'Permuting Features within Aligned Heads' is vague; the authors should specify exactly which weight matrices are permuted for the head-internal channel permutation and whether the output projection W^O is correspondingly permuted.
  3. [Section V.B, item 3] The text refers to 'row 10' when discussing the 'fnn+attn' configuration, but the table row is labeled ID 9; the numbering should be made consistent.
  4. [Section V.C] The typo 'CNN + "fnn+atnn"' should be corrected to 'CNN + "fnn+attn"'.
  5. [Section IV.B, Eq. (6)] The sentence introducing Eq. (6) contains a grammatical error ('we employs'); it should be 'we employ.' In addition, the notation s_t(BEST) would benefit from a brief explanation of how it is computed when HuBERT and MERT tie on a task.
  6. [Table II] The row label 'Transformer Attn Output' is not defined in the text; it should be clarified whether this refers to the concatenated multi-head attention output before W^O or to the per-head outputs after permutation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: permutation alignment is fit on separate calibration data and the merged-model scores are empirical, not constructed from the method's inputs.

full rationale

The correlation-permutation pipeline is not circular. Equation (1) solves for permutation matrices from feature activations of a 10,000-sample calibration set (Music4All plus LibriSpeech-100, Section IV.A); those activations are collected before any downstream evaluation. The downstream scores in Table I come from the SUPERB/MARBLE tasks, which are not used to solve Eq. (1) and do not feed back into the merge. The interpolation weight 0.9/0.1 is taken from the authors' prior work [3] (Section IV.A), but this is a minor self-citation that supplies a hyperparameter and evaluation configuration, not the load-bearing premise: the same weight is used for both the linear-interpolation baseline (ID 4) and the proposed method (ID 5), so the comparison is internally controlled. The choice of the component set S (CNN + ffn+attn) was made after inspecting ablations on the same benchmark (Section V.B), which is a model-selection and multiple-comparisons concern rather than a circular derivation; the paper does not define S in terms of the downstream scores. The abstract's statement that the gain is 'compared to linear interpolation model merging' is numerically inconsistent with Table I (the margin over ID 4 is 19.54 points, while 14.83 is the gain over HuBERT), but that is a reporting error, not an equivalence between input and output. Finally, the score formula in Eq. (6) normalizes by the best of HuBERT/MERT, but that is a metric choice and is never used to compute the permutations or the merged weights. The central claim is therefore self-contained relative to its stated assumptions; no prediction reduces to a fitted parameter or to a self-citation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method relies on a small number of design choices (interpolation weight, permutation target set, calibration set size) and on assumptions borrowed from prior permutation-merging work. No new model entities are introduced; the key free parameter is which components to permute, and it is selected based on the evaluation benchmark.

free parameters (3)
  • Interpolation weight (HuBERT) = 0.9
    The merged weights are 0.9*HuBERT + 0.1*MERT, taken from prior work [3] without tuning. It affects all reported scores; a different weight could change the ranking of configurations.
  • Permuted component set S = CNN layers, attention head order, head channels, FFN W2
    Which components to permute is the core design choice; the best set was selected by comparing ablations on the evaluation benchmark itself (Section V.B).
  • Calibration dataset size = 5,000 samples per domain
    5000 Music4All and 5000 LibriSpeech samples used to compute permutations; no sensitivity analysis reported.
assumptions (4)
  • domain assumption HuBERT and MERT share identical base architecture
    The method requires matching weight tensor shapes for permutation and interpolation; this holds for HuBERT-base and MERT-base.
  • ad hoc to paper Channel-wise cross-correlation of activations is a valid proxy for functional alignment
    The objective (Eq. 1) assumes maximizing feature cross-correlation yields permutations that lead to low-loss interpolation; this is carried over from Git Re-Basin and ZipIt without re-validation on audio.
  • domain assumption Reshaping activations to R^{B*Cl*Tl} -> R^{Cl*(B*Tl)} preserves channel semantics
    Section III.B asserts temporal/batch aggregation is justified by CNN translation invariance; this may not hold for deeper transformer layers where temporal structure matters.
  • standard math Inverse permutation un-permutes activations correctly during calibration
    Section III.A states the inverse permutation is applied to keep statistics clean; this is a correct application of permutation matrices, assuming the forward and inverse are consistent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A correlation-permutation approach for speech-music encoders model merging." pith.science (2026). https://pith.science/paper/WE2PJXJA

@misc{pith2026250611403,
  author       = {Pith},
  title        = {Pith review of: A correlation-permutation approach for speech-music encoders model merging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WE2PJXJA}},
  note         = {Machine review of arXiv:2506.11403}
}
read the original abstract

Creating a unified speech and music model requires expensive pre-training. Model merging can instead create an unified audio model with minimal computational expense. However, direct merging is challenging when the models are not aligned in the weight space. Motivated by Git Re-Basin, we introduce a correlation-permutation approach that aligns a music encoder's internal layers with a speech encoder. We extend previous work to the case of merging transformer layers. The method computes a permutation matrix that maximizes the model's features-wise cross-correlations layer by layer, enabling effective fusion of these otherwise disjoint models. The merged model retains speech capabilities through this method while significantly enhancing music performance, achieving an improvement of 14.83 points in average score compared to linear interpolation model merging. This work allows the creation of unified audio models from independently trained encoders.

Figures

Figures reproduced from arXiv: 2506.11403 by the authors.

Figure 1
Figure 1. Feature-wise Permutation Framework for Merging HuBERT and MERT. (a) Permutation Computation: For each layer i, compute a permutation π ∗ i that maximizes channel-wise cross-correlation between MERT’s feature activations (blue) and HuBERT’s feature activations (red). The toy 4×4 correlation matrices illustrate raw correlations (Channel-wise) on the left and reordered, near-diagonal correlations (Maximized Channel-wis… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 39 canonical work pages

  1. [1]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021

    2021

  2. [2]

    MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training,

    Y . Li, R. Yuan, G. Zhang, Y . Ma, X. Chen, H. Yin, C.-L. Lin, A. Ragni, E. Benetos, N. Gyenge, R. B. Dannenberg, R. Liu, W. Chen, G. G. Xia, Y . Shi, W.-F. Huang, Y .-T. Guo, and J. Fu, “MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training,” International Conference on Learning Representations (ICLR), 2024

    work page 2024

  3. [3]

    Distilling a speech and music encoder with task arithmetic,

    F. Ritter-Gutierrez, Y .-C. Lin, J.-C. Wei, J. H. M. Wong, E. S. Chng, N. F. Chen, and H. yi Lee, “Distilling a speech and music encoder with task arithmetic,” inInterspeech, 2025

    work page 2025

  4. [4]

    Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,

    C.-y. Huang, W.-C. Chen, S.-w. Yang, A. T. Liu, C.-A. Li, Y .-X. Lin, W.- C. Tseng, A. Diwan, Y .-J. Shih, J. Shiet al., “Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,”International Conference on Learning Representations (ICLR), 2025

    work page 2025

  5. [5]

    Efficient training of self-supervised speech foundation models on a compute budget,

    A. T. Liu, Y .-C. Lin, H. Wu, S. Winkler, and H.-Y . Lee, “Efficient training of self-supervised speech foundation models on a compute budget,” in2024 IEEE Spoken Language Technology Workshop (SLT), 2024, pp. 961–968

    work page 2024

  6. [6]

    Editing models with task arithmetic,

    G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi, “Editing models with task arithmetic,”The Eleventh International Conference on Learning Representations (ICLR), 2023

    work page 2023

  7. [7]

    Ties- merging: Resolving interference when merging models,

    P. Yadav, D. Tam, L. Choshen, C. A. Raffel, and M. Bansal, “Ties- merging: Resolving interference when merging models,”Advances in Neural Information Processing Systems, vol. 36, 2024

    work page 2024

  8. [8]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch,

    L. Yu, B. Yu, H. Yu, F. Huang, and Y . Li, “Language models are super mario: Absorbing abilities from homologous models as a free lunch,” in International Conference on Machine Learning (ICML), 2024

    work page 2024

Show all 48 references
  1. [9]

    Task arithmetic for language expansion in speech translation,

    Y .-F. Cheng, H. Futami, Y . Kashiwagi, E. Tsunoo, W. S. Teo, S. Arora, and S. Watanabe, “Task arithmetic for language expansion in speech translation,”arXiv preprint arXiv:2409.11274, 2024

    2024 arXiv

  2. [10]

    Task arithmetic can mitigate synthetic-to-real gap in automatic speech recognition,

    H. Su, H. Farn, F.-Y . Sun, S.-T. Chen, and H.-y. Lee, “Task arithmetic can mitigate synthetic-to-real gap in automatic speech recognition,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, E...

    2024

  3. [11]

    An attribute interpolation method in speech synthesis by model merging,

    M. Murata, K. Miyazaki, and T. Koriyama, “An attribute interpolation method in speech synthesis by model merging,” inInterspeech 2024, 2024

    2024

  4. [12]

    Selective attention merging for low resource tasks: A case study of child asr,

    N. B. Shankar, Z. Wang, E. Eren, and A. Alwan, “Selective attention merging for low resource tasks: A case study of child asr,”ICASSP, 2025

    2025

  5. [13]

    The role of permutation invariance in linear mode connectivity of neural networks,

    R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur, “The role of permutation invariance in linear mode connectivity of neural networks,” inInternational Conference on Learning Representations (ICLR), 2022

    2022

  6. [14]

    Git re-basin: Merging models modulo permutation symmetries,

    S. K. Ainsworth, J. Hayase, and S. S. Srinivasa, “Git re-basin: Merging models modulo permutation symmetries,”International Conference on Learning Representations (ICLR), 2023

    2023

  7. [15]

    Zipit! merging models from different tasks without training,

    G. Stoica, D. Bolya, J. B. Bjorner, T. N. Hearn, and J. Hoffman, “Zipit! merging models from different tasks without training,”International Conference on Learning Representations (ICLR), 2024

    2024

  8. [16]

    Merging text transformer models from different initializations,

    N. Verma and M. Elbayad, “Merging text transformer models from different initializations,”Transactions on Machine Learning Research (TMLR), 2024

    2024

  9. [17]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” inAdvances in Neural Information Processing Systems, 2017

    2017

  10. [18]

    Robust speech recognition via large-scale weak super- vision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak super- vision,” inICML, 2023

    2023

  11. [19]

    Best-first model merging for hidden markov model induction,

    A. Stolcke and S. M. Omohundro, “Best-first model merging for hidden markov model induction,”arXiv preprint, 1994

    1994

  12. [20]

    State-dependent phoneme-based model merging for dialectal chinese speech recognition,

    L. Liu, T. F. Zheng, and W. Wu, “State-dependent phoneme-based model merging for dialectal chinese speech recognition,”Speech Communica- tion, 2008

    2008

  13. [21]

    A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (rover),

    J. G. Fiscus, “A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (rover),” in1997 IEEE work- shop on automatic speech recognition and understanding proceedings. IEEE, 1997, pp. 347–354

    1997

  14. [22]

    Speech recognition system combination for machine translation,

    M. J. Gales, X. Liu, R. Sinha, P. C. Woodland, K. Yu, S. Matsoukas, T. Ng, K. Nguyen, L. Nguyen, J.-L. Gauvainet al., “Speech recognition system combination for machine translation,” in2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP’07, vol...

    2007

  15. [23]

    Combination of end-to-end and hybrid models for speech recognition

    J. H. M. Wong, Y . Gaur, R. Zhao, L. Lu, E. Sun, J. Li, and Y . Gong, “Combination of end-to-end and hybrid models for speech recognition.” inInterspeech, 2020, pp. 1783–1787

    2020

  16. [24]

    Ensembles of hybrid and end-to-end speech recognition

    A. K. Parikh, L. ten Bosch, and H. van den Heuvel, “Ensembles of hybrid and end-to-end speech recognition.” inProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), N. Calzolari, M.-Y . Kan, V ...

    2024

  17. [25]

    Combining multiple speech recognizers using voting and language model information,

    H. Schwenk and J.-L. Gauvain, “Combining multiple speech recognizers using voting and language model information,” in6th International Conference on Spoken Language Processing (ICSLP 2000), 2000, pp. vol. 2, 915–918

    2000

  18. [26]

    Neural network ensembles,

    L. K. Hansen and P. Salamon, “Neural network ensembles,”IEEE transactions on pattern analysis and machine intelligence, vol. 12, no. 10, pp. 993–1001, 1990

    1990

  19. [27]

    Checkpoint ensembles: En- semble methods from a single training process

    H. Chen, S. Lundberg, and S. Lee, “Checkpoint ensembles: En- semble methods from a single training process.”arXiv preprint arXiv:1710.03282, 2017

    2017 arXiv

  20. [28]

    What is being transferred in transfer learning?

    B. Neyshabur, H. Sedghi, and C. Zhang, “What is being transferred in transfer learning?”Advances in neural information processing systems, vol. 33, pp. 512–523, 2020

    2020

  21. [29]

    On implementing 2d rectangular assignment algorithms,

    D. F. Crouse, “On implementing 2d rectangular assignment algorithms,” IEEE Transactions on Aerospace and Electronic Systems, vol. 52, no. 4, pp. 1679–1696, 2016

    2016

  22. [30]

    Group normalization,

    Y . Wu and K. He, “Group normalization,” inProceedings of the European Conference on Computer Vision (ECCV), September 2018

    2018

  23. [31]

    Gaussian error linear units (gelus),

    D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” 2023

    2023

  24. [32]

    What does BERT look at? An analysis of BERT’s attention,

    K. Clark, U. Khandelwal, O. Levy, and C. D. Manning, “What does BERT look at? An analysis of BERT’s attention,” inProceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics, 2019, pp. 275–286

    2019

  25. [33]

    Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned,

    E. V oita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov, “Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned,” inProceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL). Association for ...

    2019

  26. [34]

    Music4all: A new music database and its applications,

    I. A. P. Santana, F. Pinhelli, J. Donini, L. G. Catharin, R. B. Mangolin, Y . M. G. Costa, V . D. Feltrim, and M. A. Domingues, “Music4all: A new music database and its applications,”2020 International Conference on Systems, Signals and Image Processing (IWSSIP), pp. 399–404, 2020

    2020

  27. [35]

    Librispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” inICASSP, 2015, pp. 5206–5210

    2015

  28. [36]

    Superb: Speech processing universal performance benchmark,

    S. wen Yang, P.-H. Chi, Y .-S. Chuang, C.-I. Lai, K. Lakhotia, Y . Y . Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T. hsien Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. rahman Mohamed, and H. yi Lee, “Superb: Speech processing univer...

    2021

  29. [37]

    Marble: Music audio representation benchmark for universal evaluation,

    R. Yuan, Y . Ma, Y . Li, G. Zhang, X. Chen, H. Yin, Y . Liu, J. Huang, Z. Tian, B. Denget al., “Marble: Music audio representation benchmark for universal evaluation,”Advances in Neural Information Processing Systems, 2023

    2023

  30. [38]

    Speech commands: A dataset for limited-vocabulary speech recognition,

    P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,”ArXiv, 2018

    2018

  31. [39]

    Speech model pre-training for end-to-end spoken language understanding,

    L. Lugosch, M. Ravanelli, P. Ignoto, V . S. Tomar, and Y . Bengio, “Speech model pre-training for end-to-end spoken language understanding,” in Interspeech 2019, 2019, pp. 814–818

    2019

  32. [40]

    V oxceleb: Large- scale speaker verification in the wild,

    A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, “V oxceleb: Large- scale speaker verification in the wild,”Computer Speech & Language, 2020

    2020

  33. [41]

    Iemocap: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,”Language resources and evaluation, vol. 42, pp. 335–359, 2008

    2008

  34. [42]

    V ocalset: A singing voice dataset

    J. Wilkins, P. Seetharaman, A. Wahl, and B. Pardo, “V ocalset: A singing voice dataset.” inISMIR, 2018, pp. 468–474

    2018

  35. [43]

    Neural audio synthesis of musical notes with wavenet autoencoders,

    J. Engel, C. Resnick, A. Roberts, S. Dieleman, M. Norouzi, D. Eck, and K. Simonyan, “Neural audio synthesis of musical notes with wavenet autoencoders,” inInternational conference on machine learning. PMLR, 2017, pp. 1068–1077

    2017

  36. [44]

    Musical genre classification of audio signals,

    G. Tzanetakis and P. Cook, “Musical genre classification of audio signals,”IEEE Transactions on speech and audio processing, vol. 10, no. 5, pp. 293–302, 2002

    2002

  37. [45]

    ESC: Dataset for Environmental Sound Classification,

    K. J. Piczak, “ESC: Dataset for Environmental Sound Classification,” in Proceedings of the 23rd Annual ACM Conference on Multimedia. ACM Press, 2015, pp. 1015–1018

    2015

  38. [46]

    Ml-superb: Multi- lingual speech universal performance benchmark,

    J. Shi, D. Berrebbi, W. Chen, H.-L. Chung, E.-P. Hu, W. P. Huang, X. Chang, S.-W. Li, A. Mohamed, H.-y. Leeet al., “Ml-superb: Multi- lingual speech universal performance benchmark,”Interspeech, 2023

    2023

  39. [47]

    Layer-wise analysis of a self- supervised speech representation model,

    A. Pasad, J.-C. Chou, and K. Livescu, “Layer-wise analysis of a self- supervised speech representation model,”2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 914–921, 2021

    2021

  40. [48]

    Comparative layer-wise analysis of self-supervised speech models,

    A. Pasad, B. Shi, and K. Livescu, “Comparative layer-wise analysis of self-supervised speech models,” inICASSP. IEEE, 2023, pp. 1–5

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.