Robust fine-tuning of speech recognition models via model merging: application to disordered speech

Alexandre Ducorroy; Rachid Riad

REVIEW 4 major objections 8 minor 40 references

Robust fine-tuning of speech recognition models via model merging: application to disordered speech

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Averaging several fine-tuned Whisper models cuts word-error rate on disordered speech by 12 percent over standard fine-tuning.

desk verdict A workmanlike application of known model-merging tricks to dysarthric ASR with encouraging held-out test1 numbers, but the headline dev-set gains are inflated by selection on the same labels and the fine-tuning baseline is too underspecified to trust the magnitude. read the letter →

arxiv 2505.20477 v1 pith:WCMHHM36 submitted 2025-05-26 eess.AS cs.SD

Alexandre Ducorroy , Rachid Riad This is my paper

classification eess.AScs.SD

keywords modelmergingweightaveragingfine-tuningdysarthricspeechautomaticrecognitionWhisperAccessibilityProjectlow-resourceASR

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that weight averaging of Whisper models fine-tuned on dysarthric speech outperforms standard single-model fine-tuning, and that the gain grows as more fine-tuned models are merged. On the Speech Accessibility Project benchmark, averaging models from multiple independent fine-tuning runs reduced word error rate by 12% relative to classic fine-tuning, and by 16.2% on long-form audio, the hardest segment for disordered speech. The authors argue that merging is a practical adaptation method because it needs no extra hyperparameter search, costs nothing at inference, and helps even with only one or ten hours of training data. If the claim holds, model merging offers a practical way to reduce errors in assistive speech recognition for people with communication disabilities.

What carries the argument

The load-bearing mechanism is weight averaging in parameter space. Because all models share Whisper's pre-trained initialization, their fine-tuned weights are permutation-aligned, so a simple arithmetic mean of parameters, $\theta = \frac{1}{n}\sum_k \theta_k$, produces a usable model. The method exploits linear mode connectivity: models sharing a pre-training trajectory can be interpolated without a loss spike, so the average lands in a low-loss basin that generalizes better than individual checkpoints. The paper's three variants — MAST (averaging checkpoints along one trajectory), MAcT (averaging best checkpoints across hyperparameter-diverse trajectories), and SMAcT (greedily adding a candidate only if it lowers dev WER) — all use this same averaging operation.

What would settle it

Re-run the same experiment on the SAP development set with a carefully tuned single-model baseline: perform the same hyperparameter search and early stopping on the dev subset, then compare its WER with MAcT/SMAcT using the same number of checkpoints and the same compute budget. If the tuned baseline reaches or beats the merged WER, the central claim is refuted; otherwise it stands.

Watch

Extended reading notes

Core claim

The paper's central discovery is that fine-tuning Whisper large-v3 on the SAP dysarthric speech corpus and then averaging the weights of the resulting checkpoints yields a model that transcribes disordered speech better than the model produced by classic fine-tuning. Averaging checkpoints saved along one fine-tuning trajectory (MAST) improves WER from 15.0 to 13.9 on the development set; averaging the best checkpoints of 30 runs with varied hyperparameters (MAcT) reaches 13.2, a 12% relative reduction; the selectively pruned variant (SMAcT) matches 13.2 on dev and gives 10.8 WER on held-out test1. The same procedure improves long utterances over 30 seconds by 16.2% relative, remains beneficial with 1 or 10 hours of training data, and transfers to smaller Whisper architectures, though gains are smaller there.

Load-bearing premise

The measured gains assume the classic fine-tuning baseline was trained with comparable care and budget; the paper does not report the baseline's hyperparameters, number of runs, or early-stopping choices, so part of the improvement could come from an under-tuned baseline rather than from merging.

Editorial extensions

If this is right

Merging more fine-tuned checkpoints continuously lowers WER on dysarthric speech, with no saturation reported between 1 and 30 merged models.
Merging helps most on long-form audio over 30 seconds, where standard fine-tuning struggles; MAcT gives a 16.2% relative WER reduction in that segment.
The gains persist in low-data settings: a selective merge on 1 hour of training data improves WER from 21.2 to 19.0, and on 10 hours from 18.5 to 17.1.
The technique transfers to Whisper base and large-v3-turbo, producing large gains on base (43.6 to 31.4 WER with MAST), though the benefit scales with model size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One implication the authors leave implicit is that weight-space merging is prediction-agnostic, so it could be combined with test-time ensembling or language-model rescoring, and the two gains may stack.
If linear mode connectivity is the cause, merging should also help on other high-variability, low-resource speech domains such as child speech, accented speech, or vocal pathology; the paper's cited child-ASR result already points in that direction.
Because the dev subset used to choose merges includes all 553 audios longer than 30 seconds, the selective method is partly tuned to long-form performance; a testable extension is to repeat selection on a balanced dev subset to see whether long-form gains trade off against short-utterance accuracy.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a legitimate, useful extension of model merging (SWA, model soups, greedy selection) to Whisper fine-tuning on dysarthric speech from the Speech Accessibility Project. The held-out test1 results are genuinely encouraging - SMAcT gets 10.8 WER vs 14.5 for a single fine-tuned model - and the low-data ablations (1h, 10h) are the kind of practical evidence that makes the paper worth reading. The authors also report the challenge ranking and are honest that they did not beat the top teams.

What is actually new: the application to dysarthric speech, the focus on long-form audio as a major error source, and the ablation showing gains persist with very little training data. The techniques themselves are standard. That is fine; not every paper needs to invent a new method.

Now the soft spots, in proportion. The stress-test note is right: the headline 12% and 16.2% relative gains are computed on a dev subset that was also used to select which checkpoints to merge (MAcT picks best per trajectory; SMAcT greedily adds models based on dev WER). That overlap can inflate gains. What mitigates it: the held-out test1 improvement is actually larger than the dev improvement, so the direction is probably real. What does not mitigate it: the fine-tuned baseline is a single run with unreported hyperparameters, number of runs, or early stopping. A weak baseline inflates any improvement. Also, there are no error bars or significance tests anywhere, so we cannot tell whether a 0.5-point WER difference means anything.

The paper also overclaims. The abstract says merging "consistently improves" ASR, but Table 4 shows Turbo MAcT at 20.7 vs fine-tuned at 20.8 - essentially nothing. And "generalized across model architectures" is too strong; the Base model shows a big MAST gain but MAcT is worse than MAST, and the Turbo results are flat. Those are minor overstatements, but they should be fixed.

Who is this for? Practitioners fine-tuning Whisper for disordered speech or other low-resource domains, and researchers working on ASR adaptation. They will get a cheap, plausible recipe and a useful baseline comparison. The paper deserves a serious referee - not a desk reject - because the held-out challenge scores give it external validity. But it needs a revision: report baseline details, add error bars or at least multiple runs, separate the selection and evaluation sets, and tone down the architecture generalization claim. I would take it to review, and I would ask for those changes before accepting.

Referee Report

4 major / 8 minor

Summary. The paper proposes to improve Whisper-based ASR for disordered speech by merging fine-tuned model weights instead of using a single fine-tuned model. Three strategies are compared: MAST (averaging checkpoints along one fine-tuning trajectory), MAcT (averaging best checkpoints across multiple fine-tuning runs with varied hyperparameters), and SMAcT (greedily adding checkpoints to the average when doing so reduces WER). Experiments on the SAP challenge data report dev-set WER improvements over standard fine-tuning (15.0 -> 13.9/13.2/13.2), a 16.2% relative improvement on long audios, better held-out test1 WER for MAcT and SMAcT, plus ablations on training set size and on Whisper base/large-v3-turbo architectures. The central claim is that multi-run merging is a simple, inference-cost-free way to consistently improve disordered-speech ASR.

Significance. If the central claim holds, the paper offers a practical and easily reproducible adaptation technique for a challenging real-world population, requiring no extra inference cost and no additional hyperparameter search. The paper's strengths are its use of the SAP challenge's held-out test sets, the inclusion of low-data and cross-architecture ablations, and a clear algorithmic description of the merging procedures. However, the magnitude of the reported gains is not yet fully established: the same dev subset is used for model selection and for reporting final dev WERs, the fine-tuning baseline is underspecified, and the held-out confirmation rests mostly on one test set with a single baseline. These issues are fixable with additional experiments or careful re-analysis, so the work is promising but needs revision before the claims can be accepted at face value.

major comments (4)

[Section 3.2 and Section 4 (Figure 1 and Table 2)] The reported dev WERs are not independent of model selection. Candidate models for MAcT and SMAcT are selected using a 15% subset of the development set (Section 3.2), and Figure 1 reports WERs evaluated on a subset of the same development set; Section 4 then reports the final MAST/MAcT/SMAcT dev WERs of 13.9/13.2/13.2. Because the selection criterion and the evaluation metric are computed on overlapping labels, the headline 12% relative dev-set gain can partly reflect selection on the evaluation set rather than a general property of weight averaging. Please separate the selection subset from the evaluation subset, or report results on a truly held-out validation set for the main comparisons.
[Section 3.2 and Table 2] The classic fine-tuning baseline is underspecified. No information is given on how many fine-tuning runs were performed for the baseline, which checkpoint was selected, what early-stopping rule was used, or which hyperparameter values were used for the 'Fine-tuned' row. Without this, the relative improvement attributed to merging could be inflated by an under-tuned baseline. Please specify the full training protocol for the baseline, including the number of runs, validation-based checkpoint selection, and the exact hyperparameters, so that the comparison is fair and reproducible.
[Section 4 (Tables 2-4)] All WER and SEM values are single numbers without error bars, standard deviations, or significance tests. The claim that merging 'consistently improves' performance is weakened by Table 4, where Turbo MAcT (20.7) is essentially tied with fine-tuning (20.8), and by Table 3, where MAST on 1 hour of data (21.2) matches fine-tuning (21.2). Please provide variability estimates, such as standard deviations over independent training runs or bootstrap confidence intervals over test utterances, for at least the main comparisons.
[Table 2] The held-out evidence is more limited than the narrative suggests. Only SMAcT was submitted for evaluation on test2, with no fine-tuned baseline or other merging baseline on that set, so the test2 results cannot be attributed to merging. The central claim therefore rests mainly on test1, which is a single held-out test set with one fine-tuned baseline. Please provide the baseline and additional merging results on test2, or explicitly qualify the claim as supported by test1 only.

minor comments (8)

[Section 3.1, Eq. (1)] The index notation in Eq. (1) is inconsistent: the same symbol θi is used for a scalar parameter and for a whole model, and the summation runs from k=1 to n−1 while models are indexed from 0. Please rewrite it as θ_i^{avg} = (1/n) Σ_{k=0}^{n−1} θ_i^{(k)}.
[Section 4 and Table 2] The text says SMAcT 'consistently achieved the best results', but Table 2 shows MAcT and SMAcT with identical dev WER (13.2); please make the statement precise.
[Section 4 and Table 2] The text reports a final WER of 12.38 on test2, whereas Table 2 reports 12.36; please reconcile the two numbers.
[Figure 1] It is unclear whether the three numeric columns in Figure 1 correspond to Fine-tuned, MAST, and MAcT, or to Fine-tuned, MAST/MAcT, and SMAcT; the caption says 'We used 30 different checkpoints for each strategy' but SMAcT is not explicitly shown. Please clarify the methods displayed in the figure.
[Section 3.2] The description of the 15% development subset gives the number of long audios (553) but not the total number of utterances or the random sampling procedure/seed; please provide these details for reproducibility.
[Table 4] The table caption says 'training on the full set only on 6000 steps', but Section 3.1 states that the base model was fine-tuned for 7200 steps; please correct this inconsistency.
[Section 1] Reference [19] is duplicated in the phrase '[19, 19]'; one of the citations should presumably be a different reference.
[Various sections] There are several typos and wording issues, including 'Large mod eks' in Section 4, 'artchitectures' in Section 4, 'libary' in Section 3.2, and 'probabilites' in Section 3.1; a careful proofread is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical comparisons on held-out SAP test sets, with no fitted parameter renamed as a prediction and no load-bearing self-citation chain.

full rationale

The paper's central claim is an empirical comparison of model merging strategies against classic fine-tuning on dysarthric speech. The merging methods are evaluated on the SAP challenge's unshared test sets, which are external to the development set used for checkpoint selection. The model selection procedure uses a 15% subset of the development set (Section 3.1), and Figure 1 also reports WER on a development subset; this creates a mild selection-bias concern for the dev-set numbers, but it is not a circular derivation. The selection criterion is not definitionally equal to the reported quantity, the reported gains are not forced by construction, and the held-out test results (SMAcT 10.8 WER on test1 and 12.36 on test2, Table 2) provide independent evidence that the improvement is not merely an artifact of optimizing on the evaluation set. The comparison to classic fine-tuning may be affected by the underspecified baseline hyperparameters and training budget, but that is a robustness and reproducibility concern, not circularity. No self-citation is load-bearing: the cited prior work on model soups, diverse weight averaging, and linear mode connectivity is external to the authors. Under the requested circularity standards, there is no equation that reduces to its own input, no fitted parameter presented as a prediction, and no self-citation chain that forbids alternatives.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper does not introduce fitted constants or new entities. It relies on the pretrained Whisper model, the SAP dataset labels, and the empirical hypothesis that fine-tuned models from a shared initialization can be weight-averaged without performance loss. The hand-chosen hyperparameter ranges, number of merged models, dev subset composition, and decoding settings are config choices that affect the magnitude of reported gains.

free parameters (4)

Hyperparameter sampling ranges (learning rate, weight decay, dropout, SpecAugment) = lr 3e-6 to 8e-6; weight decay 0.07 to 0.2; dropouts and SpecAugment 0 to 0.1
Chosen by hand to generate diverse fine-tuning trajectories; the exact ranges affect the diversity and quality of merged models, and thus the reported WER gains.
Number of merged models = 30 for main results, 10 for ablations
A compute-limited choice; the paper shows WER decreases as more models are merged, so the number directly affects the magnitude of the reported improvement.
Dev subset used for model selection = ~15% of dev set, enriched with all 553 long audios
Used to select best checkpoints and for SMAcT inclusion decisions; its composition affects which models are merged and the reported dev WER.
Decoding parameters = temperature=0, length penalty=0.6
Fixed by hand for all transcription generation; affects absolute WER values and therefore the size of relative gains.

assumptions (3)

domain assumption Weight averaging across fine-tuned models from the same pretrained initialization yields a usable model.
The entire method assumes parameter correspondence and a connected low-loss basin; the paper invokes linear mode connectivity [30, 31] in Section 4 to explain why merging works.
domain assumption The SAP development set is a valid proxy for the held-out test distributions.
Models are selected on a 15% dev subset, and the paper reports final dev and test WER without analyzing distribution shift between dev and test.
domain assumption Whisper's pretrained weights and the chosen fine-tuning budget are appropriate for dysarthric speech.
Section 3.2 uses Whisper large-v3 and fixed step counts without validating domain fit or label quality beyond the provided benchmark.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Robust fine-tuning of speech recognition models via model merging: application to disordered speech." pith.science (2026). https://pith.science/paper/WCMHHM36

@misc{pith2026250520477,
  author       = {Pith},
  title        = {Pith review of: Robust fine-tuning of speech recognition models via model merging: application to disordered speech},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WCMHHM36}},
  note         = {Machine review of arXiv:2505.20477}
}

read the original abstract

Automatic Speech Recognition (ASR) has advanced with Speech Foundation Models (SFMs), yet performance degrades on dysarthric speech due to variability and limited data. This study as part of the submission to the Speech Accessibility challenge, explored model merging to improve ASR generalization using Whisper as the base SFM. We compared fine-tuning with single-trajectory merging, combining models from one fine-tuning path, and multi-run merging, merging independently trained models. Our best multi-run merging approach achieved a 12% relative decrease of WER over classic fine-tuning, and a 16.2% relative decrease on long-form audios, a major loss contributor in dysarthric ASR. Merging more and more models led to continuous gains, remained effective in low-data regimes, and generalized across model architectures. These results highlight model merging as an easily replicable adaptation method that consistently improves ASR without additional inference cost or hyperparameter tuning.

Figures

Figures reproduced from arXiv: 2505.20477 by the authors.

**Figure 1.** WER comparison between classical Fine-tuning vs Merging strategies of Whisper evaluated on subset of SAP development set. We used 30 different checkpoints for each strategy, and WER are reported for different lengths of audios. Librispeech clean dataset [13], their performance degrades significantly for individuals with articulatory impairments, such as those with dysarthria due to neurological disorders. Even wi… view at source ↗

**Figure 2.** Evolution of WER when merging models compared to single-model evaluations (black cross). The upper figure illustrates the WER progression when merging models along a single optimization trajectory, while the lower figure shows WER when merging models from different trajectories. In both figures, the black cross represents the WER of individual models used in the merging process. Training size 1h 10h Full set 3 Fr… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 32 canonical work pages

[1]

Robust fine-tuning of speech recognition models via model merging: application to disordered speech

Introduction Automatic Speech Recognition (ASR) has evolved from a spe- cialized tool confined to speech labs and research environments into a mainstream technology that can be deployed with just a few lines of code1. This transformation has been driven by the development of Speech Foundation Models (SFMs), which are trained on large and diverse datasets ...

work page Pith review arXiv 2025
[2]

SAP Challenge The SAP Dataset is divided into four subsets: a training set, a development set, and two unshared test sets (test1 and test2) used for evaluating submissions. The provided training dataset comprises speech recordings from 368 patients, totaling 289 hours of audio data and the de- velopment set consists of 53 patients, amounting to 43 hours o...

work page
[3]

Model Merging Model merging strategies have demonstrated performance im- provements in out-of-distribution predictions for vision models

Methods 3.1. Model Merging Model merging strategies have demonstrated performance im- provements in out-of-distribution predictions for vision models

work page
[4]

As shown in Fig- ure 1, MAST outperformed standard fine-tuning, demonstrat- ing the benefits of weight averaging along a single optimization 2https://github.com/openai/whisper path

Results and discussions We evaluated model merging strategies trained on the full train set, including MAST, MAcT and SMAcT. As shown in Fig- ure 1, MAST outperformed standard fine-tuning, demonstrat- ing the benefits of weight averaging along a single optimization 2https://github.com/openai/whisper path. MAcT further improved performance by integrating d...

work page
[5]

Our findings demonstrate that model merg- ing, particularly selective merging across multiple trajectories, significantly improves WER compared to traditional fine-tuning methods

Conclusions In this paper, we explored the potential of model merging techniques to enhance the performance of ASR systems for dysarthric speech. Our findings demonstrate that model merg- ing, particularly selective merging across multiple trajectories, significantly improves WER compared to traditional fine-tuning methods. This approach not only leverage...

work page
[6]

Benchmarking children’s asr with supervised and self-supervised speech foundation mod- els,

R. Fan, N. B. Shankar, and A. Alwan, “Benchmarking children’s asr with supervised and self-supervised speech foundation mod- els,” arXiv preprint arXiv:2406.10507, 2024

arXiv 2024
[7]

Whisper-at: Noise-robust automatic speech recognizers are also strong general audio event taggers,

Y . Gong, S. Khurana, L. Karlinsky, and J. Glass, “Whisper-at: Noise-robust automatic speech recognizers are also strong general audio event taggers,” in Interspeech, 2023

work page 2023
[8]

Data determines distributional robustness in contrastive language image pre-training (clip),

A. Fang, G. Ilharco, M. Wortsman, Y . Wan, V . Shankar, A. Dave, and L. Schmidt, “Data determines distributional robustness in contrastive language image pre-training (clip),” in International Conference on Machine Learning. PMLR, 2022, pp. 6216–6234

work page 2022

Show all 40 references

[9]

Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

2021
[10]

Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1505–1518, 2022

2022
[11]

wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,

A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,” Advances in neural information processing systems, 2020

2020
[12]

Towards better domain adaptation for self-supervised models: A case study of child asr,

R. Fan, Y . Zhu, J. Wang, and A. Alwan, “Towards better domain adaptation for self-supervised models: A case study of child asr,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1242–1252, 2022

2022
[13]

Robust speech recognition via large-scale weak supervision,

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

2023
[14]

Community-supported shared infrastructure in support of speech accessibility,

M. Hasegawa-Johnson, X. Zheng, H. Kim, C. Mendes, M. Dickin- son, E. Hege, C. Zwilling, M. M. Channell, L. Mattie, H. Hodges, L. Ramig, M. Bellard, M. Shebanek, L. Sar ι, K. Kalgaonkar, D. Frerichs, J. P. Bigham, L. Findlater, C. Lea, S. Herrlinger, P. Korn, S. Abou-Zahra, R. ...

2024
[15]

Improving domain generalization in speech emotion recognition with whisper,

E. Goron, L. Asai, E. Rut, and M. Dinov, “Improving domain generalization in speech emotion recognition with whisper,” in ICASSP, 2024

2024
[16]

Trans- ferring speech-generic and depression-specific knowledge for alzheimer’s disease detection,

Z. Cui, W. Wu, W.-Q. Zhang, J. Wu, and C. Zhang, “Trans- ferring speech-generic and depression-specific knowledge for alzheimer’s disease detection,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023

2023
[17]

Automated speech analysis for risk detection of depression, anxiety, insomnia, and fatigue: Algorithm development and validation study,

R. Riad, M. Denais, M. de Gennes, A. Lesage, V . Oustric, X. N. Cao, S. Mouchabac, and A. Bourla, “Automated speech analysis for risk detection of depression, anxiety, insomnia, and fatigue: Algorithm development and validation study,”Journal of Medical Internet Research, 2024

2024
[18]

To- wards inclusive automatic speech recognition,

S. Feng, B. M. Halpern, O. Kudina, and O. Scharenborg, “To- wards inclusive automatic speech recognition,” Computer Speech & Language, vol. 84, p. 101567, 2024

2024
[19]

Disordered speech data collection: Lessons learned at 1 million utterances from project euphonia

R. L. MacDonald, P.-P. Jiang, J. Cattiau, R. Heywood, R. Cave, K. Seaver, M. A. Ladewig, J. Tobin, M. P. Brenner, P. C. Nel- son et al., “Disordered speech data collection: Lessons learned at 1 million utterances from project euphonia.” in Interspeech, vol. 2021, 2021, pp. 4833–4837

2021
[20]

Thus, we now explore various model merging approaches to enhance Whisper

and reinforcement learning policies [26]. Thus, we now explore various model merging approaches to enhance Whisper
[21]

Diverse weight averaging for out-of- distribution generalization,

A. Ram ´e, M. Kirchmeyer, T. Rahier, A. Rakotomamonjy, P. Gal- linari, and M. Cord, “Diverse weight averaging for out-of- distribution generalization,” in NeurIPS, 2022

2022
[22]

Investigating self- supervised pretraining frameworks for pathological speech recog- nition,

L. P. Violeta, W. C. Huang, and T. Toda, “Investigating self- supervised pretraining frameworks for pathological speech recog- nition,” in Interspeech 2022, 2022, pp. 41–45

2022
[23]

Training data augmentation for dysarthric automatic speech recognition by text- to-dysarthric-speech synthesis,

W.-Z. Leung, M. Cross, A. Ragni, and S. Goetze, “Training data augmentation for dysarthric automatic speech recognition by text- to-dysarthric-speech synthesis,” in Interspeech 2024 , 2024, pp. 2494–2498

2024
[24]

Fine-tuning strategies for dutch dysarthric speech recog- nition: Evaluating the impact of healthy, disease-specific, and speaker-specific data,

S. Leivaditi, T. Matsushima, M. Coler, S. Nayak, and V . Verkho- danova, “Fine-tuning strategies for dutch dysarthric speech recog- nition: Evaluating the impact of healthy, disease-specific, and speaker-specific data,” in Interspeech 2024. ISCA, 2024, pp. 1295–1299

2024
[25]

Averaging weights leads to wider optima and better generaliza- tion,

P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. Wilson, “Averaging weights leads to wider optima and better generaliza- tion,” in UAI, 2018

2018
[26]

Model soups: averaging weights of multi- ple fine-tuned models improves accuracy without increasing in- ference time,

M. Wortsman, G. Ilharco, S. Y . Gadre, R. Roelofs, R. Gontijo- Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y . Carmon, S. Kornblith et al. , “Model soups: averaging weights of multi- ple fine-tuned models improves accuracy without increasing in- ference time,” in Internation...

2022
[27]

Formally, we denote by θ0,

for transcribing dysarthric speech. Formally, we denote by θ0, . . . , θn−1 the set of weights of different models. All models have the same dimensions and their parameters have correspondence due to the pre-training. We obtain the average model by computing for each sub- para...
[28]

Robust fine-tuning of zero-shot models,

M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, R. G. Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong et al., “Robust fine-tuning of zero-shot models,” inProceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, 2022, pp. 7959–7971

2022
[29]

Learning and trans- ferring mid-level image representations using convolutional neu- ral networks,

M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning and trans- ferring mid-level image representations using convolutional neu- ral networks,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 1717–1724

2014
[30]

Selective atten- tion merging for low resource tasks: A case study of child asr,

N. B. Shankar, Z. Wang, E. Eren, and A. Alwan, “Selective atten- tion merging for low resource tasks: A case study of child asr,” arXiv preprint arXiv:2501.08468, 2025

2025 arXiv
[31]

Bertscore: Evaluating text generation with bert,

T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” 2020. [Online]. Available: https://arxiv.org/abs/1904.09675

2020 arXiv
[32]

Menli: Robust evaluation metrics from natural language inference,

Y . Chen and S. Eger, “Menli: Robust evaluation metrics from natural language inference,” 2023. [Online]. Available: https://arxiv.org/abs/2208.07316

2023 arXiv
[33]

Rewarded soups: towards pareto- optimal alignment by interpolating weights fine-tuned on diverse rewards,

A. Rame, G. Couairon, C. Dancette, J.-B. Gaya, M. Shukor, L. Soulier, and M. Cord, “Rewarded soups: towards pareto- optimal alignment by interpolating weights fine-tuned on diverse rewards,” Advances in Neural Information Processing Systems , 2024

2024
[34]

Robust speech recognition via large-scale weak su- pervision,

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” https://openai.com/research/whisper, 2022, openAI

2022
[35]

Transform- ers: State-of-the-art natural language processing,

T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz et al. , “Transform- ers: State-of-the-art natural language processing,” EMNLP 2020, p. 38, 2020

2020
[36]

Specaugment on large scale datasets,

D. S. Park, Y . Zhang, C.-C. Chiu, Y . Chen, B. Li, W. Chan, Q. V . Le, and Y . Wu, “Specaugment on large scale datasets,” inICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020

2020
[37]

Linear mode connectivity and the lottery ticket hypothesis,

J. Frankle, G. K. Dziugaite, D. Roy, and M. Carbin, “Linear mode connectivity and the lottery ticket hypothesis,” in International Conference on Machine Learning. PMLR, 2020, pp. 3259–3269

2020
[38]

The role of permutation invariance in linear mode connectivity of neural networks,

R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur, “The role of permutation invariance in linear mode connectivity of neural networks,” in ICLR, 2022. [Online]. Available: https://arxiv.org/abs/2110.06296

2022 arXiv
[39]

What matters for model merging at scale?

P. Yadav, T. Vu, J. Lai, A. Chronopoulou, M. Faruqui, M. Bansal, and T. Munkhdalai, “What matters for model merging at scale?” arXiv preprint arXiv:2410.03617, 2024

2024 arXiv
[40]

Arcee’s MergeKit: A toolkit for merging large language mod- els,

C. Goddard, S. Siriwardhana, M. Ehghaghi, L. Meyers, V . Karpukhin, B. Benedict, M. McQuade, and J. Solawetz, “Arcee’s MergeKit: A toolkit for merging large language mod- els,” in EMNLP, 2024

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Robust fine-tuning of speech recognition models via model merging: application to disordered speech

Introduction Automatic Speech Recognition (ASR) has evolved from a spe- cialized tool confined to speech labs and research environments into a mainstream technology that can be deployed with just a few lines of code1. This transformation has been driven by the development of Speech Foundation Models (SFMs), which are trained on large and diverse datasets ...

work page Pith review arXiv 2025

[2] [2]

SAP Challenge The SAP Dataset is divided into four subsets: a training set, a development set, and two unshared test sets (test1 and test2) used for evaluating submissions. The provided training dataset comprises speech recordings from 368 patients, totaling 289 hours of audio data and the de- velopment set consists of 53 patients, amounting to 43 hours o...

work page

[3] [3]

Model Merging Model merging strategies have demonstrated performance im- provements in out-of-distribution predictions for vision models

Methods 3.1. Model Merging Model merging strategies have demonstrated performance im- provements in out-of-distribution predictions for vision models

work page

[4] [4]

As shown in Fig- ure 1, MAST outperformed standard fine-tuning, demonstrat- ing the benefits of weight averaging along a single optimization 2https://github.com/openai/whisper path

Results and discussions We evaluated model merging strategies trained on the full train set, including MAST, MAcT and SMAcT. As shown in Fig- ure 1, MAST outperformed standard fine-tuning, demonstrat- ing the benefits of weight averaging along a single optimization 2https://github.com/openai/whisper path. MAcT further improved performance by integrating d...

work page

[5] [5]

Our findings demonstrate that model merg- ing, particularly selective merging across multiple trajectories, significantly improves WER compared to traditional fine-tuning methods

Conclusions In this paper, we explored the potential of model merging techniques to enhance the performance of ASR systems for dysarthric speech. Our findings demonstrate that model merg- ing, particularly selective merging across multiple trajectories, significantly improves WER compared to traditional fine-tuning methods. This approach not only leverage...

work page

[6] [6]

Benchmarking children’s asr with supervised and self-supervised speech foundation mod- els,

R. Fan, N. B. Shankar, and A. Alwan, “Benchmarking children’s asr with supervised and self-supervised speech foundation mod- els,” arXiv preprint arXiv:2406.10507, 2024

arXiv 2024

[7] [7]

Whisper-at: Noise-robust automatic speech recognizers are also strong general audio event taggers,

Y . Gong, S. Khurana, L. Karlinsky, and J. Glass, “Whisper-at: Noise-robust automatic speech recognizers are also strong general audio event taggers,” in Interspeech, 2023

work page 2023

[8] [8]

Data determines distributional robustness in contrastive language image pre-training (clip),

A. Fang, G. Ilharco, M. Wortsman, Y . Wan, V . Shankar, A. Dave, and L. Schmidt, “Data determines distributional robustness in contrastive language image pre-training (clip),” in International Conference on Machine Learning. PMLR, 2022, pp. 6216–6234

work page 2022