Pith. sign in

REVIEW 4 major objections 8 minor 35 references

Speech Recognition With LLMs Adapted to Disordered Speech Using Reinforcement Learning

T0 review · 4 major / 8 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A small LLM-based speech recognizer adapts to disordered speech better through meaning-aware reinforcement learning than through continued supervised fine-tuning.

desk verdict Worth refereeing, but the headline RL-vs-SFT comparison rests on a single under-specified baseline that regresses WER, so the paper needs a matched-budget retraining check. read the letter →

arxiv 2501.00039 v1 pith:4V4HNTST submitted 2024-12-25 eess.AS cs.CLcs.LGcs.SD

Chirag Nagpal , Subhashini Venugopalan , Jimmy Tobin , Marilyn Ladewig , Katherine Heller , Katrin Tomanek This is my paper · ORCID
classification eess.AScs.CLcs.LGcs.SD
keywords automaticspeechrecognitionlargelanguagemodelsreinforcementlearningdisorderedmeaningpreservationdomainadaptationtokenization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a small, off-the-shelf large language model can be turned into a speech recognizer, and that adapting it to disordered speech works better with reinforcement learning that rewards meaning preservation than with continued supervised fine-tuning. The authors take Gemma 2B, swap its least-used text tokens for audio tokens produced by a frozen speech encoder, and supervise it on a mixture of standard and disordered speech. They then tune it with Proximal Policy Optimization using a reward that combines word error rate with a meaning-preservation score produced by a second, smaller LLM judge. On the Euphonia disordered-speech test set, the best RL-tuned model raises meaning preservation from 48.2 to 55.7 without a statistically significant word-error penalty, while continued supervised fine-tuning makes the starting model worse. A speech-language pathologist's ratings on 220 transcripts agree with the automated metric, which matters because standard word error rates understate comprehension failures in disordered speech.

What carries the argument

The central mechanism is the reward function $R(x,y;y^*) = \gamma \cdot \text{MP}(y,y^*) + \log(1 - \text{WER}(y,y^*))$, where MP is a meaning-preservation score from a Gemma-2B judge trained on 2,840 human-labeled transcript pairs, and WER is the word error rate against the ground-truth transcript. The logarithmic transform puts WER on a scale comparable to MP, and $\gamma$ trades off the two signals. Around that reward, the other load-bearing pieces are the audio-token vocabulary swap (1,024 cluster ids from a frozen USM encoder replace the least frequent text tokens, so the decoder-only LLM consumes audio with no architecture change) and PPO with KL regularization for optimization.

What would settle it

Re-run continued supervised fine-tuning on the same Euphonia training split with a documented grid over learning rate, number of steps, input dropout, and checkpoint selection by validation WER and MP; if an adequately tuned SFT model reaches or beats the RLHF model's MP of 55.7 at a WER of 42.6 or lower on the Euphonia test set, the paper's claim that RL is superior to fine-tuning would be contradicted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the tuning strategy—not the ASR architecture—is what enables an LLM to adapt to disordered speech. The authors first make the LLM audio-capable by mapping 1,024 cluster ids of speech-encoder embeddings onto the last 1,024 text tokens of the Gemma 2B vocabulary and supervised-tuning on a 30:70 Euphonia-to-LibriSpeech mixture. From that starting point, continued supervised fine-tuning on disordered speech alone degrades the model (WER 57.1, MP 42.8 on Euphonia test), while reinforcement learning on the same data with the reward $R(x,y;y^*) = \gamma \cdot \text{MP}(y,y^*) + \log(1 - \text{WER}(y,y^*))$ improves it across all tested values of $\gamma$, with $\gamma = 1.0$ giving the best trade-off (WER 42.6, MP 55.7). The authors report significance against the WER-only RL run ($p<0.0003$ for MP, $p=0.54$ for WER) and corroborate the automated MP metric with a human evaluation by a speech-language pathologist on 220 examples.

Load-bearing premise

The paper's central comparison assumes the continued supervised fine-tuning baseline was tuned at least as carefully as the RLHF runs; the paper does not document the search space or checkpoint selection for that baseline, saying only that 'even with significant hyperparameter tuning' it did not help.

Editorial extensions

If this is right

  • The same vocabulary-swap-plus-RL recipe could in principle adapt LLM-based ASR to other low-resource domains, such as accented speech or child speech, where word-level errors understate comprehension.
  • Because the RL step uses only target-domain audio and transcripts plus an LLM judge, it offers a path for adaptation without retraining the speech encoder or changing the LLM architecture.
  • The result implies that when the goal is meaning preservation rather than exact word match, a learned semantic reward is a usable objective for speech-recognition tuning.
  • The paper's 220-example human evaluation indicates the automated meaning-preservation score tracks a speech-language pathologist's judgment, supporting its use as a reward signal.
  • The authors do not claim state-of-the-art recognition; the claim is specifically that RL tuning outperforms continued supervised fine-tuning for this domain shift.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the authors leave implicit: run the same RL-with-MP-reward recipe on a non-disordered low-resource domain to see whether the advantage over continued SFT is a general property of the tuning method or specific to disordered speech.
  • The comparison is against a continued-SFT baseline whose tuning effort is not documented; a controlled comparison with an equally well-tuned SFT baseline would clarify whether the gain comes from RL itself or from the meaning-preservation objective.
  • The MP judge is a single 2B model trained on 2,840 pairs; using a larger judge or training it on the target domain could change the reward landscape and the reported $\gamma=1.0$ numbers, so the numerical gains are partly tied to this particular judge.
  • By replacing the least frequent text tokens, the method sacrifices rare-token text coverage in exchange for audio capability; multilingual or Unicode-heavy text generation could degrade even though the paper reports little loss on LibriSpeech dev-clean.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The manuscript proposes a three-stage recipe for adapting an LLM to disordered-speech ASR. The authors replace the 1,024 least-frequent text tokens of Gemma 2B with audio codebook tokens obtained by clustering USM embeddings (clusters learned on LibriSpeech), fine-tune the resulting model on mixtures of LibriSpeech and the Euphonia disordered-speech corpus, and then apply PPO with a reward that combines an LLM-based Meaning Preservation (MP) score and log-transformed WER, R = γ·MP + ln(1 − WER). The main empirical claim is that RLHF adaptation from the 30:70 SFT checkpoint substantially improves over continued supervised fine-tuning on Euphonia: Table III reports WER 41.0–42.6 and MP 50.4–55.7 for RLHF at different γ values, versus WER 57.1 and MP 42.8 for continued SFT, and γ = 1.0 improves MP by 7.5 points over the starting SFT model with a statistically insignificant WER change (p = 0.54). A 220-example human evaluation by a speech-language pathologist supports the γ = 1.0 preference over γ = 0.0, and severity-stratified results show the largest gains on moderate and severe speech. The authors are explicit that the resulting model does not outperform existing ASR systems.

Significance. If confirmed, the result is significant for accessibility-oriented ASR: it offers an LLM-native way to adapt speech recognition to a high-variance, low-resource domain, and it makes the WER-versus-meaning-preservation trade-off explicit, which matters for disordered speech where WER alone understates comprehension. Strengths worth naming: the system is built on an open model (Gemma 2B) and public corpora; the paper honestly states that the model does not beat existing ASR systems; the automated MP preference between the two RLHF variants is validated with human speech-language-pathologist ratings (Spearman 0.684/0.639); the severity-stratified analysis (Fig. 3a) locates the gains precisely; and the inclusion of WER in the reward to mitigate reward hacking is a sensible design choice. The principal risks are that MP is both the reward and the primary evaluation metric and comes from the authors' own reward model, and that the central RL-versus-SFT comparison rests on a single under-specified continued-SFT baseline. These issues are fixable within the paper's scope, but they currently make the headline claim stronger than the presented evidence.

major comments (4)
  1. [V-B / Table III] The headline claim that RLHF is substantially better than continued supervised fine-tuning rests entirely on one continued-SFT row in Table III (WER 57.1, MP 42.8) that is worse than the starting 30:70 SFT checkpoint it is adapted from (WER 50.4, MP 48.2). Because the continued-SFT run trains on the same distribution that produced the starting checkpoint, a properly tuned and early-stopped run should at least match it; the regression suggests an early-stopping or recipe failure rather than a property of supervised fine-tuning. Section V-B says only that 'even with significant hyper parameter tuning' continued SFT does not help, and it does not report the search space, number of trials, learning-rate range, data mixture, or checkpoint-selection criterion. Please provide a fully specified, budget-matched continued-SFT baseline (ideally one whose WER matches or improves on the starting model) and the same level of hyperparameter detail that the RLHF runs receive; without this, the central comparison may reflect uneven effort.
  2. [IV (Eq. 1) and V-A/V-B] The MP score is used simultaneously as the RL reward term γ·MP in Eq. (1) and as the primary evaluation metric, and the Gemma-2B MP reward model is trained by the same group on the same task family (2,840 pairs; AUC 0.87 vs. 0.89 for the 62B judge of [16]). Part of the reported MP gain may therefore reflect overoptimization toward this particular judge rather than meaning preservation itself. In addition, the MP advantage of RLHF(γ=1.0) over Continued SFT (55.7 vs. 42.8) is confounded with the 14.5-point WER gap (42.6 vs. 57.1); since WER and MP are correlated, the advantage cannot be cleanly attributed to the reward choice. The human evaluation in Table IV compares only γ=0.0 vs. γ=1.0 and thus does not independently validate the central RL-vs-SFT claim. I ask that the main comparison be re-scored with an external judge (e.g., the 62B model of [16]) or with human labels on a sample that includes the continued-SFT outputs.
  3. [V-B] The significance statements that carry the paper's claims are reported without the information needed to assess them: the text reports p < 1×10−8 for RLHF-vs-SFT, and p < 0.0003 / p = 0.54 for γ=1.0-vs-γ=0.0, but gives no sample sizes, no confidence intervals, and no statement of whether the two-sided t-tests are paired and computed per utterance, per speaker, or per phrase. Without this, the reader cannot judge whether the deltas are consistent across speakers or driven by a few outliers. Please report per-condition means with confidence intervals (e.g., bootstrap over speakers) and state the test unit explicitly.
  4. [IV (Reinforcement Learning)] The PPO configuration is not reproducible as described: the paper names only PPO with a clipped objective and KL regularization and says that checkpoints were selected on validation WER/MP, but it omits the learning rate, the KL coefficient or target, the number of PPO steps, the rollout and minibatch sizes, and how many reward-model evaluations were used. These details are central to a methods claim about RLHF for ASR adaptation; please report them or cite a specific public recipe that was followed.
minor comments (8)
  1. [IV] Typo: 'demonstarted' should be 'demonstrated'.
  2. [Fig. 1 caption] Typo: 'disoredred' should be 'disordered'.
  3. [V-E] The sentence 'the SLP assessed ... on a 3-point 1scale' is garbled, and the footnote defining the scale ('Meaning Completely Preserved, Mild Error, Severe Error') has no marker in the text; please fix the wording and marker.
  4. [Table III caption] The asterisk note says '* is significant w.r.t WER (γ = 0.00) strategy', but the asterisks are placed on MP values; clarify that the significance is relative to the RLHF-WER (γ = 0.00) row.
  5. [III (Tuning details)] Several free parameters (1024 clusters, 16th USM layer, input dropout 5×10−2, the 30:70 mixture ratio) are stated as outcomes of experiments ('pilot experiments ... showed less difference', 'we found 5 × 10−2 to be the best value') without the supporting results; a small ablation table or figure would strengthen the paper.
  6. [V-B] The large WER improvement of RLHF-WER over the starting SFT checkpoint (50.4 → 41.0) and over Continued SFT (57.1 → 41.0) is left unexplained; since the γ=0.0 reward is just ln(1−WER), a sentence on why policy-gradient optimization extracts more WER gain from the same data than continued SFT would help.
  7. [V-E] Please specify how the 220 human-evaluated examples were sampled and from which split (test or dev); Table IV reports only aggregate percentages.
  8. [V-A] The claim that 'Training only on the Librispeech data or only the Euphonia datasets leads to extremely poor generalization' is not fully supported by Table I, which shows the Librispeech-only row but no Euphonia-only row; please add the Euphonia-only numbers or qualify the claim.

Circularity Check

1 steps flagged · score 4.0 of 10

MP reward/evaluation circularity is partial; central RL-vs-SFT claim retains independent WER and human-eval support.

  1. fitted input called prediction [Section IV 'Reinforcement Learning', Eq. (1); Section V-B Table III]
    "The final reward signal R used for alignment is given by Eqn. 1 R(x, y; y∗) := γ · MP(y, y∗) + ln(1 − WER(y, y∗)) (1) ... We use Proximal Policy Optimization (PPO) ... for optimizing the combined reward in Equation 1. ... We report results on val. and test splits. (Table III reports MP↑ scores for each tuning strategy.)"

    The MP scores reported as the main meaning-preservation outcome (e.g., base SFT 48.2 vs RLHF-γ=1.0 55.7 on Euphonia test) come from the same Gemma-2B judge that is the MP term of the RL reward in Eq. (1). The policy is trained with PPO to maximize that exact judge, so the reported MP gain is, by construction, a measure of how well the model optimized its own reward objective rather than an independent semantic measurement. The human SLP evaluation (Table IV) is independent but only contrasts the two RLHF variants (γ=0.0 vs γ=1.0), not RLHF versus continued SFT, leaving the headline cross-method MP comparison anchored to the judge that was also optimized.

full rationale

The token-repurposing and SFT stages (Sections II-III) are self-contained: audio tokens are derived from USM clusters and the model is tuned on LibriSpeech/Euphonia mixtures, with checkpoint selection on WER. The central RLHF-versus-SFT comparison is not entirely circular because WER, an external metric, improves substantially (50.4 to 41.0 on Euphonia test) under RLHF, and the human SLP evaluation independently supports the γ=1.0 vs γ=0.0 meaning-preservation comparison. However, the paper's meaning-preservation claim for RLHF over SFT rests on an MP judge that was trained by the same group on the same style of data and used both as the RL reward (Eq. 1) and as the evaluation metric (Table III); the continued-SFT baseline is also under-specified in Section V-B, though that is a correctness risk rather than a circularity. Overall, the MP-evaluation loop is a partial circularity but the central claim retains independent WER content, so the score is 4 rather than higher.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper's empirical claims rest on modeling choices about audio quantization, vocabulary remapping, and the reward model. These are not mathematical axioms but unproved domain assumptions that bound the result. Free parameters are tuned on validation or pilot runs rather than derived from first principles.

free parameters (5)
  • Reward weight gamma = 1.0 selected as best trade-off; 0.0, 0.25, and 0.50 also evaluated
    Chosen on validation set to balance WER and meaning preservation. The headline result that gamma=1.0 improves MP without significant WER drop depends on this choice.
  • Number of audio clusters = 1024
    Set after pilot experiments varying layers and cluster sizes. This quantization granularity determines how much speech information reaches the LLM.
  • USM embedding layer = 16th layer
    Chosen in pilot experiments; embeddings from this layer are clustered into audio tokens. The paper notes pilot results showed less difference across layers and cluster sizes.
  • Input dropout = 5e-2
    Reported as the best value among experimented input dropouts during supervised fine-tuning.
  • Euphonia:Librispeech SFT mixture ratio = 30:70
    Selected after comparing LibriSpeech-only, 50:50, and 30:70 mixtures on validation. The choice affects the starting SFT model used for RLHF.
assumptions (4)
  • domain assumption 1024 k-means clusters of USM layer-16 embeddings are a sufficient discrete audio representation for ASR.
    Section II-A states pilot experiments showed less difference across layers and cluster sizes. This quantization is the only speech input to the LLM, so all downstream results are bounded by it.
  • domain assumption Replacing the last 1024 low-frequency text tokens with audio tokens does not materially degrade the LLM's language capabilities.
    Section III asserts this should not affect underlying capabilities, but no direct measurement of language degradation is reported.
  • domain assumption The Gemma-2B Meaning Preservation reward model approximates human meaning-preservation judgments in the disordered speech domain.
    Section IV reports an AUC of 0.87 on about 1k examples, close to the 0.89 in prior work. This model is used both as the RL reward and as the main evaluation metric.
  • domain assumption WER computed against ground-truth transcripts is a valid reward signal for adaptation.
    Section IV includes WER in the reward to mitigate reward hacking. This assumes reference transcripts are available during adaptation, which is true in this experimental setup.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Speech Recognition With LLMs Adapted to Disordered Speech Using Reinforcement Learning." pith.science (2026). https://pith.science/paper/4V4HNTST

@misc{pith2026250100039,
  author       = {Pith},
  title        = {Pith review of: Speech Recognition With LLMs Adapted to Disordered Speech Using Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4V4HNTST}},
  note         = {Machine review of arXiv:2501.00039}
}
read the original abstract

We introduce a large language model (LLM) capable of processing speech inputs and show that tuning it further with reinforcement learning on human preference (RLHF) enables it to adapt better to disordered speech than traditional fine-tuning. Our method replaces low-frequency text tokens in an LLM's vocabulary with audio tokens and enables the model to recognize speech by fine-tuning it on speech with transcripts. We then use RL with rewards based on syntactic and semantic accuracy measures generalizing the LLM further to recognize disordered speech. While the resulting LLM does not outperform existing systems for speech recognition, we find that tuning with reinforcement learning using custom rewards leads to substantially better performance than supervised fine-tuning of the language model, specifically when adapting to speech in a different setting. This presents a compelling alternative tuning strategy for speech recognition using large language models.

Figures

Figures reproduced from arXiv: 2501.00039 by the authors.

Figure 1
Figure 1. Using an LLM for ASR: Our approach involves first clustering the audio embeddings to the LLMs vocabulary space. Followed by supervised finetuning on a mixture of disoredred speech. In the final step we use reinforcement learning to improve the model’s output in terms of Word Error Rate and human assessed meaning preservation quality. to replace text tokens in the language model’s vocabulary. This setting while intro… view at source ↗
Figure 2
Figure 2. Cross entropy Loss vs. Learning Steps of the Gemma [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Higher strength of γ leads to improvement in meaning preservation on the Euphonia val and test sets, and for all severity levels. E. Human evaluation We performed human evaluations, where an SLP assessed the error type and severity of the error (meaning preservation) on a 3-point1 scale on 220 examples from the RLHF-WER TABLE IV: Human evaluation of model outputs support the analysis from automated metrics. Statisti… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 11 canonical work pages

  1. [16]

    Large language models as a proxy for human evaluation in assessing the comprehensibility of disordered speech transcription,

    K. Tomanek, J. Tobin, S. Venugopalan, R. Cave, K. Seaver, J. R. Green, and R. Heywood, “Large language models as a proxy for human evaluation in assessing the comprehensibility of disordered speech transcription,” in ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2024, pp. 10 846– 10 850

    work page 2024

  2. [1]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    arXiv 2024

  3. [2]

    Paligemma: A versatile 3b vlm for transfer,

    L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello et al. , “Paligemma: A versatile 3b vlm for transfer,” arXiv preprint arXiv:2407.07726, 2024

    arXiv 2024

  4. [3]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, Y . Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth et al. , “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805 , 2023

    arXiv 2023

  5. [4]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems , vol. 36, 2023

    work page 2023

  6. [5]

    Speechgpt: Empowering large language models with intrinsic cross- modal conversational abilities,

    D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y . Zhou, and X. Qiu, “Speechgpt: Empowering large language models with intrinsic cross- modal conversational abilities,” arXiv preprint arXiv:2305.11000 , 2023

    arXiv 2023

  7. [6]

    Audiopalm: A large language model that can speak and listen,

    P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Bor- sos, F. d. C. Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonovet al., “Audiopalm: A large language model that can speak and listen,” arXiv preprint arXiv:2306.12925, 2023

    arXiv 2023

  8. [7]

    Prompting large language models with speech recognition abilities,

    Y . Fathullah, C. Wu, E. Lakomkin, J. Jia, Y . Shangguan, K. Li, J. Guo, W. Xiong, J. Mahadeokar, O. Kalinli et al. , “Prompting large language models with speech recognition abilities,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 13 351–13 355

    work page 2024

Show all 35 references
  1. [8]

    Large- scale language model rescoring on long-form data,

    T. Chen, C. Allauzen, Y . Huang, D. Park, D. Rybach, W. R. Huang, R. Cabrera, K. Audhkhasi, B. Ramabhadran, P. J. Moreno et al., “Large- scale language model rescoring on long-form data,” in ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Proces...

    2023

  2. [9]

    Adapting gpt, gpt-2 and bert language models for speech recognition,

    X. Zheng, C. Zhang, and P. C. Woodland, “Adapting gpt, gpt-2 and bert language models for speech recognition,” in 2021 IEEE Automatic speech recognition and understanding workshop (ASRU) . IEEE, 2021, pp. 162–168

    2021

  3. [10]

    Rescorebert: Discriminative speech recog- nition rescoring with bert,

    L. Xu, Y . Gu, J. Kolehmainen, H. Khan, A. Gandhe, A. Rastrow, A. Stolcke, and I. Bulyko, “Rescorebert: Discriminative speech recog- nition rescoring with bert,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022,...

    2022

  4. [11]

    On decoder-only architecture for speech-to-text and large language model integration,

    J. Wu, Y . Gaur, Z. Chen, L. Zhou, Y . Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liu et al. , “On decoder-only architecture for speech-to-text and large language model integration,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) . IEEE, 2023, pp. 1–8

    2023

  5. [12]

    An embarrassingly simple approach for llm with strong asr capacity,

    Z. Ma, G. Yang, Y . Yang, Z. Gao, J. Wang, Z. Du, F. Yu, Q. Chen, S. Zheng, S. Zhang et al. , “An embarrassingly simple approach for llm with strong asr capacity,” arXiv preprint arXiv:2402.08846 , 2024

    2024 arXiv

  6. [13]

    Assessing asr model quality on disordered speech using bertscore,

    J. Tobin, Q. Li, S. Venugopalan, K. Seaver, R. Cave, and K. Tomanek, “Assessing asr model quality on disordered speech using bertscore,” arXiv preprint arXiv:2209.10591 , 2022

    2022 arXiv

  7. [14]

    Semantic distance: A new metric for asr perfor- mance analysis towards spoken language understanding,

    S. Kim, A. Arora, D. Le, C.-F. Yeh, C. Fuegen, O. Kalinli, and M. L. Seltzer, “Semantic distance: A new metric for asr perfor- mance analysis towards spoken language understanding,” arXiv preprint arXiv:2104.02138, 2021

    2021 arXiv

  8. [15]

    Clinical bertscore: An improved measure of automatic speech recogni- tion performance in clinical settings,

    J. Shor, R. A. Bi, S. Venugopalan, S. Ibara, R. Goldenberg, and E. Rivlin, “Clinical bertscore: An improved measure of automatic speech recogni- tion performance in clinical settings,” arXiv preprint arXiv:2303.05737 , 2023

    2023 arXiv

  9. [17]

    Bertscore: Evaluating text generation with bert,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” arXiv preprint arXiv:1904.09675 , 2019

    1904 arXiv

  10. [18]

    Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,

    T. Kudo, “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” arXiv preprint arXiv:1808.06226, 2018

    2018 arXiv

  11. [19]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  12. [20]

    Google usm: Scaling automatic speech recognition beyond 100 languages,

    Y . Zhang, W. Han, J. Qin, Y . Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V . Axelrod, G. Wang et al. , “Google usm: Scaling automatic speech recognition beyond 100 languages,” arXiv preprint arXiv:2303.01037 , 2023

    2023 arXiv

  13. [21]

    Gemma: Open models based on gemini research and technology,

    G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivi `ere, M. S. Kale, J. Love et al. , “Gemma: Open models based on gemini research and technology,” arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  14. [22]

    Vaswani, arXiv preprint arXiv:1706.03762 , 2017

    A. Vaswani, arXiv preprint arXiv:1706.03762 , 2017

    2017 arXiv

  15. [23]

    Fast transformer decoding: One write-head is all you need,

    N. Shazeer, “Fast transformer decoding: One write-head is all you need,” arXiv preprint arXiv:1911.02150 , 2019

    1911 arXiv

  16. [24]

    Librispeech: An asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An asr corpus based on public domain audio books,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210

    2015

  17. [25]

    Disordered speech data collection: Lessons learned at 1 million utterances from project euphonia

    R. L. MacDonald, P.-P. Jiang, J. Cattiau, R. Heywood, R. Cave, K. Seaver, M. A. Ladewig, J. Tobin, M. P. Brenner, P. C. Nelson et al. , “Disordered speech data collection: Lessons learned at 1 million utterances from project euphonia.” in Interspeech, vol. 2021, 2021, pp. 4833–4837

    2021

  18. [26]

    Au- tomatic speech recognition of disordered speech: Personalized models outperforming human listeners on short phrases

    J. R. Green, R. L. MacDonald, P.-P. Jiang, J. Cattiau, R. Heywood, R. Cave, K. Seaver, M. A. Ladewig, J. Tobin, M. P. Brenner et al., “Au- tomatic speech recognition of disordered speech: Personalized models outperforming human listeners on short phrases.” in Interspeech, vol....

    2021

  19. [27]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022

    2022

  20. [28]

    Automatic speech recog- nition of conversational speech in individuals with disordered speech,

    J. Tobin, P. Nelson, B. MacDonald, R. Heywood, R. Cave, K. Seaver, A. Desjardins, P.-P. Jiang, and J. R. Green, “Automatic speech recog- nition of conversational speech in individuals with disordered speech,” Journal of Speech, Language, and Hearing Research , pp. 1–10, 2024

    2024

  21. [29]

    Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking,

    J. Eisenstein, C. Nagpal, A. Agarwal, A. Beirami, A. D’Amour, D. Dvi- jotham, A. Fisch, K. Heller, S. Pfohl, D. Ramachandran et al., “Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking,” arXiv preprint arXiv:2312.09244 , 2023

    2023 arXiv

  22. [30]

    Scaling laws for reward model overoptimization,

    L. Gao, J. Schulman, and J. Hilton, “Scaling laws for reward model overoptimization,” in International Conference on Machine Learning . PMLR, 2023, pp. 10 835–10 866

    2023

  23. [31]

    Reward gaming in conditional text generation,

    R. Y . Pang, V . Padmakumar, T. Sellam, A. P. Parikh, and H. He, “Reward gaming in conditional text generation,” arXiv preprint arXiv:2211.08714, 2022

    2022 arXiv

  24. [32]

    Transforming and combining rewards for aligning large language models,

    Z. Wang, C. Nagpal, J. Berant, J. Eisenstein, A. D’Amour, S. Koyejo, and V . Veitch, “Transforming and combining rewards for aligning large language models,” arXiv preprint arXiv:2402.00742 , 2024

    2024 arXiv

  25. [33]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,” arXiv preprint arXiv:1707.06347 , 2017

    2017 arXiv

  26. [34]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand et al. , “Mixtral of experts,” arXiv preprint arXiv:2401.04088 , 2024

    2024 arXiv

  27. [35]

    Aurora- m: The first open source multilingual language model red-teamed according to the us executive order,

    T. Nakamura, M. Mishra, S. Tedeschi, Y . Chai, J. T. Stillerman, F. Friedrich, P. Yadav, T. Laud, V . M. Chien, T. Y . Zhuoet al., “Aurora- m: The first open source multilingual language model red-teamed according to the us executive order,” arXiv preprint arXiv:2404.00399 , 2024

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.