Pith. sign in

REVIEW 3 major objections 5 minor 24 references

Analyzing and Fine-Tuning Whisper Models for Multilingual Pilot Speech Transcription in the Cockpit

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned Whisper cuts cockpit speech errors from 68% to 26%

desk verdict A useful new cockpit dataset and a plausible fine-tuning recipe, but the headline WER is a post-selection optimum because the test set doubles as the validation set. read the letter →

arxiv 2506.21990 v1 pith:YGKW7SDK submitted 2025-06-27 cs.CL cs.AIcs.LGeess.AS

Kartheek Kumar Reddy Nareddy , Sarah Ternus , Julia Niebling This is my paper
classification cs.CLcs.AIcs.LGeess.AS
keywords automaticspeechrecognitionWhispercockpitmultilingualASRLoRAfine-tuningtextnormalizationworderrorrateaviation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports that automatic transcription of multilingual cockpit speech, which state-of-the-art Whisper models handle poorly, can be substantially improved by combining custom text normalization with low-rank fine-tuning on a small domain dataset. On the authors' test set of 40 audio clips, the word error rate of Whisper Large fell from 68.49% for the pretrained model with no normalization to 26.26% after LoRA fine-tuning and their Proposed II normalization. The authors argue that the normalization is more effective after fine-tuning than before, and that this combination is the main driver of the improvement. If this result transfers beyond the recorded scenarios, it would make automated cockpit transcription practical for human-factors research and speech-driven cockpit assistance.

What carries the argument

Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, and the Proposed II normalization pipeline. LoRA trains only a small set of low-rank matrices inserted into the frozen Whisper model, so roughly 1% of parameters are learned, making fine-tuning feasible on a single V100 GPU with a 158-clip training set. The Proposed II pipeline first applies 'Proposed I' -- expanding ICAO spelling-alphabet words to letters (e.g., 'DELTA' to 'D'), removing filler words, and normalizing compound-word spellings such as 'takeoff'/'take off' -- and then applies the Whisper English normalizer. This machinery does two jobs: it reduces surface-form mismatches so WER reflects meaning-relevant errors, and, after fine-tuning, it aligns the model's output with the conventions used to score it, which is why the paper finds the largest gains when normalization is applied to fine-tuned outputs.

What would settle it

Re-run the evaluation on a fresh held-out set of cockpit recordings from different pilots and simulator runs, selecting the learning rate on a separate validation set; if the word error rate on the fresh set is much higher than 26.26%, the claimed improvement from small-data fine-tuning would not generalize.

Watch

Extended reading notes

Core claim

The central claim is that Whisper models can be adapted to multilingual cockpit speech by combining Low-Rank Adaptation (LoRA) fine-tuning with a custom normalization scheme, and that this combination works far better than either alone. Using their own labeled corpus of about 85 minutes of cockpit simulator audio and 130 minutes of pilot interviews in German and English, the authors report that the pretrained Whisper Large model has a 68.49% word error rate without normalization on their 40-clip test set; after LoRA fine-tuning at learning rate 1e-3 and applying the Proposed II normalizer, the WER drops to 26.26%. For Whisper Turbo the same recipe yields a drop from 70.20% to 28.01%. The paper interprets these results as evidence that normalization is especially effective after the model has been fine-tuned to the domain, and that the recipe works across the Whisper model family, while noting that the final WER remains too high for reliable deployment.

Load-bearing premise

The reported error rates assume the 40 test clips are independent of the 158 training clips and representative of cockpit speech, and that choosing the learning rate on the same test set does not inflate the headline numbers.

Editorial extensions

If this is right

  • Combining LoRA fine-tuning with the Proposed II normalizer outperforms either technique alone for every Whisper model size tested, with the largest gains on Large and Turbo models.
  • The custom normalizer (ICAO alphabet expansion, filler removal, compound-word normalization) beats generic Whisper normalizers on raw model output, and its edge grows after fine-tuning.
  • A test-set WER near 26% is still short of reliable deployment, so the authors call for prompting strategies, larger training sets, and context-aware error metrics as next steps.
  • The approach requires only about 1% of model parameters to be trainable, making fine-tuning feasible on a single GPU with a corpus of roughly 200 short clips.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the learning rate was selected by lowest WER on the same 40-clip test set, the reported 26.26% is a post-selection optimum, and an independent validation split would likely show a smaller gain.
  • The ICAO-alphabet and filler-word normalizations are specific to aviation communication, so their benefit likely transfers to other aviation ASR tasks such as air-traffic control more than to general conversational transcription.
  • The multilingual errors shown in Table 16, such as German 'Gut' transcribed as English 'Good', suggest that a language-switch prompt or per-segment language identification could yield further gains beyond what the paper reports.
  • The 40-clip test set shares scenario types and possibly speakers with training data, so a leave-one-scenario-out evaluation would be a stricter test of the method's generality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper collects roughly 85 minutes of cockpit simulator recordings and 130 minutes of pilot interview audio, manually transcribes them, and evaluates the Whisper ASR model family on this domain. The authors propose three custom transcript-normalization schemes (Proposed I, II, III) and combine them with LoRA fine-tuning of Whisper models. Using a test set of 40 audio files, they report that fine-tuning Whisper Large-v3 with LoRA plus the Proposed II normalizer reduces WER from 68.49% (pre-trained, no normalization) to 26.26%, and that Whisper Turbo improves from 70.20% to 28.01% under similar treatment. The paper also compares results across Tiny, Base, Small, Medium, Turbo, and Large models, and across several cockpit scenarios.

Significance. If the reported results are unbiased, the paper would demonstrate a substantial practical improvement for a relatively unexplored ASR domain—multilingual intra-cockpit speech—using parameter-efficient fine-tuning, and would provide a useful comparison of normalization strategies across the Whisper model family. The authors are transparent about the high absolute WER and openly state in Section 4 that the 26% WER is not suitable for reliable deployment. The systematic comparison of normalization orders and learning rates is a useful empirical contribution, and the LoRA fine-tuning details (parameter counts, percentages) are clearly tabulated. However, the central claim is weakened by an evaluation design in which the test set is used to select both the learning rate and the normalization scheme, with no validation split or uncertainty quantification. As a result, the headline numbers should be interpreted as post-selection optima rather than as unbiased estimates of operational performance.

major comments (3)
  1. [Section 3.3 and 3.4, Tables 2-3 and 10-15] The headline WER results (26.26% for Large, 28.01% for Turbo) are minima over 21 configurations (3 learning rates × 7 normalizers) evaluated on the same 40-file test set. The test set is therefore used both for hyperparameter selection and for final reporting, making the reported numbers post-selection optima rather than unbiased estimates of performance on new cockpit speech. No validation set, repeated runs, or confidence intervals are provided, so the magnitude of the selection bias is unknown. I ask the authors to introduce a held-out validation set for selecting learning rates and normalizers, or to use nested cross-validation, and to report the variability of WER across test files (e.g., bootstrap confidence intervals).
  2. [Section 2.4 (Dataset split)] The construction of the 158-file training set and 40-file test set is not described. It is not stated whether the split was random, stratified, or scenario-disjoint, nor whether the same speakers or recording sessions appear in both partitions. If the same speaker or session contributes to both training and testing, the reported WER reductions may be inflated by speaker/session adaptation rather than reflecting generalization to new cockpit speech. Please specify the split procedure and, if feasible, provide a speaker-disjoint or scenario-disjoint evaluation to support the generalization claim.
  3. [Section 5 (Conclusion) and Table 3] The conclusion states that the Turbo model's WER was reduced "from 70.20% (pretrained Whisper Turbo model without normalization baseline) to 28.01% (fine-tuned Whisper Turbo model with the Proposed II normalization scheme)". However, Table 3 shows that 28.01% is achieved with the English normalizer at lr=1e-5, while Proposed II at the same learning rate gives 28.24%. The conclusion therefore misattributes the best Turbo result to Proposed II and overstates the advantage of the proposed normalization scheme; it should be corrected.
minor comments (5)
  1. [Section 2.3] The individual contributions of the three novel components in Proposed I (ICAO-alphabet normalization, filler-word removal, and compound-word normalization) are not ablated; the claim that these components are responsible for the observed improvement is an inference from the combined schemes and is not directly tested.
  2. [Section 2.4] Several training details are omitted: batch size, number of epochs, optimizer settings, warmup steps, and the LoRA rank and alpha hyperparameters. Reporting these would improve reproducibility.
  3. [References] Reference [14] is malformed ("et al. Liao, Y."); it should be formatted as a proper author list and title.
  4. [Tables 2-3 and 10-11] Tables 2 and 3 in the main text are identical to Tables 10 and 11 in the supplementary material; this duplication should be removed or clearly cross-referenced.
  5. [Figure 1] The y-axis label is given as "WER (%)", but the figure lacks a clear description of axis scales and legend placement; please ensure all text is legible in the final version.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline 26.26% WER is a post-selection optimum: the test set is used both to select the learning rate/normalizer and to report the winning configuration's result.

  1. fitted input called prediction [Section 2.4 (Dataset Split), Section 3.3 (Learning rate selection), Section 3.4 (Headline result), Table 2]
    "For fine-tuning, the dataset was divided into a training set consisting of 158 audio files and a test set containing 40 audio files. [...] multiple learning rates from {1e-5, 1e-4, 1e-3} were tested. [...] After fine-tuning (with a learning rate of 1e-3), the model’s WER without normalization was 55.65%, and further decreased to 26.26% when combined with the same normalization method."

    The test set is used to select the best learning rate (and normalization scheme) because Tables 2, 10-15 report WER for every configuration on the same test set, and Section 3.3 says the 'optimal values varied' and chooses per model. Section 3.4 then reports the single best configuration (lr=1e-3, Proposed II) as the headline 26.26%. Consequently, the reported value is the minimum over the 21 tested configurations, not an unbiased estimate of generalization. The hyperparameters are effectively fitted to the test outcome, so the claim that the proposed fine-tuning reduces WER to 26.26% is a post-selection artifact rather than an independent prediction.

full rationale

This paper is an empirical study without a theoretical derivation chain, so there is no definitional or self-citation circularity. However, the central reported improvement (WER dropping from 68.49% to 26.26%) is not an independent evaluation: the authors choose the best learning rate and normalizer based on WER computed on the same 40-file test set that they later use to report the final result. No separate validation set, repeated runs, or confidence intervals are provided, making the headline number a post-selection optimum. This is a form of 'fitted input called prediction' because the hyperparameters are selected using the test quantity, forcing the reported performance to be favorable by construction. The normalization itself is applied symmetrically to references and predictions, so it is not circular, and no load-bearing self-citations appear. The main issue is selection bias rather than full definitional circularity, but it materially undermines the claim as a predictive result.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central empirical claim rests on a small, non-public dataset; the main free parameters are learning rate chosen on the test set and unspecified LoRA rank/alpha. No new theoretical entities are introduced.

free parameters (2)
  • learning rate = 1e-3 for Large, 1e-5 for Turbo, 1e-3 for Small/Medium (best of {1e-5,1e-4,1e-3})
    Chosen per model by lowest WER on the 40-file test set; no validation set, so the reported best numbers are selected on test data.
  • LoRA hyperparameters (rank, alpha, dropout)
    Paper states LoRA with approximately 1% learnable parameters but does not report rank or alpha, so a key configuration is unspecified.
assumptions (3)
  • domain assumption Manual transcripts are accurate ground truth for both German and English segments, including code-switches and ICAO alphabet usage.
    All WER scores depend on these labels (Section 2.1); no inter-annotator agreement or transcript verification is reported.
  • ad hoc to paper WER computed after removing filler words and normalizing compound words measures transcription accuracy.
    Proposed normalization deletes filler words from both hypothesis and reference, which can lower WER without improving verbatim transcription (Section 2.3).
  • standard math Word Error Rate as implemented by jiwer is an accepted standard metric.
    The paper uses jiwer's implementation (Section 2.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analyzing and Fine-Tuning Whisper Models for Multilingual Pilot Speech Transcription in the Cockpit." pith.science (2026). https://pith.science/paper/YGKW7SDK

@misc{pith2026250621990,
  author       = {Pith},
  title        = {Pith review of: Analyzing and Fine-Tuning Whisper Models for Multilingual Pilot Speech Transcription in the Cockpit},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGKW7SDK}},
  note         = {Machine review of arXiv:2506.21990}
}
read the original abstract

The developments in transformer encoder-decoder architectures have led to significant breakthroughs in machine translation, Automatic Speech Recognition (ASR), and instruction-based chat machines, among other applications. The pre-trained models were trained on vast amounts of generic data over a few epochs (fewer than five in most cases), resulting in their strong generalization capabilities. Nevertheless, the performance of these models does suffer when applied to niche domains like transcribing pilot speech in the cockpit, which involves a lot of specific vocabulary and multilingual conversations. This paper investigates and improves the transcription accuracy of cockpit conversations with Whisper models. We have collected around 85 minutes of cockpit simulator recordings and 130 minutes of interview recordings with pilots and manually labeled them. The speakers are middle aged men speaking both German and English. To improve the accuracy of transcriptions, we propose multiple normalization schemes to refine the transcripts and improve Word Error Rate (WER). We then employ fine-tuning to enhance ASR performance, utilizing performance-efficient fine-tuning with Low-Rank Adaptation (LoRA). Hereby, WER decreased from 68.49 \% (pretrained whisper Large model without normalization baseline) to 26.26\% (finetuned whisper Large model with the proposed normalization scheme).

Figures

Figures reproduced from arXiv: 2506.21990 by the authors.

Figure 1
Figure 1. Baseline Word Error Rate (WER) comparison across [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 20 canonical work pages

  1. [1]

    Amodei et al

    D. Amodei et al. Deep speech 2 : End-to-end speech recog- nition in english and mandarin. In Proceedings of The 33rd International Conference on Machine Learning, pages 173–

    work page

  2. [2]

    Ardila, M

    R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. We- ber. Common voice: A massively-multilingual speech cor- pus. arXiv preprint arXiv:1912.06670, 2019. 1

    arXiv 1912

  3. [3]

    Baevski, Y

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli. wav2vec 2.0: A framework for self-supervised learning of speech rep- resentations. In Advances in Neural Information Processing Systems, pages 12449–12460, 2020. 1

    work page 2020

  4. [4]

    Bayes and N

    T. Bayes and N. Price. An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 53:370–418, 1763. 1

    work page

  5. [5]

    Bengio, R

    Y . Bengio, R. Ducharme, P. Vincent, and C. Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155, 2003. 1

    work page 2003

  6. [6]

    Benzeghiba, R

    M. Benzeghiba, R. De Mori, O. Deroo, S. Dupont, T. Erbes, D. Jouvet, L. Fissore, P. Laface, A. Mertins, and C. Ris. Au- tomatic speech recognition and speech variability: A review. Speech communication, 49(10-11):763–786, 2007. 1

    work page 2007

  7. [7]

    W. Chan, D. Park, C. Lee, Y . Zhang, Q. Le, and M. Norouzi. Speechstew: Simply mix all available speech recognition data to train one large neural network. arXiv preprint arXiv:2104.02133, 2021. 1

    arXiv 2021

  8. [8]

    G. Chen, S. Chai, G. Wang, J. Du, W. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, and J. Zhang. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed au- dio. arXiv preprint arXiv:2106.06909, 2021. 1

    arXiv 2021

Show all 24 references
  1. [9]

    Collobert, J

    R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. Natural language processing (almost) from scratch. 2011. 1

    2011

  2. [10]

    S. S. Das, N. Chan, D. Wages, and J. H. Hansen. Applica- tion of automatic speech recognition in call classification. In 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing, pages IV–3896. IEEE, 2002. 1

    2002

  3. [11]

    Galvez, G

    D. Galvez, G. Diamos, J. Ciro, J. F. Cer ´on, K. Achorn, A. Gopi, D. Kanter, M. Lam, M. Mazumder, and V . J. Reddi. The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage. arXiv preprint arXiv:2111.09344, 2021. 1

    2021 arXiv

  4. [12]

    Geirhos, J

    R. Geirhos, J. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665– 673, 2020. 1

    2020

  5. [13]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, Lu Wang, and W. Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. 2

    2022

  6. [14]

    et al. Liao, Y . Prompt-conditioning fine-tuning for domain-specific speech recognition. arXiv preprint arXiv:2307.10274, 2023. 1

    2023 arXiv

  7. [15]

    Matarneh, S

    R. Matarneh, S. Maksymova, V . Lyashenko, and N. Belova. Speech recognition systems: A comparative review. 2017. 1

    2017

  8. [16]

    Panayotov, G

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur. Lib- rispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing, pages 5206–5210. IEEE, 2015. 1

    2015

  9. [17]

    Papenfuss and C

    A. Papenfuss and C. A. Schmidt. Using automatic speech recognition to evaluate team processes in aviation - first ex- periences and open questions. In Engineering Psychology and Cognitive Ergonomics, page 501–513, 2023. 2

    2023

  10. [18]

    Rijal, S

    S. Rijal, S. Adhikari, M. Dahal, M. Awale, and V . Ojha. Whisper finetuning on nepali language. 2023. Preprint. 1

    2023

  11. [19]

    Schmidt, M

    C. Schmidt, M. Stadtschnitzer, and J. Koehler. The Fraun- hofer IAIS audio mining system: Current state and future directions. In Speech Communication; 12. ITG Symposium , pages 1–5, 2016. 2

    2016

  12. [20]

    van Doorn, J

    J. van Doorn, J. Sun, J. Hoekstra, P. Jonk, and V . de Vries. Whisper-ATC: Open models for air traffic control automatic speech recognition with accuracy. In International Confer- ence on Research in Air Transportation, 2024. 1

    2024

  13. [21]

    J. J. Williams, M. Borge, J. Levin, H. Guo, and C. P. Ros´e. A comparative analysis of automatic speech recognition errors in small group classroom discourse.Proceedings of the ACM on Human-Computer Interaction, 7(CSCW2):1–27, 2023. 1

    2023

  14. [22]

    Zhang, D

    Y . Zhang, D. S. Park, W. Han, J. Qin, A. Gulati, J. Shor, A. Jansen, Y . Xu, Y . Huang, and S. Wang. Bigssl: Exploring the frontier of large-scale semi-supervised learning for auto- matic speech recognition. IEEE Journal of Selected Topics in Signal Processing, 16(6):1519–153...

    2022

  15. [23]

    Additional Results 6.1. Dataset Adaptation: Scenario Comparison In the supplementary results, we additionally compare tran- scription performance across four distinct operational sce- narios: Takeoff briefings and checklists (10 scenarios, 20 minutes), ECAM actions (11 scenari...

  16. [24]

    For example, ”Gut” is Table 14

    Challenges with multi-lingual speech Table 16 shows instances where the Whisper model tran- scriptions struggles with unexpected translation, often mis- interpreting words or phrases based on phonetic similari- ties rather than contextual meaning. For example, ”Gut” is Table 1...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.