Towards Robust Speech Recognition for Jamaican Patois Music Transcription

Daniel Geddez; Dimitri Johnson; Jordan Madden; Matthew Stone

REVIEW 4 major objections 6 minor 16 references

Towards Robust Speech Recognition for Jamaican Patois Music Transcription

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that fine-tuning Whisper models on tens of hours of manually transcribed Jamaican Patois music sharply reduces word error rate, and that the remaining error follows a power law in model size and data size that can predict…

desk verdict Good dataset and plausible fine-tuning gains, but the scaling-law 'validation' is in-sample and the WER metric is nonstandard, so the headline quantitative claim is not supported. read the letter →

arxiv 2507.16834 v1 pith:S2NVAAE4 submitted 2025-07-15 eess.AS cs.AIcs.CL

Jordan Madden , Matthew Stone , Dimitri Johnson , Daniel Geddez This is my paper · ORCID

classification eess.AScs.AIcs.CL

keywords AutomaticSpeechRecognitionJamaicanPatoisMusicTranscriptionLow-ResourceLanguagesWhisperfine-tuningScalinglawsWordErrorRate

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Jamaican Patois is widely spoken but nearly absent from speech datasets, so off-the-shelf speech recognition produces poor captions for Patois music; the paper reports a 0.89 word error rate for the untuned Whisper Large model. The paper's central claim is that a relatively small amount of curated data changes this: fine-tuning Whisper variants on 42.58 hours of manually transcribed Patois music lowers word error rate to 0.70 for the smallest model and 0.30 for the medium model, all beating the much larger zero-shot Whisper Large. The paper further claims that the measured word error rates obey a power law, $WER = 158.06 \cdot M^{-0.255} \cdot D^{-0.269}$, so performance at other model sizes and data amounts can be predicted without running new experiments. If this is right, it provides a concrete data-centric route to accessible captions for Patois music and to building language technology for a low-resource creole language.

What carries the argument

The load-bearing object is the power-law scaling relation $WER = A \cdot M^{-\alpha} \cdot D^{-\beta}$, where $M$ is the number of model parameters and $D$ is the number of training hours. The paper fits this relation to twelve experimental points, four Whisper variants at 20, 35, and 40 training hours, by linear regression on the log-transformed equation, yielding $\log WER = 5.063 - 0.255 \log M - 0.269 \log D$. Its job is to turn a small grid of fine-tuning runs into a predictive tool for untested model sizes and dataset sizes. The enabling resource is the dataset itself: 5,110 thirty-second clips totaling 42.58 hours of manually transcribed Patois music, which the paper says is the largest public dataset of its kind.

What would settle it

Fine-tune Whisper Large on the same Patois music dataset at 20, 35, and 40 hours, and compare its measured word error rates to the values the fitted power law predicts for that model size and those data amounts; a systematic gap, or a large deviation at the 40-hour point, would show the law does not extrapolate across the Whisper family.

Watch

Extended reading notes

Core claim

The discovery is that domain-specific fine-tuning on a modest transcribed corpus outweighs raw model scale for Patois music transcription. Fine-tuning Whisper Tiny on 40 hours of Patois music gives a word error rate of 0.70, while the untuned Whisper Large sits at 0.89; the medium variant reaches 0.30 on the same data. The paper also reports that the error falls as a power law of model size and dataset size, with fitted exponents of -0.255 and -0.269, and uses that law to predict word error rates for configurations it did not train, including a fine-tuned Whisper Large. The agreement between the predicted and measured values in the tested grid is presented as validation that the scaling relation captures the behavior of the Whisper family on this task.

Load-bearing premise

The paper's predictive claims rest on the assumption that word error rate falls as a clean power law of model size and training hours, and that the twelve measured points are reliable and independent enough to pin down the three fitted parameters; if that shape is wrong, the predicted word error rates for untested configurations do not follow.

Editorial extensions

If this is right

Fine-tuned Whisper Tiny, roughly 50 times smaller than Whisper Large, transcribes Patois music more accurately than the untuned Large model, showing that targeted fine-tuning can beat raw model scale for low-resource musical dialects.
The fitted scaling law lets a practitioner estimate the expected word error rate for any Whisper variant at any training-data budget before spending compute, guiding model selection and data collection.
Extrapolating the law to larger fine-tuned models predicts further word error reductions, which the paper takes as evidence that additional model scale will continue to help Patois transcription.
The improved transcriptions stand to benefit downstream systems that consume ASR output for Jamaican music, such as the LLM-based lyric filtering pipeline discussed in the paper.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the 20-, 35-, and 40-hour subsets appear to be nested, refitting the scaling law on independent splits of the dataset would test whether the exponents are stable or an artifact of overlapping training data.
A direct check of the law's extrapolative power would be to fine-tune Whisper Large on the same data and compare its measured word error rate to the value the fitted curve predicts for that parameter count and dataset size.
If the scaling behavior carries over to spoken Patois, the same data-centric recipe could seed broader language technology for the language, including conversational ASR and text corpora for language models.
The fitted exponents suggest that, over the tested range, increasing model size and increasing training hours contribute roughly comparable error reductions, so the choice between compute and data collection is a practical trade-off rather than a foregone conclusion.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The useful part of this paper is the dataset and the demonstration that fine-tuning Whisper on Patois music improves accuracy. The part that is not yet supported is the scaling law.

What's new: a manually transcribed corpus of 42+ hours of Patois music (if it is actually released; the text says 'publicly available' but I don't see a link), and the first attempt I know of to fit Whisper scaling exponents for dialectal music. The fine-tuning results are plausible and, taken at face value, are a real result: even Tiny at 40h (0.70) beats zero-shot Large (0.89) on their test set. That is worth knowing.

Soft spots: First, Eq. (1) is not standard edit-distance WER; it is a per-word positional match rate with no alignment handling, and there are no error bars or test-set details. Second, the scaling law's 'validation' is entirely in-sample: they fit A, alpha, beta to the same 12 points plotted in Fig. 2, so the agreement is forced, not predictive. Dataset size only spans 20-40 hours (a factor of two), and the subsets are likely nested, so the residuals are not independent. The claim that Eq. (5) predicts untested sizes/data amounts is therefore unsupported. Third, the conclusion mentions a 'scalable data flywheel' that never appears in the body; that needs to be either described or cut. There are also minor figure/table numbering errors (the text says Figure 2 when it means Figure 1).

The qualitative conclusion—fine-tuning helps, more data helps, larger models help—holds up. The quantitative scaling law does not.

This paper is for the low-resource ASR / music captioning community. As a workshop submission it has value; as a journal paper the scaling-law claim is a load-bearing problem. I would not publish it as is. If the authors release the dataset, recompute standard WER, and validate the fit on held-out model/data combinations, it could become a useful reference. For a serious venue, I'd send it to referees because the dataset and fine-tuning comparison are citable, but with a clear request for major revision.

Given the central scaling result is unsupported, my verdict is reject/revision.

Referee Report

4 major / 6 minor

Summary. The paper introduces a new corpus of 5,110 30-second clips (about 42.6 hours) of manually transcribed Jamaican Patois music, fine-tunes four Whisper model sizes (tiny, base, small, medium) on 20, 35, and 40 hours of this data, and compares them to zero-shot Whisper Large. It then fits a power-law scaling model, WER = 158.06 * M^-0.255 * D^-0.269 (Eq. 5), to the 12 empirical fine-tuning results and claims that the close agreement between the fitted curve and the data validates the scaling law, which it further uses to predict performance for untested model sizes and data amounts. The paper's stated contributions are the dataset, the fine-tuning experiments, and the scaling law.

Significance. If the quantitative results held, the dataset would be a valuable community resource, and the qualitative finding that fine-tuning a small Whisper model on only tens of hours of Patois music can outperform zero-shot Whisper Large (claimed WER 0.70 versus 0.89) would be practically significant for low-resource ASR. The data-centric framing and the goal of making Patois music accessible are commendable. However, the paper's quantitative claims are currently undermined by a nonstandard evaluation metric, missing test-set details, and an in-sample validation of the scaling law. The strengths are the dataset itself and the plausible qualitative trend that larger models and more data improve performance; the fitted scaling law as presented is not a validated predictive tool.

major comments (4)

[Section 4.1, Eq. (1)] The metric defined in Eq. (1) is not the standard Word Error Rate. Standard WER is an edit-distance measure (Levenshtein distance at the word level) that accounts for substitutions, deletions, and insertions. The proposed definition compares words at the same index and requires equal-length transcripts, so it cannot handle the misalignments that dominate real ASR errors. All WER values in Table 1 and Figures 1-2 are therefore not comparable to standard WER numbers, and the central quantitative claims (e.g., Whisper Tiny 0.70 versus Whisper Large 0.89) are not established. Please re-run the evaluation with a standard WER implementation or explicitly rename the metric and justify why it is appropriate for this task.
[Section 4.2, Table 1] The paper does not describe any train/validation/test split. It is unclear whether the 'Best WER' results in Table 1 are computed on a held-out test set, on the validation set, or even on the training set. Figure 1 shows training curves labeled as Word Error Rate, but no details are given about the evaluation data. Without a clear statement of the test-set size, composition, and the number of evaluation clips, the generalization claims are unsupported. Please report the evaluation protocol and include error bars or confidence intervals over test clips.
[Section 4.3, Eq. (5) and Figure 2] The 'validation' of the scaling law is in-sample: the same 12 experimental points used to fit A, alpha, and beta are plotted against the fitted curve in Figure 2. With three free parameters and only 12 points, a high in-sample agreement is expected and does not demonstrate predictive power. The three data amounts (20, 35, and 40 hours) span only a factor of two, and the 35- and 40-hour training sets are likely nested supersets of the 20-hour set, which violates the independence assumption underlying the regression. The claim that Eq. (5) can predict WER for untested model sizes and data amounts is therefore unsupported. Please provide a genuine out-of-sample test (e.g., hold out one model size or one data amount), use cross-validation, or report uncertainty intervals for the fitted parameters.
[Section 4.3, final paragraph] The statement that the scaling law 'lends credibility to our estimate for the performance of the Whisper Large model, despite it not being fine-tuned in our experiments' is not justified. The Whisper Large row in Table 1 is a zero-shot result, which is not described by Eq. (5), and no fine-tuned Large data point exists within the fitted range. Extrapolating the scaling law to a model size outside the fitted range is an unsupported assumption, not a validated prediction. This claim should be removed or explicitly reframed as a conjecture.

minor comments (6)

[Section 3.1, problem formulation] The notation 'F : Rn → In' is unclear; the codomain likely should be a sequence space for tokens, not a set of n-dimensional vectors. Please clarify or correct the notation.
[Section 4.2] The text says 'Figure 2 shows how the Word Error Rate changes as training progresses,' but the figure with training curves is labeled Fig. 1, and the predicted-WER plot is labeled Fig. 2. Please renumber the figures and fix the cross-references.
[Introduction and throughout] There are several typographical errors, including 'Patos' instead of 'Patois' and 'hopetocontributetothebroaderecosystem' missing spaces. Please copyedit the manuscript.
[Table 1] The Whisper Large* row does not indicate the number of hours used or that it is zero-shot; please mark clearly that it was not fine-tuned and was evaluated on the same test set as the fine-tuned models.
[Section 4.1] The phrase 'we use of the Word Error Rate metric' should be 'we use the Word Error Rate metric,' and in Section 4.3 'we went on to calculated' should be 'we went on to calculate.'
[Section 3.2] The dataset is described as 'the largest publicly available dataset of its kind,' but no download link or repository is provided in the manuscript. Please include an availability statement or URL.

Circularity Check

1 steps flagged · score 6.0 of 10

Scaling-law 'validation' uses the same 12 points used to fit Eq. 5; the predicted-WER claim reduces to in-sample regression.

fitted input called prediction [Section 4.3, Eqs. (2)-(5), Figure 2, and Table 1]
"For each combination of dataset size and Whisper model variant used in our experiments, we went on to calculated the predicted WER using the derived scaling law. These predictions are visualized in Figure 2. As shown, the predicted WER values closely align with the empirical results reported in Table 1, providing strong validation for the accuracy of our scaling model."

The parameters A, alpha, and beta in Eq. (5) are obtained by linear regression on the empirical WER values from Table 1, i.e., the same 12 combinations of model size and dataset size. The 'predicted' WER for those combinations is therefore the fitted curve evaluated at those same points; agreement with Table 1 is guaranteed by the least-squares fit rather than by independent evidence. Calling this alignment 'strong validation' presents the fit's own residuals as a confirmation of its predictive power. With only 12 points, three fitted parameters, and nested dataset sizes (20, 35, 40 hours), in-sample closeness is expected even if the power-law form is arbitrary. No held-out data, cross-validation, or uncertainty estimate is reported, so the claim that Eq.

full rationale

The paper's fine-tuning results themselves are not circular: Whisper models are trained on the curated Patois corpus and evaluated against ground-truth transcriptions, giving plausible WER reductions that do not depend on the scaling law. The circularity is confined to the scaling-law contribution, the claimed predictive centerpiece. Equation (5) is fitted by regression on the 12 experimental WER values, and the paper then 'validates' it by showing that the fitted curve passes near those same experimental points. That is self-consistency, not validation, and the extrapolation to Whisper Large is an unvalidated projection from the same fitted curve. The paper's self-citation to Stone et al. [14] is motivational rather than load-bearing and does not affect circularity. Overall, the qualitative result is independent, but the headline predictive claim is statistically forced by the regression, warranting a score of 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The scaling-law contribution rests on the power-law ansatz and on the quality of the manual transcriptions; the WER metric is a nonstandard definition. No new physical entities are introduced.

free parameters (3)

A (scaling constant) = 158.06
Fitted by linear regression in Eq. (4) to the 12 experimental WER values.
alpha (model-size exponent) = 0.255
Fitted in Eq. (3) to experimental data.
beta (data-size exponent) = 0.269
Fitted in Eq. (3) to experimental data.

assumptions (4)

domain assumption WER follows a power-law function of model size M and dataset size D: WER = A M^-alpha D^-beta.
This functional form is assumed in Section 4.3 without justification; it is a modeling choice.
domain assumption The manually curated transcriptions are accurate and consistent.
The dataset section describes manual annotation but provides no annotator agreement or quality checks.
domain assumption The WER metric defined in Eq. (1) is a valid measure of transcription error.
The paper uses a simplified word-level indicator function that omits insertions and deletions; standard WER uses edit distance.
domain assumption Fine-tuning on 20/35/40 hours of Patois music transfers to the general Patois music domain.
No held-out test set description or domain shift analysis; the evaluation split is not clearly defined.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Robust Speech Recognition for Jamaican Patois Music Transcription." pith.science (2026). https://pith.science/paper/S2NVAAE4

@misc{pith2026250716834,
  author       = {Pith},
  title        = {Pith review of: Towards Robust Speech Recognition for Jamaican Patois Music Transcription},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S2NVAAE4}},
  note         = {Machine review of arXiv:2507.16834}
}

read the original abstract

Although Jamaican Patois is a widely spoken language, current speech recognition systems perform poorly on Patois music, producing inaccurate captions that limit accessibility and hinder downstream applications. In this work, we take a data-centric approach to this problem by curating more than 40 hours of manually transcribed Patois music. We use this dataset to fine-tune state-of-the-art automatic speech recognition (ASR) models, and use the results to develop scaling laws for the performance of Whisper models on Jamaican Patois audio. We hope that this work will have a positive impact on the accessibility of Jamaican Patois music and the future of Jamaican Patois language modeling.

Figures

Figures reproduced from arXiv: 2507.16834 by the authors.

**Figure 1.** Word Error Rate vs Training Steps for various sizes of the Whisper Models. where N is the number of words in a sample transcription, i represents the index of a word in the transcription, x is the generated transcript, y is the ground truth transcript, and I(·) is the indicator function I(x, y) = 1 if x ̸= y and 0 if x = y. Additionally, to measure the loss during training and validation, we made use of the Cross-En… view at source ↗

**Figure 2.** Predicted Word Error Rate vs Model Size on a Logarithmic Scale most performant model in the Whisper family) has a WER of approximately 0.05, while on Jamaican Patois, it has a WER of 0.89 as seen in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 10 canonical work pages

[1]

In: 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

Anoop, C.S., Prathosh, A., Ramakrishnan, A.: Unsupervised domain adap- tation schemes for building asr in low-resource languages. In: 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). pp. 342–349. IEEE (2021)

work page 2021
[2]

Advances in neural information processing systems33, 12449–12460 (2020)

Baevski, A., Zhou, Y., Mohamed, A., Auli, M.: wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems33, 12449–12460 (2020)

2020
[3]

Computer Speech & Language71, 101262 (2022)

Biswas, A., Yılmaz, E., van der Westhuizen, E., de Wet, F., Niesler, T.: Code-switched automatic speech recognition in five south african languages. Computer Speech & Language71, 101262 (2022)

work page 2022
[4]

arXiv preprint arXiv:2106.09488 (2021)

Droppo, J., Elibol, O.: Scaling laws for acoustic models. arXiv preprint arXiv:2106.09488 (2021)

arXiv 2021
[5]

IEEE/ACM Transactions on Audio, Speech, and Language Processing30, 2280–2294 (2022)

Gao, X., Gupta, C., Li, H.: Automatic lyrics transcription of polyphonic music with lyrics-chord multi-task learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing30, 2280–2294 (2022)

work page 2022
[6]

IEEE/ACM transactions on audio, speech, and language processing29, 3451–3460 (2021)

Hsu, W.N., Bolte, B., Tsai, Y.H.H., Lakhotia, K., Salakhutdinov, R., Mohamed, A.: Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing29, 3451–3460 (2021)

work page 2021
[7]

arXiv preprint arXiv:2001.08361 (2020)

Kaplan, J., et al.: Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

arXiv 2020
[8]

In: Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics: Student Research Workshop

Karunanayake, Y., Thayasivam, U., Ranathunga, S.: Transfer learning based free-form speech command classification for low-resource languages. In: Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics: Student Research Workshop. pp. 288–294 (2019)

work page 2019

Show all 16 references

[9]

In: Interspeech

Ko, T., Peddinti, V., Povey, D., Khudanpur, S.: Audio augmentation for speech recognition. In: Interspeech. vol. 2015, p. 3586 (2015)

2015
[10]

In: SoutheastCon 2025

LeFevre, G., Hosier, J., Zhou, Y., Gurbani, V.K.: Llm selection: Improving asr transcript quality via zero-shot prompting. In: SoutheastCon 2025. pp. 1440–1445. IEEE (2025)

2025
[11]

arXiv preprint arXiv:1912.01703 (2019)

Paszke, A.: Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703 (2019)

2019 arXiv
[12]

Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I.: Robustspeechrecognitionvialarge-scaleweaksupervision.In:International conference on machine learning. pp. 28492–28518. PMLR (2023)

2023
[13]

The Bell system technical journal27(3), 379–423 (1948)

Shannon, C.E.: A mathematical theory of communication. The Bell system technical journal27(3), 379–423 (1948)

1948
[14]

Stone, M., Mansingh, G.: Ai tool for cleaning up lewd music: A jamaican perspective (research-in-progress) (2024)

2024
[15]

In: Proceedings of the 2020 conference on 10 J

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.: Transformers: State-of-the- art natural language processing. In: Proceedings of the 2020 conference on 10 J. Madden et al. empirical methods in natural l...

2020
[16]

arXiv preprint arXiv:2105.01051 (2021)

Yang, S.w., Chi, P.H., Chuang, Y.S., Lai, C.I.J., Lakhotia, K., Lin, Y.Y., Liu, A.T., Shi, J., Chang, X., Lin, G.T., et al.: Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2105.01051 (2021)

2021 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

In: 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

Anoop, C.S., Prathosh, A., Ramakrishnan, A.: Unsupervised domain adap- tation schemes for building asr in low-resource languages. In: 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). pp. 342–349. IEEE (2021)

work page 2021

[2] [2]

Advances in neural information processing systems33, 12449–12460 (2020)

Baevski, A., Zhou, Y., Mohamed, A., Auli, M.: wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems33, 12449–12460 (2020)

2020

[3] [3]

Computer Speech & Language71, 101262 (2022)

Biswas, A., Yılmaz, E., van der Westhuizen, E., de Wet, F., Niesler, T.: Code-switched automatic speech recognition in five south african languages. Computer Speech & Language71, 101262 (2022)

work page 2022

[4] [4]

arXiv preprint arXiv:2106.09488 (2021)

Droppo, J., Elibol, O.: Scaling laws for acoustic models. arXiv preprint arXiv:2106.09488 (2021)

arXiv 2021

[5] [5]

IEEE/ACM Transactions on Audio, Speech, and Language Processing30, 2280–2294 (2022)

Gao, X., Gupta, C., Li, H.: Automatic lyrics transcription of polyphonic music with lyrics-chord multi-task learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing30, 2280–2294 (2022)

work page 2022

[6] [6]

IEEE/ACM transactions on audio, speech, and language processing29, 3451–3460 (2021)

Hsu, W.N., Bolte, B., Tsai, Y.H.H., Lakhotia, K., Salakhutdinov, R., Mohamed, A.: Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing29, 3451–3460 (2021)

work page 2021

[7] [7]

arXiv preprint arXiv:2001.08361 (2020)

Kaplan, J., et al.: Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

arXiv 2020

[8] [8]

In: Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics: Student Research Workshop

Karunanayake, Y., Thayasivam, U., Ranathunga, S.: Transfer learning based free-form speech command classification for low-resource languages. In: Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics: Student Research Workshop. pp. 288–294 (2019)

work page 2019