{"id":"16d44f0a-f9ed-407f-99c7-82d473c25655","arxiv_id":"2507.16834","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning Whisper on 42 hours of Jamaican Patois music improves transcription error rates, and a fitted power law predicts error from model and data size.","lead":"This paper introduces a manually transcribed dataset of Jamaican Patois music and fine-tunes Whisper speech recognition models on it. It claims that even small fine-tuned models outperform the much larger pre-trained Whisper, and fits a scaling law to predict WER from model and data size.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Scaling-law 'validation' is in-sample only; no held-out test or independence check, so predictive claim for untested model sizes is unsupported.","rationale":"The reader's weakest assumption identified the reliability and independence of the 12 points used to fit the scaling law and the in-sample nature of its validation. My concern is the same load-bearing issue, sharpened to the specific circular step: Figure 2 validates Eq. (5) against the very data used to fit it, so the predictive claim for untested configurations is unsupported. The qualitative fine-tuning improvement is credible, but the scaling law is the paper's main novel quantitative contribution; if it cannot be shown to predict held-out conditions, the paper's title claim of 'robust' transcription is reduced to an empirical table. As a result, the reader's REJECT verdict remains appropriate, so no verdict change is needed. The concrete leave-one-model-out test would settle whether the scaling law has any out-of-sample predictive power.","tokens_in":87,"tokens_out":6979,"duration_ms":95535,"concrete_test":"Perform leave-one-model-out cross-validation: exclude all three data points for Whisper Medium from the regression, refit Eq. (4) on the remaining 9 points, and predict Medium's WER at 20, 35, and 40 hours. Compare these predictions to the observed 0.40, 0.34, and 0.30 in Table 1; if the mean absolute error exceeds 0.05, the scaling law does not generalize to the held-out model size. Also report the overlap among the 20/35/40-hour subsets; if nested, refit using only disjoint incremental data (e.g., first 20h, next 15h, next 5h) and check whether the beta exponent changes materially.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's quantum centerpiece is the scaling law in Eq. (5), but its only 'validation' is Figure 2, which plots the fitted curve against the same 12 experimental points used to estimate A, alpha, and beta. With 3 parameters and only 4 model sizes x 3 dataset sizes (D spanning a mere factor of 2, from 20 to 40 hours), a high in-sample R^2 is expected even if the power-law form is arbitrary. The observations are also assumed independent, yet the 20/35/40-hour training sets are likely nested; if so, residuals are correlated and the apparent tightness of the fit is inflated. No hold-out data, cross-validation, or uncertainty estimate is reported, so the claim that Eq. (5) predicts WER for untested model sizes and data amounts is not supported. The qualitative result that fine-tuning improves performance is plausible, but the scaling law, a headline contribution, rests on circular evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new corpus of 5,110 30-second clips (about 42.6 hours) of manually transcribed Jamaican Patois music, fine-tunes four Whisper model sizes (tiny, base, small, medium) on 20, 35, and 40 hours of this data, and compares them to zero-shot Whisper Large. It then fits a power-law scaling model, WER = 158.06 * M^-0.255 * D^-0.269 (Eq. 5), to the 12 empirical fine-tuning results and claims that the close agreement between the fitted curve and the data validates the scaling law, which it further uses to predict performance for untested model sizes and data amounts. The paper's stated contributions are the dataset, the fine-tuning experiments, and the scaling law.","tokens_in":5485,"tokens_out":8049,"duration_ms":93532,"significance":"If the quantitative results held, the dataset would be a valuable community resource, and the qualitative finding that fine-tuning a small Whisper model on only tens of hours of Patois music can outperform zero-shot Whisper Large (claimed WER 0.70 versus 0.89) would be practically significant for low-resource ASR. The data-centric framing and the goal of making Patois music accessible are commendable. However, the paper's quantitative claims are currently undermined by a nonstandard evaluation metric, missing test-set details, and an in-sample validation of the scaling law. The strengths are the dataset itself and the plausible qualitative trend that larger models and more data improve performance; the fitted scaling law as presented is not a validated predictive tool.","major_comments":[{"comment":"The metric defined in Eq. (1) is not the standard Word Error Rate. Standard WER is an edit-distance measure (Levenshtein distance at the word level) that accounts for substitutions, deletions, and insertions. The proposed definition compares words at the same index and requires equal-length transcripts, so it cannot handle the misalignments that dominate real ASR errors. All WER values in Table 1 and Figures 1-2 are therefore not comparable to standard WER numbers, and the central quantitative claims (e.g., Whisper Tiny 0.70 versus Whisper Large 0.89) are not established. Please re-run the evaluation with a standard WER implementation or explicitly rename the metric and justify why it is appropriate for this task.","section":"Section 4.1, Eq. (1)"},{"comment":"The paper does not describe any train/validation/test split. It is unclear whether the 'Best WER' results in Table 1 are computed on a held-out test set, on the validation set, or even on the training set. Figure 1 shows training curves labeled as Word Error Rate, but no details are given about the evaluation data. Without a clear statement of the test-set size, composition, and the number of evaluation clips, the generalization claims are unsupported. Please report the evaluation protocol and include error bars or confidence intervals over test clips.","section":"Section 4.2, Table 1"},{"comment":"The 'validation' of the scaling law is in-sample: the same 12 experimental points used to fit A, alpha, and beta are plotted against the fitted curve in Figure 2. With three free parameters and only 12 points, a high in-sample agreement is expected and does not demonstrate predictive power. The three data amounts (20, 35, and 40 hours) span only a factor of two, and the 35- and 40-hour training sets are likely nested supersets of the 20-hour set, which violates the independence assumption underlying the regression. The claim that Eq. (5) can predict WER for untested model sizes and data amounts is therefore unsupported. Please provide a genuine out-of-sample test (e.g., hold out one model size or one data amount), use cross-validation, or report uncertainty intervals for the fitted parameters.","section":"Section 4.3, Eq. (5) and Figure 2"},{"comment":"The statement that the scaling law 'lends credibility to our estimate for the performance of the Whisper Large model, despite it not being fine-tuned in our experiments' is not justified. The Whisper Large row in Table 1 is a zero-shot result, which is not described by Eq. (5), and no fine-tuned Large data point exists within the fitted range. Extrapolating the scaling law to a model size outside the fitted range is an unsupported assumption, not a validated prediction. This claim should be removed or explicitly reframed as a conjecture.","section":"Section 4.3, final paragraph"}],"minor_comments":[{"comment":"The notation 'F : Rn → In' is unclear; the codomain likely should be a sequence space for tokens, not a set of n-dimensional vectors. Please clarify or correct the notation.","section":"Section 3.1, problem formulation"},{"comment":"The text says 'Figure 2 shows how the Word Error Rate changes as training progresses,' but the figure with training curves is labeled Fig. 1, and the predicted-WER plot is labeled Fig. 2. Please renumber the figures and fix the cross-references.","section":"Section 4.2"},{"comment":"There are several typographical errors, including 'Patos' instead of 'Patois' and 'hopetocontributetothebroaderecosystem' missing spaces. Please copyedit the manuscript.","section":"Introduction and throughout"},{"comment":"The Whisper Large* row does not indicate the number of hours used or that it is zero-shot; please mark clearly that it was not fine-tuned and was evaluated on the same test set as the fine-tuned models.","section":"Table 1"},{"comment":"The phrase 'we use of the Word Error Rate metric' should be 'we use the Word Error Rate metric,' and in Section 4.3 'we went on to calculated' should be 'we went on to calculate.'","section":"Section 4.1"},{"comment":"The dataset is described as 'the largest publicly available dataset of its kind,' but no download link or repository is provided in the manuscript. Please include an availability statement or URL.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The stress-test concern about in-sample scaling-law validation is well-founded. I also find the WER metric in Eq. (1) to be a more fundamental problem because it invalidates all numerical comparisons unless the evaluation is redone with standard edit-distance WER. The dataset and the qualitative fine-tuning trend are plausible and could form the basis of a usable paper after a major revision. If the authors are unable to re-run the evaluation with a standard metric and provide a genuine out-of-sample validation of the scaling law, I would recommend rejection over accept."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful part of this paper is the dataset and the demonstration that fine-tuning Whisper on Patois music improves accuracy. The part that is not yet supported is the scaling law.\n\nWhat's new: a manually transcribed corpus of 42+ hours of Patois music (if it is actually released; the text says 'publicly available' but I don't see a link), and the first attempt I know of to fit Whisper scaling exponents for dialectal music. The fine-tuning results are plausible and, taken at face value, are a real result: even Tiny at 40h (0.70) beats zero-shot Large (0.89) on their test set. That is worth knowing.\n\nSoft spots: First, Eq. (1) is not standard edit-distance WER; it is a per-word positional match rate with no alignment handling, and there are no error bars or test-set details. Second, the scaling law's 'validation' is entirely in-sample: they fit A, alpha, beta to the same 12 points plotted in Fig. 2, so the agreement is forced, not predictive. Dataset size only spans 20-40 hours (a factor of two), and the subsets are likely nested, so the residuals are not independent. The claim that Eq. (5) predicts untested sizes/data amounts is therefore unsupported. Third, the conclusion mentions a 'scalable data flywheel' that never appears in the body; that needs to be either described or cut. There are also minor figure/table numbering errors (the text says Figure 2 when it means Figure 1).\n\nThe qualitative conclusion—fine-tuning helps, more data helps, larger models help—holds up. The quantitative scaling law does not.\n\nThis paper is for the low-resource ASR / music captioning community. As a workshop submission it has value; as a journal paper the scaling-law claim is a load-bearing problem. I would not publish it as is. If the authors release the dataset, recompute standard WER, and validate the fit on held-out model/data combinations, it could become a useful reference. For a serious venue, I'd send it to referees because the dataset and fine-tuning comparison are citable, but with a clear request for major revision.\n\nGiven the central scaling result is unsupported, my verdict is reject/revision.","headline":"Good dataset and plausible fine-tuning gains, but the scaling-law 'validation' is in-sample and the WER metric is nonstandard, so the headline quantitative claim is not supported.","tokens_in":5971,"tokens_out":4734,"would_cite":false,"duration_ms":53539,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that fine-tuning Whisper models on tens of hours of manually transcribed Jamaican Patois music sharply reduces word error rate, and that the remaining error follows a power law in model size and data size that can predict…","keywords":["Automatic Speech Recognition","Jamaican Patois","Music Transcription","Low-Resource Languages","Whisper fine-tuning","Scaling laws","Word Error Rate"],"falsifier":"Fine-tune Whisper Large on the same Patois music dataset at 20, 35, and 40 hours, and compare its measured word error rates to the values the fitted power law predicts for that model size and those data amounts; a systematic gap, or a large deviation at the 40-hour point, would show the law does not extrapolate across the Whisper family.","tokens_in":5100,"feed_emoji":"🎵","tokens_out":10031,"duration_ms":100430,"temperature":0.7,"pith_summary":"Jamaican Patois is widely spoken but nearly absent from speech datasets, so off-the-shelf speech recognition produces poor captions for Patois music; the paper reports a 0.89 word error rate for the untuned Whisper Large model. The paper's central claim is that a relatively small amount of curated data changes this: fine-tuning Whisper variants on 42.58 hours of manually transcribed Patois music lowers word error rate to 0.70 for the smallest model and 0.30 for the medium model, all beating the much larger zero-shot Whisper Large. The paper further claims that the measured word error rates obey a power law, $WER = 158.06 \\cdot M^{-0.255} \\cdot D^{-0.269}$, so performance at other model sizes and data amounts can be predicted without running new experiments. If this is right, it provides a concrete data-centric route to accessible captions for Patois music and to building language technology for a low-resource creole language.","feed_headline":"Fine-tuned Whisper models beat zero-shot Whisper Large on Patois music","feed_subtitle":"A 42-hour transcribed dataset drops Patois music word error from 0.89 to 0.30.","key_machinery":"The load-bearing object is the power-law scaling relation $WER = A \\cdot M^{-\\alpha} \\cdot D^{-\\beta}$, where $M$ is the number of model parameters and $D$ is the number of training hours. The paper fits this relation to twelve experimental points, four Whisper variants at 20, 35, and 40 training hours, by linear regression on the log-transformed equation, yielding $\\log WER = 5.063 - 0.255 \\log M - 0.269 \\log D$. Its job is to turn a small grid of fine-tuning runs into a predictive tool for untested model sizes and dataset sizes. The enabling resource is the dataset itself: 5,110 thirty-second clips totaling 42.58 hours of manually transcribed Patois music, which the paper says is the largest public dataset of its kind.","core_discovery":"The discovery is that domain-specific fine-tuning on a modest transcribed corpus outweighs raw model scale for Patois music transcription. Fine-tuning Whisper Tiny on 40 hours of Patois music gives a word error rate of 0.70, while the untuned Whisper Large sits at 0.89; the medium variant reaches 0.30 on the same data. The paper also reports that the error falls as a power law of model size and dataset size, with fitted exponents of -0.255 and -0.269, and uses that law to predict word error rates for configurations it did not train, including a fine-tuned Whisper Large. The agreement between the predicted and measured values in the tested grid is presented as validation that the scaling relation captures the behavior of the Whisper family on this task.","pith_inferences":["Because the 20-, 35-, and 40-hour subsets appear to be nested, refitting the scaling law on independent splits of the dataset would test whether the exponents are stable or an artifact of overlapping training data.","A direct check of the law's extrapolative power would be to fine-tune Whisper Large on the same data and compare its measured word error rate to the value the fitted curve predicts for that parameter count and dataset size.","If the scaling behavior carries over to spoken Patois, the same data-centric recipe could seed broader language technology for the language, including conversational ASR and text corpora for language models.","The fitted exponents suggest that, over the tested range, increasing model size and increasing training hours contribute roughly comparable error reductions, so the choice between compute and data collection is a practical trade-off rather than a foregone conclusion."],"forward_implications":["Fine-tuned Whisper Tiny, roughly 50 times smaller than Whisper Large, transcribes Patois music more accurately than the untuned Large model, showing that targeted fine-tuning can beat raw model scale for low-resource musical dialects.","The fitted scaling law lets a practitioner estimate the expected word error rate for any Whisper variant at any training-data budget before spending compute, guiding model selection and data collection.","Extrapolating the law to larger fine-tuned models predicts further word error reductions, which the paper takes as evidence that additional model scale will continue to help Patois transcription.","The improved transcriptions stand to benefit downstream systems that consume ASR output for Jamaican music, such as the LLM-based lyric filtering pipeline discussed in the paper."],"supporting_citations":[{"why":"Supplies the Whisper model family that is fine-tuned and the pre-trained Whisper Large baseline that the paper's results must beat.","marker":"[12]"},{"why":"Provides the power-law scaling hypothesis for neural language models that the paper adapts into its Eq. (2).","marker":"[7]"},{"why":"Offers prior evidence that scaling laws apply to acoustic models, justifying the log-linear fit to WER.","marker":"[4]"},{"why":"Describes the downstream ASR-to-LLM pipeline for cleaning Jamaican music that the fine-tuned models are intended to improve.","marker":"[14]"}],"fun_headline_variants":["Fine-tuning beats scale for Patois music transcription","40 hours of Patois data cuts Whisper error from 0.89 to 0.30","Whisper fine-tuned on 40h Patois music beats Large model","Fine-tuning beats model size for Patois music ASR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's predictive claims rest on the assumption that word error rate falls as a clean power law of model size and training hours, and that the twelve measured points are reliable and independent enough to pin down the three fitted parameters; if that shape is wrong, the predicted word error rates for untested configurations do not follow.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning beats scale for Patois music transcription","40 hours of Patois data cuts Whisper error from 0.89 to 0.30","Whisper fine-tuned on 40h Patois music beats Large model","Fine-tuning beats model size for Patois music ASR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000793,"raw_usage":{"total_tokens":3429,"prompt_tokens":820,"completion_tokens":2609,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":436,"completion_tokens_details":{"reasoning_tokens":2526}},"tokens_in":436,"tokens_out":2609,"duration_ms":21021,"temperature":1.0,"reasoning_tokens":2526,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:19:36.732834+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune Whisper Large on the same Patois music dataset at 20, 35, and 40 hours, and compare its measured word error rates to the values the fitted power law predicts for that model size and those data amounts; a systematic gap, or a large deviation at the 40-hour point, would show the law does not extrapolate across the Whisper family.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Whisper model family that is fine-tuned and the pre-trained Whisper Large baseline that the paper's results must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the downstream ASR-to-LLM pipeline for cleaning Jamaican music that the fine-tuned models are intended to improve."}],"review_version":1}