{"id":"bac04d8e-9467-4e5a-b892-c6564709f211","arxiv_id":"2412.19785","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Language-family prompt tuning and a BPE-token-extended tokenizer improve Whisper's WER and inference speed on eight Indian languages, with 250 added tokens per language as the best configuration.","lead":"A study extends Whisper for eight Indian languages by adding language-family prompts and expanding the tokenizer with new BPE tokens, reporting lower word error rates and faster inference on the Kathbath dataset. The work is a practical engineering contribution, but the claimed accuracy gains are mixed across model sizes and some benchmark comparisons are not controlled.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Core comparisons are confounded: proposed models receive 5 fine-tuning epochs while the W-M FT and W-S FT baselines receive 3, so the reported WER gains from prompt-tuning and tokenizer may be training-budget effects.","rationale":"The reader's weakest assumption focused on generalizability of y=250 and the validity of family prompts, with significance testing as a remedy. I agree those are secondary concerns. The primary load-bearing issue is the unequal training budget in the main comparisons: every proposed model was fine-tuned for 5 epochs while the non-proposed FT baselines were fine-tuned for 3. This makes the central WER comparisons uninterpretable as an ablation of prompt-tuning or tokenization. It is a concrete, checkable flaw, not a matter of disagreement with the field's consensus. The paper could be made acceptable by retraining baselines at matched epochs and adding error bars; hence the conditional verdict remains appropriate rather than a rejection.","tokens_in":7715,"tokens_out":11941,"duration_ms":113015,"concrete_test":"Retrain the W-M FT and W-S FT baselines (and the Table V 'w/o Prompt' variants) for exactly 5 epochs using the same Kathbath data, learning rate 1e-5, gradient accumulation steps, and seeds as the proposed models, then recompute Tables I and V. If the 5-epoch FT baseline reaches the same average WER as the prompt/tokenizer variants within run-to-run noise, the proposed techniques confer no measurable benefit; if a gap remains, the claims survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V's setup says: 'All the proposed prompting-based and tokenizer-based Whisper models are fine-tuned for five epochs and three epochs for all the remaining models.' If 'remaining models' includes the central baselines, then Table I's W-M FT (20.15% average WER) and Table V's W-S FT/W-L FT 'w/o Prompt' columns were trained for 3 epochs, while the proposed W-M FT w/ Pro (18.53%), W-M FT w/ Tok (19.67%), and the prompt variants were trained for 5 epochs. The headline WER improvements are therefore not controlled: extra gradient steps alone could produce the lower WER. This confound is more fundamental than Table II's in-sample choice of y=250, because it affects the internal ablations that support the paper's 'balance between optimal WER and inference speed' claim. Even if the tokenizer's speed benefit is genuine (it reduces generated tokens), the accuracy side of the trade-off is unsubstantiated unless the baselines are retrained for the same number of epochs.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes two modifications to OpenAI's Whisper for automatic speech recognition of eight Indian languages (four Indo-Aryan and four Dravidian) using the Kathbath dataset. The first modification is prompt-tuning with language-family tokens (<indo> and <dra>) prepended to the decoder input. The second is an extended BPE tokenizer that adds per-language tokens to Whisper's token head. The authors report experiments on Whisper Small, Medium, and Large, claiming that prompt-tuning improves WER, the tokenizer reduces inference time by cutting the number of generated tokens, and the combined model balances accuracy and speed.","tokens_in":7944,"tokens_out":5872,"duration_ms":53608,"significance":"If the empirical claims were supported, the paper would offer a practical, low-cost recipe for adapting a strong multilingual ASR model to related low-resource languages: family-level prompts require only two new tokens, and a small tokenizer extension reduces autoregressive decoding steps. The idea of exploiting language family structure through prompts is well motivated and potentially useful. However, the evaluation as presented is compromised by an unequal training budget between baselines and proposed models, test-set-based selection of the token count, and a complete lack of statistical significance testing; these issues presently prevent the paper from establishing its central accuracy claims. The inference-speed part of the claim is more plausible because it is tied to the mechanical reduction in generated tokens, but even that comparison is not cleanly reported.","major_comments":[{"comment":"The sentence 'All the proposed prompting-based and tokenizer-based Whisper models are fine-tuned for five epochs and three epochs for all the remaining models' means that the W-M FT, W-S FT, W-L FT baselines and the w/o Tok variants in Table IV are trained for three epochs, while every proposed variant is trained for five epochs. As a result, the WER improvements in Tables I, IV, and V are confounded with additional gradient steps; a three-epoch baseline is not an adequate control. This confound directly affects the paper's central claims that prompt-tuning improves accuracy and that the combined model achieves an optimal balance between WER and inference speed. The authors must retrain all models with a matched number of epochs (or matched number of optimizer steps) and re-report all accuracy tables.","section":"Section V, Experimental Setup"},{"comment":"The number of added tokens per language, y=250, is selected because it yields the best average WER on the same Kathbath test set used to report final results. This is a test-set selection bias: the reported 19.67% average for y=250 is the minimum over the four values {1000, 500, 250, 125} evaluated on the test set, so it is an optimistic estimate of generalization for that hyperparameter. Please select y on a held-out validation set or via nested cross-validation, and describe the selection procedure explicitly.","section":"Table II and Section V"},{"comment":"The comparison to WhisperX and Faster Whisper is not controlled for fine-tuning. The w/o Tok columns of Table IV show very high WER (e.g., 109.54% for Gujarati with WhisperX and 111.54% with Faster Whisper), which is consistent with no fine-tuning on the target languages, while the w/ Tok columns show much lower WER (14.99% and 26.22% respectively) after presumably fine-tuning with the extended tokenizer. Any improvement due to training on Kathbath is thus being attributed to the tokenizer. The authors must state explicitly whether the w/ Tok WhisperX/Faster Whisper models were fine-tuned, with what data, for how many epochs, and either match the training protocol across all systems or restrict the tokenizer comparison to models trained under identical conditions.","section":"Table IV"},{"comment":"The abstract claims that 'prompt-tuning enhances accuracy across various Whisper model sizes, including Small, Medium, and Large,' but Table V shows that for W-S FT the prompt variant degrades WER on Marathi (20.65 to 22.69), Bengali (17.84 to 18.61), and Malayalam (41.79 to 42.37), with an average improvement of only 24.54 to 23.81. This mixed evidence does not support the blanket claim. Please qualify the claim to the languages and model sizes where the improvement is consistent, or provide statistical evidence that the average improvement is reliable.","section":"Abstract and Table V"},{"comment":"No error bars, number of seeds, or significance tests are reported anywhere in the paper. Given that several headline differences are small (e.g., Table I: W-M FT at 20.15% average versus W-M FT w/ Pro + Tok at 20.05%), the observed differences may be within run-to-run noise. Report standard deviations over multiple seeds or a paired significance test (e.g., bootstrap over utterances) for the central comparisons in Tables I, IV, and V.","section":"All experimental sections"}],"minor_comments":[{"comment":"The added BPE tokens are said to be 'derived from datasets emphasizing common sequences in Indian languages,' but the corpus, its size, and the merge algorithm are not specified, making the tokenizer impossible to reproduce. Please disclose these details, ideally by releasing the tokenizer merge file and the vocabulary addition.","section":"Section IV.C"},{"comment":"The inference-time measurements are reported in minutes without specifying the total audio duration or the number of utterances timed. Please add this information so the reader can interpret the speed numbers.","section":"Tables I, IV, and V"},{"comment":"The example sentences for Hindi and Malayalam are not shown in the original script or in a transliteration, so the claims of 27 vs. 19 and 79 vs. 31 tokens cannot be checked. Include the actual sentences and their tokenizations.","section":"Table III"},{"comment":"Hyperparameters such as the optimizer, batch size, learning-rate schedule, warmup, and total number of training steps are not reported. Providing only epochs and gradient accumulation steps is insufficient for replication.","section":"Section V"},{"comment":"It is unclear whether the language-family prompt tokens (<indo>, <dra>) are new learnable embeddings and whether the <SOTP> token is the standard Whisper <|startofprev|> token. Please clarify the initialization and training of these prompt tokens.","section":"Section IV.B"},{"comment":"The conclusion contains a typo: 'our consistently experiments show' should be 'our experiments consistently show.'","section":"Section VI"},{"comment":"The notation W-m with m in {S, M, L} is defined in Section V, but later the paper uses W-S, W-M, and W-L without restating the mapping; consider adding a brief reminder in the table captions.","section":"Notation throughout"}],"recommendation":"major_revision","confidential_remarks":"The unequal training epochs between the proposed models and the baselines is a serious confound that likely invalidates the accuracy claims as currently reported. The underlying ideas are not inherently flawed, and a revised paper with matched training budgets, a held-out hyperparameter selection procedure, disclosed tokenizer details, and significance testing could make a useful contribution. I would also encourage the editor to ask the authors to deposit code, tokenizer files, and training configurations if the paper is to be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know before reading this paper is that its central WER comparisons are not controlled. Section V says all proposed prompt- and tokenizer-based models are fine-tuned for five epochs, while all remaining models get three. That means the W-M FT baseline in Table I and the \"w/o Prompt\" columns in Table V were trained for fewer steps than the proposed models. The reported accuracy gains from prompt-tuning and the tokenizer could just be extra gradient steps. That is a load-bearing flaw, not a minor omission.\n\nThe speed side of the story is more plausible. Table III shows that adding BPE tokens cuts generated token counts substantially for Hindi and Malayalam, which should directly reduce autoregressive decoding time. But Table IV is again confounded: the \"w/ Tok\" rows for WhisperX and Faster Whisper show WER dropping from roughly 87% to 19%, which makes sense only if those models were fine-tuned, while the \"w/o Tok\" baselines are presumably off-the-shelf. That is not a controlled comparison.\n\nTo give credit: the combination of language-family prompts (<indo>/<dra>) with BPE extension for eight Indian languages on Kathbath is a reasonable application of known techniques. The paper is clearly written and the token-count intuition is useful. But the methods themselves are incremental: prompt tuning for Whisper is in [7] and [15], and BPE extension is in [8] and [17]. The novelty is mostly in the specific combination and the evaluation.\n\nOther soft spots: y=250 is chosen on the same test set used to report final numbers, so the token-count sweep in Table II is in-sample selection. The abstract's claim of accuracy gains across all model sizes is contradicted by the Small results in Table V, where Marathi, Bengali, and Malayalam degrade with the prompt. There are no error bars or significance tests anywhere, and no code or detailed tokenizer training procedure is released.\n\nThis paper deserves a serious referee, but only with the expectation of major revision. The fixes are straightforward: train all baselines for five epochs, validate y on held-out data, run a properly fine-tuned WhisperX/Faster Whisper baseline, and report confidence intervals. If the authors do that, the result could be a modest but useful engineering recipe for low-resource Indian-language ASR. As is, I would not cite it for the accuracy numbers, and I would keep the speed claim tentative. Send it to review with those conditions, or desk reject if the authors are unwilling to rerun the experiments.","headline":"The accuracy claims are confounded by unequal training epochs; the speed idea is plausible but the paper needs major revision before the results can be trusted.","tokens_in":8446,"tokens_out":3015,"would_cite":false,"duration_ms":32407,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding language-family prompts and per-language tokenizer tokens improves Whisper's accuracy and cuts inference time on eight Indian languages.","keywords":["automatic speech recognition","Whisper","prompt tuning","tokenizer","Indian languages","low-resource languages","inference speed","word error rate"],"falsifier":"Run the same prompt-tuning and tokenizer modifications on a held-out Indian-language corpus that was not used to pick the token count, or on Indo-Aryan and Dravidian languages outside the tested eight. The claim would be falsified if, on those languages, removing the family prompt does not raise WER or adding 250 tokens per language does not reduce generated-token counts and decoding time.","tokens_in":7505,"feed_emoji":"🎙️","tokens_out":8425,"duration_ms":74706,"temperature":0.7,"pith_summary":"This paper tries to establish that Whisper can be made both more accurate and much faster on eight Indian languages by two cheap modifications that touch only its token interface. The first is prompt-tuning with language-family information: the decoder is given a token such as <indo> for Indo-Aryan languages or <dra> for Dravidian languages, so linguistically related languages can share phonetic and structural knowledge during fine-tuning. The second is a tokenizer extension that adds 250 language-specific byte-pair-encoding tokens per language, shrinking the number of tokens each transcription requires and therefore the number of autoregressive decoding steps. On the paper's Kathbath experiments, prompt-tuning yields the best word error rate, the tokenizer yields the fastest inference, and the two together give a tunable balance between accuracy and speed. This would matter because it offers a low-cost path to making a general-purpose multilingual ASR model serve low-resource languages without retraining the architecture.","feed_headline":"Family prompts and extra tokens cut Whisper's error rate and latency","feed_subtitle":"Two fine-tuning tweaks make Whisper transcribe eight Indian languages faster and more accurately.","key_machinery":"The load-bearing mechanism is the decoder token interface of Whisper. At inference, Whisper conditions on a fixed token prefix (<SOT>, language ID, task, no-timestamps); the paper inserts an extra learned token <SOTP> followed by a family prompt (<indo> or <dra>) before this prefix, so the decoder can condition on family-level phonetic regularities. For speed, the paper extends the byte-pair-encoding tokenizer that Whisper reuses, adding 250 new BPE tokens per language and widening the final token-head projection with random rows for those tokens while keeping the original weights. The tokenizer change compresses each output sentence into fewer tokens, and because decoding is autoregressive, fewer tokens means fewer decoder steps; the prompt change re-routes the decoder's attention through shared family structure, and that shared structure is what carries the accuracy gain.","core_discovery":"On the paper's own terms, the discovery is that Whisper's multilingual decoder can be steered toward Indian languages by two complementary interventions that touch only its token interface. Inserting a learned language-family prompt (<indo> or <dra>) before the start-of-transcription token during fine-tuning encodes the linguistic similarity shared by Hindi, Gujarati, Marathi, and Bengali on one side and Tamil, Telugu, Kannada, and Malayalam on the other, cutting average WER from the multilingual fine-tuned baseline of 20.15% to 18.53% in the best prompt-only configuration on Whisper Medium. Replacing the stock tokenizer with one augmented by per-language BPE tokens (250 per language) shrinks the number of generated tokens—27 to 19 for a Hindi sentence and 79 to 31 for Malayalam—and with it the wall-clock inference time, from 45.15 to 23.75 minutes on average for the fine-tuned Medium model. Using both modifications together retains most of the WER gain and most of the speed gain, which the paper presents as an accuracy-latency trade-off the user can adjust.","pith_inferences":["A natural extension the paper does not run is to test the same two modifications on languages from other families, such as Tibeto-Burman or Austroasiatic languages of India; if the family-prompt gain is real, it should transfer to any family with enough shared phonetics, not just Indo-Aryan and Dravidian.","Because the decoder's cost scales with the number of tokens, the tokenizer's speed advantage should grow with utterance length and in streaming or real-time settings; the reported batch timings may understate that benefit.","The token count was selected on the same Kathbath test set used for final reporting, so the claimed 250-token optimum is likely optimistic; a held-out development set would be needed to know the true out-of-sample optimum.","The two mechanisms are not shown to be additive in the paper—the combined model is slightly worse than the prompt-only model in WER and worse than the tokenizer-only model in speed—so the balance claim is best read as an engineering trade-off rather than evidence of complementary gains."],"forward_implications":["With prompt-tuning alone, Whisper Medium's average WER drops from 20.15% (multilingual fine-tuning) to 18.53%, and the paper reports lower WER for every one of the eight languages.","With the 250-token-per-language tokenizer alone, average inference time falls from 45.15 to 23.75 minutes on the Kathbath evaluation set while average WER stays close to the fine-tuned baseline at 19.67%.","Applying both techniques gives an intermediate operating point—20.05% WER and 39.35 minutes—so users can choose between the most accurate model, the fastest model, or a balance.","The tokenizer shortens a Hindi sentence from 27 to 19 generated tokens and a Malayalam sentence from 79 to 31 tokens, and the paper reports similar speed and WER benefits when the tokenizer is added to Whisper Small and to two other fast-inference Whisper baselines.","Prompt-tuning also helps smaller and larger models: average WER improves from 24.54% to 23.81% on Whisper Small and from 18.16% to 17.15% on Whisper Large."],"supporting_citations":[{"why":"Supplies the Whisper model that the paper fine-tunes and the pre-trained multilingual baseline for comparison.","marker":"[4]"},{"why":"Provides the Kathbath dataset, including the eight languages and the train/validation/test split used in all reported experiments.","marker":"[18]"},{"why":"Motivates language-prompt tuning for multilingual speech recognition, which the paper adapts to family-level prompts.","marker":"[7]"},{"why":"Supports the idea that adding language-specific tokens to a multilingual tokenizer improves low-resource languages, the basis for the tokenizer extension.","marker":"[17]"},{"why":"Defines the per-language fine-tuned baseline whose WER the paper compares against its multilingual models.","marker":"[9]"},{"why":"Serves as a fast-inference Whisper baseline for the tokenizer's inference-time and WER comparison.","marker":"[20]"},{"why":"Serves as the second fast-inference Whisper baseline for the same comparison.","marker":"[21]"}],"fun_headline_variants":["Family prompts and BPE tokens cut Whisper errors and speed up inference","Whisper tuned for Indian speech: family prompts and BPE tokens slash WER and latency","Cut Whisper's error rate and inference time with prompt-tuning and tokenization","Two tweaks make Whisper faster and more accurate on Indian languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that 250 added tokens per language—the number that scored best on the same Kathbath test set whose results are reported—and the two coarse family prompts are generalizable choices for other Indian languages and datasets; the paper does not validate them on held-out languages or with significance tests.","fun_headline_variants_meta":{"raw":{"variants":["Family prompts and BPE tokens cut Whisper errors and speed up inference","Whisper tuned for Indian speech: family prompts and BPE tokens slash WER and latency","Cut Whisper's error rate and inference time with prompt-tuning and tokenization","Two tweaks make Whisper faster and more accurate on Indian languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00033,"raw_usage":{"total_tokens":1819,"prompt_tokens":904,"completion_tokens":915,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":831}},"tokens_in":520,"tokens_out":915,"duration_ms":7387,"temperature":1.0,"reasoning_tokens":831,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:50:31.627970+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same prompt-tuning and tokenizer modifications on a held-out Indian-language corpus that was not used to pick the token count, or on Indo-Aryan and Dravidian languages outside the tested eight. The claim would be falsified if, on those languages, removing the family prompt does not raise WER or adding 250 tokens per language does not reduce generated-token counts and decoding time.","supporting_citations":[{"cited_title":"Indicsuperb: A speech processing universal performance benchmark for indian languages,","cited_arxiv_id":null,"evidence_quote":"Provides the Kathbath dataset, including the eight languages and the train/validation/test split used in all reported experiments."},{"cited_title":"Enhancing multilin- gual speech recognition through language prompt tuning and frame- level language adapter,","cited_arxiv_id":null,"evidence_quote":"Motivates language-prompt tuning for multilingual speech recognition, which the paper adapts to family-level prompts."},{"cited_title":"Improving low-resource languages in pre-trained multilingual language models,","cited_arxiv_id":null,"evidence_quote":"Supports the idea that adding language-specific tokens to a multilingual tokenizer improves low-resource languages, the basis for the tokenizer extension."},{"cited_title":"Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR,","cited_arxiv_id":null,"evidence_quote":"Defines the per-language fine-tuned baseline whose WER the paper compares against its multilingual models."},{"cited_title":"faster-whisper: Faster whisper transcription and translation model,","cited_arxiv_id":null,"evidence_quote":"Serves as the second fast-inference Whisper baseline for the same comparison."}],"review_version":1}