{"id":"9e85f595-3702-4dda-832c-ca2e311a7c89","arxiv_id":"2506.01439","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Whale, a 1.87B-parameter ASR model combining w2v-BERT and E-Branchformer, reports 2.4% WER on Librispeech test-clean and 3.4% CER on CSJ eval3, beating Whisper large-v3 and OWSM v3.1 on those benchmarks.","lead":"Whale is a 1.87-billion-parameter multilingual speech recognition model trained on 250,000 hours of audio in 144 languages. It reports strong English and Japanese benchmark scores, but the evaluation is clouded by the fact that some benchmark datasets also appear in its training data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper lists all four evaluation corpora (LibriSpeech, CSJ, CommonVoice, FLEURS) as training sources in §3.2 but never states that their official test splits were excluded; the headline 2.4% WER and 3.4% CER are uninterpretable until this is resolved.","rationale":"The reader's weakest_assumption—that benchmark test sets may not have been held out—is exactly the load-bearing concern. The paper's Section 3.2 explicitly names all four evaluation corpora as training sources, and Section 4 reports results on those same corpora without a single sentence stating that test splits were removed. This is not a disagreement with external consensus; it is a missing control internal to the experimental design. If the test splits were included in training, the central claim is not merely overstated but invalid, because the reported error rates would measure memorization. The concrete test I propose would settle the question by requiring the training manifests or, failing that, an external held-out Japanese evaluation. I considered other potential concerns—such as the abstract's unqualified 'outperforming' claim given Whisper's better FLEURS results—but that is a presentation issue that does not affect the validity of the reported numbers if the splits are clean. The reader's CONDITIONAL verdict is appropriate: the paper can be accepted only after the split exclusion is documented. My read does not move that verdict, so I set verdict_should_be to UNCHANGED.","tokens_in":9023,"tokens_out":7921,"duration_ms":82750,"concrete_test":"Ask the authors to provide the exact ESPnet data preparation scripts and training manifests, and to confirm in writing that the official test splits for LibriSpeech (test-clean/test-other), CSJ (eval1/eval2/eval3), CommonVoice (test split for en/ja/zh/de/fr), and FLEURS (test split for the listed languages) were excluded from all seven curriculum stages. Independently, run a leakage check by decoding a held-out Japanese spontaneous-speech set (e.g., a CSJ eval subset withheld from training or an external corpus such as JSUT or JVS) with the released Whale model (if/when available) and comparing against CSJ eval3; an absolute CER gap larger than 5 points would strongly suggest memorization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 enumerates training sources: open data includes CommonVoice, LibriSpeech, FLEURS, etc.; purchased data includes CSJ. Section 4 then reports evaluations on LibriSpeech (4.1), CSJ (4.2), CommonVoice (4.3), and FLEURS (4.4). Nowhere does the paper state that the official test splits of these corpora were excluded from the training corpus. Without such a statement, the reported superiority on LibriSpeech test-clean (2.4% vs Whisper's 2.5%) and especially CSJ eval3 (3.4% vs Whisper's 16.5%) may reflect training on the evaluation utterances rather than generalization. The magnitude of the CSJ gap (13.1 points) is precisely what one would expect from label leakage. Because no code or weights are released, the reader cannot verify split discipline from artifacts. The paper's own acknowledgement of domain mismatch on FLEURS (where Whisper still leads) makes the large CSJ advantage even more suspicious if CSJ was in training. This is the load-bearing assumption: if it fails, the central claim collapses; if it holds, the remaining evidence (showing Whisper ahead on most FLEURS languages) supports at best a 'comparable' statement, not the abstract's unqualified 'outperforming'.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Whale, a 1.87B-parameter multilingual ASR model that combines w2v-BERT SSL features, a 24-layer E-Branchformer encoder, a 6-layer Transformer decoder, and joint CTC-attention decoding. Training uses 250k hours across 144 languages, mixing public corpora, purchased datasets, and in-house Japanese recordings, with a 7-stage curriculum learning schedule and a final stage that updates previously frozen SSL parameters. Evaluation is reported on LibriSpeech, CSJ, CommonVoice, and FLEURS, with headline results of 2.4% WER on LibriSpeech test-clean and 3.4% CER on CSJ eval3, which the abstract claims outperform Whisper large-v3 and OWSM v3.1.","tokens_in":9308,"tokens_out":6325,"duration_ms":67678,"significance":"If the benchmark test splits are genuinely held out, the paper demonstrates a useful and credible recipe: combining a pretrained w2v-BERT representation with E-Branchformer, joint CTC/attention decoding, and a Japanese-heavy curriculum can beat large open and proprietary baselines on specific English and Japanese benchmarks. The architecture and training stages are described clearly, and the comparison tables cover a useful range of languages. However, no code or weights are released, the training/evaluation split discipline is never stated, and the abstract's 'outperforming' claim is not supported by the paper's own FLEURS and CommonVoice tables. The central empirical contribution therefore currently rests on an unverified assumption about held-out test sets.","major_comments":[{"comment":"Section 3.2 lists CommonVoice, LibriSpeech, FLEURS, and CSJ among the training sources, and Section 4 reports evaluation on those same corpora, yet the manuscript never states that the official test splits were excluded from the training mixture. The CSJ result is particularly affected: CSJ is a purchased training set, and the reported 3.4% CER on eval3 versus Whisper's 16.5% cannot be interpreted as generalization unless eval1/eval2/eval3 were held out. Please state per corpus which exact split was used, confirm that those utterances were not in the training data, and ideally provide a data recipe or split-removal script; without that, the headline numbers are unverifiable.","section":"§3.2, §4"},{"comment":"The abstract's claim that Whale 'achieves... outperforming Whisper large-v3 and OWSM v3.1' is contradicted by the paper's own Table 5, where Whisper large-v3 has lower WER/CER on most FLEURS languages (e.g., en 4.0 vs. 6.2, de 5.8 vs. 9.5, fr 5.3 vs. 9.7, th 30.0 vs. 37.9), and by Table 4, where Whisper wins on English CommonVoice (8.4 vs. 8.9). The claim should be scoped to the specific benchmarks and splits where it is actually true, or revised to 'comparable performance with wins on particular sets.'","section":"Abstract vs. §4.4, Table 5"},{"comment":"The manuscript does not state whether the baseline numbers for Whisper large-v3, OWSM v3.1, and OWSM CTC v3.1 were re-run under the same evaluation pipeline (including whisper-normalizer and decoding configuration) or quoted from prior publications. Because Table 2's OWSM numbers differ slightly from the cited OWSM v3.1 report, please specify the source and evaluation settings for every baseline cell so the comparisons are airtight.","section":"§4, Tables 2–5"}],"minor_comments":[{"comment":"The sentence 'The total amount of our training data is 250k' should specify '250k hours' to match Section 3.2.","section":"Introduction"},{"comment":"There are typos: 'E-Branhformer' in the Index Terms and 'FLERUS' in the Introduction should be 'E-Branchformer' and 'FLEURS'.","section":"Index Terms; §1"},{"comment":"The language-adaptation description refers to a 'target language ID' used at inference, but no language ID embedding or input token is described in the architecture; please clarify how the LID is supplied to the model.","section":"§2.2.2"},{"comment":"Only stage7 results are reported for CSJ, CommonVoice, and FLEURS; if stage6 results for these benchmarks are available, reporting them would help isolate the effect of SSL updating.","section":"Tables 3–5"},{"comment":"Applying whisper-normalizer to both references and hypotheses is reasonable for LibriSpeech and CommonVoice, but its effect on Japanese CSJ text should be explained, since fillers, punctuation, and normalizer behavior directly affect the reported CER.","section":"§4"},{"comment":"The note 'The blanks are unknown' is ambiguous: it should state explicitly whether the blank OWSM cells mean no training data or unknown training data hours.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The main issue is data-split discipline. Since this is a system paper from an industrial lab and no code or weights are released, I would request a clear per-corpus declaration of held-out splits before further consideration. If the splits are clean, the paper is a moderate but useful system description; if they are not, the central comparison is uninterpretable. I would also flag the abstract's overstatement relative to Tables 4 and 5, which should be corrected regardless."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Whale is a plausible engineering data point, not a methods paper. The authors assembled 250k hours across 144 languages, with a heavy Japanese in-house component, and coupled w2v-BERT features to an E-Branchformer encoder-decoder with CTC joint decoding and a seven-stage curriculum. That is real work, and the model appears competitive on several benchmarks. The resource table (Table 1) is useful, and they are honest that Whisper still wins on most FLEURS languages, which is a point in their favor.\n\nThe problem is the training/evaluation overlap. Section 3.2 explicitly names LibriSpeech, CSJ, CommonVoice, and FLEURS as training sources. Section 4 evaluates on those same corpora, and the paper never states that the official test splits were excluded. For LibriSpeech, test-clean is standard to put in training for some programs, but for CSJ the eval1/eval2/eval3 sets are the standard held-out portions. If those utterances were in the training mix, the 3.4% CER on eval3 is a memorization number, not a generalization number. The 13-point gap over Whisper on CSJ is exactly the kind of magnitude that leakage produces. I don't see any statement in the paper that rules this out, and no code or weights are released so readers can't check from artifacts.\n\nThat is a load-bearing omission. It may be a simple oversight — the authors might have used the standard splits without thinking to say so — but the manuscript as written does not support the headline claim. The abstract's 'outperforming Whisper large-v3 and OWSM v3.1' also overreaches the paper's own tables: Whisper leads on most FLEURS languages, and on LibriSpeech the difference is 0.1%. The accurate claim is 'comparable on English, leading on spontaneous Japanese.'\n\nAlso minor: the zero-shot adaptation in Table 5 shows a degradation on several languages despite claiming 'adaptation,' so that section's interpretation is a little loose.\n\nIf the train/test split discipline is confirmed in a revision, this paper is a valuable data point for Japanese-heavy ASR and for the practical benefit of including target-domain data. It deserves a serious referee because the question is important and the result, if genuine, is useful. But the referee should demand a clear statement of split exclusion and ideally a released model or at least an evaluation script. I would not reject on the architecture or novelty; it is an engineering contribution.\n\nMy take: conditional accept with that clarification, and the authors should temper the abstract to match their own tables.","headline":"Whale is a serious engineering effort, but the headline numbers are uninterpretable until the authors state that the official test splits of CSJ/LibriSpeech/CommonVoice/FLEURS were excluded from the 250k-hour training corpus.","tokens_in":9862,"tokens_out":1852,"would_cite":false,"duration_ms":18279,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Whale, a 1.87B-parameter multilingual ASR model, achieves a 2.4% WER on LibriSpeech test-clean and a 3.4% CER on CSJ eval3, outperforming Whisper large-v3 and OWSM v3.1.","keywords":["multilingual ASR","w2v-BERT","E-Branchformer","joint CTC-attention","curriculum learning","self-conditioned CTC","Whale","large-scale speech recognition"],"falsifier":"Compute the overlap between the exact audio files (by file path, speaker ID, or utterance fingerprint) in the training data and those in the CSJ eval1-3 and FLEURS test partitions; any nonzero overlap would invalidate the generalization reading of the headline numbers. Alternatively, evaluating the released model on a fresh, never-trained Japanese spontaneous-speech corpus and checking whether the CER stays near 3-5% would settle the claim.","tokens_in":8825,"feed_emoji":"🗣️","tokens_out":14492,"duration_ms":126375,"temperature":0.7,"pith_summary":"The paper introduces Whale, a 1.87-billion-parameter multilingual speech recognizer that combines a w2v-BERT self-supervised front end, a 24-layer E-Branchformer encoder, a 6-layer Transformer decoder, and a joint CTC-attention decoding strategy. It is trained on 250,000 hours across 144 languages, with a deliberately large Japanese component, using a seven-stage curriculum that ends by unfreezing the w2v-BERT parameters. The authors claim that Whale reaches a word error rate of 2.4% on LibriSpeech test-clean and a character error rate of 3.4% on CSJ eval3, surpassing Whisper large-v3 and OWSM v3.1. If correct, this shows that a transparent, mostly public-data model can match or beat proprietary systems on English and Japanese spontaneous speech, and that language-specific data collection is a decisive factor.","feed_headline":"2.4% WER, 3.4% CER: Whale speech model beats Whisper","feed_subtitle":"A 1.87B-parameter model trained on a Japanese-heavy, mostly public corpus outperforms Whisper and OWSM on English and Japanese.","key_machinery":"The central mechanism is the stacked architecture of a w2v-BERT front end (24 Conformer layers, pre-trained with masked prediction and contrastive loss), a 24-layer E-Branchformer encoder with self-conditioned CTC branches at layers 8 and 16, and a 6-layer Transformer decoder trained with a weighted CTC plus attention loss (0.3/0.7) and decoded by joint CTC-attention beam search. This carries the argument by showing that no new block is needed: scaling existing SSL plus an E-Branchformer encoder-decoder on a carefully mixed corpus—especially the 30k hours of Japanese—is enough to beat Whisper and OWSM on the claimed benchmarks.","core_discovery":"On its own terms, the paper's central discovery is that a large-scale multilingual ASR model built entirely from known components—w2v-BERT feature extraction, E-Branchformer encoding, and joint CTC-attention decoding—can achieve 2.4% WER on LibriSpeech test-clean and 3.4% CER on CSJ eval3, both better than the corresponding Whisper large-v3 and OWSM v3.1 numbers (Whisper: 2.5%/16.5%; OWSM: 2.4%/21.9%). The improvement on CSJ is especially large (3.4% vs 16.5% for Whisper), which the authors attribute to collecting about 30k hours of Japanese in-house data, roughly four times Whisper's Japanese data. They further report that the final curriculum stage, which updates previously frozen w2v-BERT parameters, is critical: it cuts LibriSpeech other WER from 5.8% to 4.5% while preserving clean-set performance. The paper also reports that the self-conditioned CTC language-adaptation mechanism did not help on most FLEURS languages, which they interpret as the SSL front end already absorbing the linguistic variability.","pith_inferences":["Applying the same curriculum plus a few hundred hours of targeted read speech to another under-resourced language (e.g., Swahili or Telugu) would test whether the Japanese result is a general property of the data-mixing recipe or specific to the 30k-hour Japanese corpus.","Since the zero-shot language adaptation failed, an alternative is to insert the language ID into the SSL layers themselves (e.g., via adapter tokens) and re-run the FLEURS evaluation, making the paper's negative result a natural baseline.","The paper's comparison corpus is about 250k hours; re-running the recipe at, say, 100k hours would separate how much of the gain is pure scale versus the curriculum and data mixture, which the current ablations do not disentangle."],"forward_implications":["If the reported numbers hold, a transparent recipe of public, purchased, and in-house data can yield a model that beats Whisper large-v3 on clean English and on Japanese spontaneous speech, suggesting that Whisper's web-scale data advantage is not absolute.","The 30k hours of Japanese in-house data, nearly all read speech, appears to be the decisive factor behind the CSJ result (3.4% CER vs Whisper's 16.5%), implying that targeted language-specific data collection can outperform generic web-scale sampling for a given language.","The stage-7 improvement on LibriSpeech other (5.8% to 4.5% WER) indicates that unfreezing and jointly fine-tuning the SSL front end is a cheap, high-impact step after large-scale curriculum training.","The model's weaker FLEURS performance relative to Whisper on many languages, and the failure of self-conditioned CTC adaptation, point to remaining domain-robustness and adaptation gaps that the paper itself flags."],"supporting_citations":[{"why":"Supplies the Whisper large-v3 baseline that Whale is compared against on LibriSpeech, CSJ, CommonVoice, and FLEURS.","marker":"[14]"},{"why":"Supplies the OWSM v3.1 baseline, the open Whisper-style model that Whale claims to surpass on English and Japanese.","marker":"[15]"},{"why":"Supplies the OWSM-CTC baseline, an encoder-only variant also compared on the same benchmarks.","marker":"[17]"},{"why":"Defines the w2v-BERT self-supervised front-end that Whale uses for feature extraction and later joint fine-tuning.","marker":"[18]"},{"why":"Defines the E-Branchformer block that forms the 24-layer encoder backbone.","marker":"[19]"},{"why":"Establishes the joint CTC-attention multi-task training and decoding method used by Whale.","marker":"[20]"},{"why":"Provides the LibriSpeech benchmark on which the 2.4% test-clean WER claim rests.","marker":"[23]"},{"why":"Provides the CSJ spontaneous-Japanese benchmark on which the 3.4% eval3 CER claim rests.","marker":"[24]"},{"why":"Provides the FLEURS corpus used to evaluate multilingual generalization across 15 languages and as a training source.","marker":"[6]"},{"why":"Provides the CommonVoice corpus used both as training data and as a multilingual evaluation benchmark (English, Japanese, Chinese, German, French).","marker":"[1]"}],"fun_headline_variants":["Whale ASR: 1.87B params, beats Whisper on EN and JP","Known parts, big data: Whale ASR outperforms Whisper","Whale w2v-BERT + E-Branchformer beats Whisper, OWSM","Why Whale wins: 30k hours Japanese speech data","Whale ASR: known components, big data, beats Whisper"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported benchmark scores assume that the evaluation splits of LibriSpeech, CSJ, CommonVoice, and FLEURS were excluded from the 250k-hour training corpus; the paper lists these corpora as training sources without ever stating that the test partitions were held out.","fun_headline_variants_meta":{"raw":{"variants":["Whale ASR: 1.87B params, beats Whisper on EN and JP","Known parts, big data: Whale ASR outperforms Whisper","Whale w2v-BERT + E-Branchformer beats Whisper, OWSM","Why Whale wins: 30k hours Japanese speech data","Whale ASR: known components, big data, beats Whisper"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000647,"raw_usage":{"total_tokens":2989,"prompt_tokens":982,"completion_tokens":2007,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":1905}},"tokens_in":598,"tokens_out":2007,"duration_ms":16297,"temperature":1.0,"reasoning_tokens":1905,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:43:12.072492+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the overlap between the exact audio files (by file path, speaker ID, or utterance fingerprint) in the training data and those in the CSJ eval1-3 and FLEURS test partitions; any nonzero overlap would invalidate the generalization reading of the headline numbers. Alternatively, evaluating the released model on a fresh, never-trained Japanese spontaneous-speech corpus and checking whether the CER stays near 3-5% would settle the claim.","supporting_citations":[{"cited_title":"Multilingual speech recognition with a single end-to-end model,","cited_arxiv_id":null,"evidence_quote":"Supplies the Whisper large-v3 baseline that Whale is compared against on LibriSpeech, CSJ, CommonVoice, and FLEURS."},{"cited_title":"Bytes are all you need: End-to-end multilingual speech recognition and synthe- sis with bytes,","cited_arxiv_id":null,"evidence_quote":"Supplies the OWSM v3.1 baseline, the open Whisper-style model that Whale claims to surpass on English and Japanese."},{"cited_title":"Scaling speech technology to 1,000+ languages,","cited_arxiv_id":null,"evidence_quote":"Supplies the OWSM-CTC baseline, an encoder-only variant also compared on the same benchmarks."},{"cited_title":"Less is More: Accu- rate Speech Recognition & Translation without Web-Scale Data,","cited_arxiv_id":null,"evidence_quote":"Defines the w2v-BERT self-supervised front-end that Whale uses for feature extraction and later joint fine-tuning."},{"cited_title":"W2v-BERT: Combin- ing contrastive learning and masked language modeling for self- supervised speech pre-training,","cited_arxiv_id":null,"evidence_quote":"Provides the LibriSpeech benchmark on which the 2.4% test-clean WER claim rests."},{"cited_title":"E-Branchformer: Branchformer with enhanced merging for speech recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the CSJ spontaneous-Japanese benchmark on which the 3.4% eval3 CER claim rests."},{"cited_title":"Common V oice: A Massively-Multilingual Speech Corpus,","cited_arxiv_id":null,"evidence_quote":"Provides the FLEURS corpus used to evaluate multilingual generalization across 15 languages and as a training source."},{"cited_title":"Whale: Large-Scale multilingual ASR model with w2v-BERT and E-Branchformer with large speech data","cited_arxiv_id":"2506.01439","evidence_quote":"Provides the CommonVoice corpus used both as training data and as a multilingual evaluation benchmark (English, Japanese, Chinese, German, French)."}],"review_version":1}