{"id":"7636f58e-0d94-4f64-aa5f-798ddc62054d","arxiv_id":"2501.00425","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Fine-tuning Wav2Vec2-xlsr-53 on Common Voice audio augmented with pitch shift, Gaussian noise, and band-stop filtering lowers WER and CER in Arabic, Russian, and Portuguese, though the Whisper comparison is overstated.","lead":"This paper tests whether adding pitch shifts, noise, and band-stop filters to training audio improves a Wav2Vec2 speech recognition model for Arabic, Russian, and Portuguese. It reports large error reductions versus the un-augmented model, but the comparison to the Whisper model is not supported by the numbers as presented.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 33.9%/53.2% improvements are computed against clean Wav2Vec2 (Tables 2–3), not against Whisper; the Whisper baseline is Whisper-tiny without the custom normalizer, so the abstract's central outperformance claim lacks supporting numbers.","rationale":"The reader's weakest assumption correctly targets the fairness of the Whisper baseline, and that concern is real: Whisper-tiny is not a state-of-the-art comparison; it is not fine-tuned; and the paper's own §2.1 flags that Whisper uses a custom normalizer, without establishing that it was applied here. I agree with that concern, but I see a more decisive issue: the abstract's headline percentages do not come from the Whisper comparison at all. They are the clean-versus-augmented Wav2Vec2 improvements from Tables 2 and 3. The arithmetic shows this directly, and the Whisper comparison in Table 4 would yield a different average even if it were fair. Additionally, the CER numbers conflict by roughly a factor of two across §6.1, Figure 3, Table 3, and §6.4, so the magnitude of the claimed CER improvement is not stable within the manuscript. The augmentation idea may well be useful, and a clean Wav2Vec2 improvement would be a plausible finding, but the central claim as stated—outperforming Whisper with 33.9%/53.2% average relative gains—is not supported by the evidence presented. Therefore the reader's REJECT verdict is appropriate, and I would keep it unchanged.","tokens_in":15601,"tokens_out":3957,"duration_ms":35977,"concrete_test":"Reproduce the pipeline on the same Common Voice ar/ru/pt test splits with the released/pre-trained checkpoints. Evaluate Whisper-tiny twice: with and without the official Whisper normalizer, and compute CER as well as WER. Then recompute the average relative improvements against Table 2/Table 3 and against Table 4. If the Whisper-tiny normalizer changes its WER materially, or if the average improvement versus Whisper differs from the claimed 33.9%/53.2%, the abstract's comparison is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Abstract and §1.2 claim the framework 'outperforms two previous baseline models, which are the pre-trained Wav2Vec2 and the well-known Whisper ASR model, resulting in an average relative improvement of 33.9% in WER and 53.2% in CER.' Those percentages, however, are the average of the clean-vs-augmented Wav2Vec2 improvements in Table 2 (40.65, 34.43, 26.56 → 33.9) and Table 3 (59.64, 54.26, 45.75 → 53.2). They are not relative to Whisper. Recomputing from Table 4 gives ~69.6% for Arabic, 11.8% for Russian, and 9.7% for Portuguese WER, averaging ~30.4%, not 33.9%; CER is not given for Whisper at all. The Whisper baseline is also not a state-of-the-art comparison: it is Whisper-tiny, not fine-tuned, and §2.1 says Whisper uses a custom text normalizer before WER computation, but the paper does not state that this normalizer was applied. If it was not, the reported Whisper WERs are inflated. Finally, the CER evidence is internally inconsistent: §6.1 and Figure 3 give Arabic clean CER 37.5% and augmented 19.0%, Table 3 gives 22.3% and 9.0%, and §6.4 gives 37.3% and 19.0%. A factor-of-two discrepancy in the metric underpinning the 53.2% headline claim means the central result cannot be verified from the manuscript as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an end-to-end ASR framework for low-resource languages (Arabic, Russian, Portuguese) by fine-tuning Wav2Vec2-xlsr-53 on a combination of clean audio plus three data augmentations: pitch shift, Gaussian noise, and band-stop filtering. The authors report average relative improvements of 33.9% in WER and 53.2% in CER, and claim that this approach outperforms both a standard Wav2Vec2 baseline and OpenAI's Whisper model. The evaluation uses Mozilla Common Voice training splits of 17 hours (Arabic, Portuguese) and 30 hours (Russian). A related finding is that the augmentation combination was selected based on Arabic-language experiments and then transferred to the other languages.","tokens_in":15983,"tokens_out":4179,"duration_ms":38190,"significance":"If the claims were fully supported, the paper would provide a simple, low-cost recipe for improving Wav2Vec2 in low-resource settings, which is a practically useful result. The work does include a direct empirical comparison of several augmentation combinations and reports both WER and CER, and the chosen augmentations are straightforward and easy to reproduce. However, the headline numbers are not actually computed against the stated baselines, and there are internal inconsistencies in the CER results, so the significance cannot be assessed from the manuscript as written.","major_comments":[{"comment":"The abstract and §1.2 claim that the approach 'outperforms two previous baseline models, which are the pre-trained Wav2Vec2 and the well-known Whisper ASR model, resulting in an average relative improvement of 33.9% in WER and a 53.2% relative improvement in CER.' However, the 33.9% and 53.2% figures are the average clean-versus-augmented Wav2Vec2 improvements in Tables 2 and 3 (Arabic 40.65, Russian 34.43, Portuguese 26.56; Arabic 59.64, Russian 54.26, Portuguese 45.75). They are not improvements over Whisper. Recomputing from Table 4, the relative WER improvements over Whisper are roughly 69.6% (Arabic), 11.8% (Russian), and 9.7% (Portuguese), with no CER values for Whisper. The central claim of outperforming Whisper is therefore not supported by the numbers actually presented.","section":"Abstract, §1.2, Table 2, Table 3, Table 4"},{"comment":"The Whisper baseline is not an apples-to-apples comparison. Section 2.1 explicitly notes that Whisper's WER is computed using a custom text normalizer before scoring, but the paper never states whether this normalizer was applied in the Table 4 evaluation. If it was not, the reported Whisper WERs are inflated. Additionally, Table 4 uses Whisper-tiny, the smallest Whisper variant, with no fine-tuning, while the proposed model is fine-tuned on domain data. To substantiate the claim of outperforming Whisper, the authors need to evaluate a properly normalized Whisper model (ideally with several sizes, with and without fine-tuning) and report its CER as well.","section":"§2.1, §6.3, Table 4"},{"comment":"The CER values are internally inconsistent. Arabic clean CER is reported as 37.5% in §6.1 and Figure 3, but as 22.3% in Table 3; the augmented CER is 19.0% in §6.1 and Figure 3, but 9.0% in Table 3. Section 6.4 states the reduction as 'from 37.3% to 19.0%'. These factor-of-two discrepancies for the same metric and language mean the 53.2% improvement claim cannot be verified from the manuscript. The authors must reconcile the numbers and explain which settings (e.g., with or without diacritics) produce each value.","section":"§6.1, Table 3, §6.4"}],"minor_comments":[{"comment":"The phrase 'random samples from a uniform distribution over 0.001 to 0.03 Hz' is odd; the unit Hz is likely a typo, and the range for the noise amplitude ratio should be stated in a dimensionless way.","section":"§4.2"},{"comment":"The example of Portuguese sentences yields an 85.7% WER and 17.3% CER; it would be helpful to explain how these values are computed, as the word-level error seems surprisingly high for such a short sentence.","section":"§5.3.1"},{"comment":"The table title states 'WER and CER results' but the CER column is entirely N/A; the title should be changed to reflect that only WER is reported, and the reason (no CER in Radford et al.) should be stated in the caption in a more precise way.","section":"Table 4"},{"comment":"Several references are incomplete or inconsistently formatted (e.g., 'Avci and Akpolat [2006]' without full details in the text, and some URLs are missing). The paper would benefit from a thorough reference cleanup.","section":"General"},{"comment":"The selection of the augmentation combination is based on the Arabic development results (§6.1), and then the same recipe is applied to Russian and Portuguese without any validation on those languages. This should be acknowledged as a limitation or addressed by reporting a validation split for each language.","section":"§1.2, §6.2"}],"recommendation":"major_revision","confidential_remarks":"The overarching claim of the paper is significantly overstated relative to the evidence. The 33.9% WER and 53.2% CER improvements are not Whisper-relative, and the Whisper comparison itself is not fair. The CER inconsistencies are also concerning. I believe these are fixable with a substantial rewrite and new experiments, but if the authors cannot reconcile the CER numbers or provide a valid Whisper evaluation, the paper should be rejected. I would recommend the editor to ask for a major revision with a clear request for a corrected baseline comparison and consistent metric reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline numbers don't mean what the abstract says they mean. The 33.9% WER and 53.2% CER improvements are computed between clean and augmented Wav2Vec2 fine-tuning (Tables 2 and 3), not against Whisper. The Whisper comparison in Table 4 uses Whisper-tiny, not fine-tuned, reports no CER, and never states whether Whisper's custom text normalizer was applied. The paper's central claim therefore lacks a supporting comparison. Separately, the CER numbers are internally inconsistent: Arabic clean CER is 37.5% in Figure 3 and Section 6.1 but 22.3% in Table 3, and augmented is 19.0% in one place and 9.0% in another. That factor-of-two discrepancy sits on the metric that produces the 53.2% headline, so the main result cannot be checked from the manuscript.\n\nWhat is genuinely useful here is the empirical recipe. The authors fine-tune Wav2Vec2-xlsr-53 on Common Voice Arabic, Russian, and Portuguese with 20% each of pitch-shift, Gaussian-noise, and band-stop augmented data, and they observe consistent WER/CER gains across all three languages. The augmentation methods themselves are standard, and prior work already shows that augmenting Wav2Vec2 training helps low-resource ASR, so the novelty is incremental. But the specific combination and the consistent three-language results are worth having if they survive a clean re-run.\n\nThe softer problems: no error bars or multiple seeds, no code or model release, and the augmentation set was chosen after looking at Arabic results, so the Russian/Portuguese numbers are a single fixed-configuration test rather than a fully held-out evaluation. Those issues are manageable in revision.\n\nThe load-bearing problems are the misattributed headline and the CER inconsistencies. As written, the paper is not a reliable source for the claimed effect size. I'd reject the current version and tell the authors to fix the numbers, run a fair Whisper baseline (at minimum with the normalizer applied and CER reported, ideally with a larger Whisper variant), add error bars, and release code. If they do that, it becomes a modest but legitimate empirical contribution. A serious referee could give useful guidance, so I wouldn't desk-reject; I'd send it out, but I'd expect major revision.","headline":"The 33.9%/53.2% claims compare clean vs. augmented Wav2Vec2, not Whisper, and the CER numbers are internally inconsistent; the augmentation recipe is plausible, but this version can't be trusted.","tokens_in":16543,"tokens_out":3582,"would_cite":false,"duration_ms":32878,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Augmenting fine-tuning audio with three cheap transforms—pitch shift, Gaussian noise, and band-stop filtering—cuts Wav2Vec2's word error rate by a third and character error rate by half in Arabic, Russian, and Portuguese, and beats a…","keywords":["automatic speech recognition","Wav2Vec2","data augmentation","low-resource languages","pitch shift","Gaussian noise","band-stop filter","Mozilla Common Voice"],"falsifier":"Re-run Whisper-tiny on the same Common Voice test sets after applying the same punctuation removal and Whisper's custom text normalizer, and compute its CER; if Whisper's WER then drops below the augmented Wav2Vec2 values (27.6, 35.8, 31.8) or its CER is lower than 9.0, 10.2, 11.5, the central outperformance claim fails.","tokens_in":15372,"feed_emoji":"🎙️","tokens_out":9435,"duration_ms":72634,"temperature":0.7,"pith_summary":"The paper claims that a simple data-augmentation recipe substantially improves automatic speech recognition (ASR) in low-resource languages. Fine-tuning Wav2Vec2-xlsr-53 on clean Mozilla Common Voice audio plus 20 percent each of pitch-shifted, Gaussian-noised, and band-stop-filtered copies of the same clips lowers the average word error rate by 33.9 percent and the average character error rate by 53.2 percent relative to fine-tuning on clean audio alone. The augmented model also posts lower WER than a Whisper-tiny baseline on the same test sets for Arabic, Russian, and Portuguese. The paper presents this as a cheap, model-agnostic path to better speech recognition where labeled data is scarce.","feed_headline":"Data augmentation cuts Wav2Vec2 ASR word errors by a third","feed_subtitle":"Pitch-shift, noise, and band-stop training audio also beat Whisper on Arabic, Russian, and Portuguese.","key_machinery":"The load-bearing mechanism is the three-way augmentation schedule applied to Wav2Vec2-xlsr-53, a self-supervised speech encoder pre-trained on 53 languages and adapted with a CTC output head. The authors construct a 160-percent training set: the original audio plus three independent 20-percent subsets, each transformed by one of three operations—pitch shift, Gaussian-noise addition, and band-stop filtering. The band-stop filter spectrally masks a frequency range to simulate accent variation, Gaussian noise adds robustness, and pitch shift expands the speaker and prosody landscape. The claim is that this particular combination, at 20-percent proportion per transform, produces lower CER than any single transform or any pair, with the full three-transform model giving the best Arabic CER of 19.0 percent in the initial Arabic-only sweep.","core_discovery":"The central discovery is that a modest amount of augmented audio in the fine-tuning set sharpens a pre-trained Wav2Vec2 model's transcriptions far more than the sheer size of the added data would predict. For each language, the authors fine-tune Wav2Vec2-xlsr-53 on the clean train split plus an additional 20 percent of that split, copied under three transformations: pitch shifting by a uniformly random number of semitones in $[-6,6]$, adding amplitude-scaled Gaussian noise, and applying a band-stop filter that removes a frequency range within the 0--4000 Hz band. Using all three transforms together yields the best results of the seven combinations tested. On held-out Common Voice test sets, the augmented model reduces Arabic WER from 46.5 to 27.6 percent, Russian from 54.6 to 35.8, and Portuguese from 43.3 to 31.8, with corresponding CER falling to 9.0, 10.2, and 11.5 percent. The paper further claims that these numbers beat the Whisper-tiny baseline, whose Arabic WER alone is reported at 90.9 percent.","pith_inferences":["One implication the paper leaves implicit is that the 20-percent augmentation proportion may not be optimal; holding all else fixed and varying the per-transform percentage would map the benefit curve.","The recipe's generality could be tested by applying the same three transforms to a different pre-trained ASR architecture, such as HuBERT or a fine-tuned Whisper-small, to see whether the gains are architecture-specific.","Computing Whisper-tiny's CER with the same text normalization would reveal whether the character-level advantage is specific to Wav2Vec2 or a general effect of augmentation.","A practical takeaway: teams with only dozens of hours of labeled speech can use these three off-the-shelf audio transforms to cut error rates without collecting or labeling new data."],"forward_implications":["If the recipe transfers, other low-resource languages can adopt the same three augmentations without changing model architecture.","The reported gains imply that synthetic spectral and prosodic variation can offset limited labeled audio for pre-trained ASR models.","The large CER reductions suggest character-level errors—diacritics in Arabic especially—are disproportionately corrected by augmentation.","The method sets a simple, reproducible baseline for future low-resource ASR systems to compare against."],"supporting_citations":[{"why":"Supplies the Wav2Vec2 architecture and the CTC fine-tuning procedure that the augmentation recipe modifies.","marker":"Baevski et al. [2020a]"},{"why":"Identifies the Wav2Vec2-xlsr-53 multilingual pre-trained checkpoint fine-tuned in the experiments.","marker":"Deschamps-Berger et al. [2022]"},{"why":"Defines the Whisper-tiny baseline that the paper claims to outperform on WER.","marker":"Radford et al. [2023]"},{"why":"Provides the Mozilla Common Voice datasets used for training and evaluation in Arabic, Russian, and Portuguese.","marker":"Ardila et al. [2019]"}],"fun_headline_variants":["Tweak Wav2Vec2 audio, slash word errors by 34%","Augmented audio boosts Wav2Vec2, beats Whisper on 3 languages","Pitch, noise, filters: cheap ASR upgrade for low-resource languages","Three audio tweaks cut Wav2Vec2 errors, outperform Whisper"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed outperformance over Whisper rests on the premise that the Whisper-tiny baseline is a fair state-of-the-art comparison, measured under the same preprocessing and scoring conditions as the augmented Wav2Vec2 model, which the paper does not fully establish.","fun_headline_variants_meta":{"raw":{"variants":["Tweak Wav2Vec2 audio, slash word errors by 34%","Augmented audio boosts Wav2Vec2, beats Whisper on 3 languages","Pitch, noise, filters: cheap ASR upgrade for low-resource languages","Three audio tweaks cut Wav2Vec2 errors, outperform Whisper"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000554,"raw_usage":{"total_tokens":2701,"prompt_tokens":1065,"completion_tokens":1636,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":1549}},"tokens_in":681,"tokens_out":1636,"duration_ms":10979,"temperature":1.0,"reasoning_tokens":1549,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:51:08.411828+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Whisper-tiny on the same Common Voice test sets after applying the same punctuation removal and Whisper's custom text normalizer, and compute its CER; if Whisper's WER then drops below the augmented Wav2Vec2 values (27.6, 35.8, 31.8) or its CER is lower than 9.0, 10.2, 11.5, the central outperformance claim fails.","supporting_citations":[{"cited_title":"Investigating transformer encoders and fusion strategies for speech emotion recognition in emergency call center conversations","cited_arxiv_id":null,"evidence_quote":"Identifies the Wav2Vec2-xlsr-53 multilingual pre-trained checkpoint fine-tuned in the experiments."}],"review_version":1}