{"id":"41c82801-0b28-44f3-a312-b6dc5c9e2fc4","arxiv_id":"2412.13788","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new Arabic ASR leaderboard ranks 14 open-source models on five multi-dialect datasets and analyzes robustness, speaker bias, and efficiency.","lead":"This paper introduces an open benchmark that compares 14 open-source Arabic speech recognition models on five multi-dialect test sets, plus robustness, speaker, and efficiency analyses. It gives Arabic ASR developers a single reference for choosing general-purpose models, though training-data overlap and missing error bars require caution.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The acknowledged Common Voice training/test overlap is unquantified; the suspiciously low CV-18 WER for asafaya/hubert-large-arabic-transcribe (8.01% vs. 67.82% on SADA) suggests leakage that could materially shift mid-table rankings.","rationale":"This is the same load-bearing weakness the reader identified, and the manuscript itself supplies strong internal evidence for it: the asafaya/hubert-large model achieves 8.01% WER on CV-18.0, an outlier roughly three to five times better than its own performance on every other test set, and far better than any non-CV-trained model on the same set. That pattern is exactly what training/test overlap produces. The paper's own caveat in Section 2.2 and Section 3.5 is a concession of the risk, but the dismissal is not backed by data. Since the central deliverable is the ranking, an unquantified leak that can plausibly move mid-table models by several positions prevents the leaderboard from being treated as a definitive reference. The issue is easily testable via utterance-level overlap analysis, and the correction (or an explicit caveat plus recomputed ranks) is straightforward, so CONDITIONAL acceptance is appropriate rather than rejection. I agree with the reader's assessment and suggest no change to the verdict.","tokens_in":11167,"tokens_out":6106,"duration_ms":54033,"concrete_test":"For each evaluated model whose model card lists training on any Common Voice release (speechbrain/asr-wav2vec2-commonvoice-14-ar, jonatasgrosman/wav2vec2-large-xlsr-53-arabic, and asafaya/hubert-large-arabic-transcribe, with the last checked on HuggingFace), download the corresponding training-subset metadata and the CV-18.0 Arabic test split; compute utterance-level overlap using the `path` and `client_id` fields. If overlap is nonzero, re-evaluate those models only on the non-overlapping portion of the CV-18.0 test set, recompute the Average WER, and compare the resulting leaderboard order with Table 1. If any model moves more than one rank, the paper's Section 3.5 claim is unsupported and the leaderboard must be corrected or caveated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The leaderboard's central value is a trustworthy ranking of general Arabic ASR performance, so training/evaluation overlap is a direct threat to that claim. Section 2.2 acknowledges that Common Voice shuffles splits between releases and that models trained on earlier versions may be biased, but it dismisses the effect with 'this will not significantly impact the overall benchmark' without any quantitative check. The data themselves raise a red flag: in Table 1, asafaya/hubert-large-arabic-transcribe achieves 8.01% WER on CV-18.0, far lower than its scores on SADA (67.82%), MASC-clean (32.94%), MASC-noisy (50.16%), and MGB-2 (37.51%). No other model comes close to that CV-18.0 number, and for the CV-fine-tuned speechbrain and jonatasgrosman models the CV-18.0 scores are also disproportionately low relative to other test sets. Because the paper does not report the training data for hubert-large (or for whitefox123/w2v-bert-2.0), the most plausible explanation is that these models were fine-tuned on an earlier Common Voice release whose utterances leaked into the CV-18.0 test split. Even if only the hubert model is affected, its Average WER would rise from 39.29% to roughly 47% if CV-18.0 were excluded, dropping it several places in the table. The two explicitly CV-fine-tuned models sit near the bottom, so their inflation would not change the top of the leaderboard, but the middle of the table and the paper's claim of a reliable general-performance reference are at stake. The assertion that the impact is negligible needs empirical support before the rankings can be treated as definitive.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Open Universal Arabic ASR Leaderboard, a continuous benchmark for open-source Arabic ASR models. It evaluates 14 models in zero-shot mode on five multi-dialect test sets (SADA, Common Voice 18.0, MASC test-clean, MASC test-noisy, and MGB-2), reporting WER/CER, average rankings, robustness across acoustic conditions and dialects, speaker age/gender analysis, and inference efficiency/memory. The authors claim this is the largest Arabic ASR benchmark to date and release evaluation code and a public leaderboard.","tokens_in":11490,"tokens_out":5019,"duration_ms":46375,"significance":"If the reported rankings are trustworthy, this would be a valuable community resource: it covers most currently available open Arabic ASR models, uses multiple multi-dialect datasets, and adds practical efficiency and robustness dimensions that most prior Arabic ASR comparisons lack. The paper ships open-source evaluation code, states a clear text-normalization recipe, and documents model sources. The main risk is training/test overlap with Common Voice, which the paper acknowledges in Section 2.2 and Section 3.5 but does not quantify; the suspiciously low CV-18.0 WER of one model suggests leakage and could change mid-table rankings.","major_comments":[{"comment":"The claim that Common Voice split shuffling 'will not significantly impact the overall benchmark' is not supported by evidence. In Table 1, asafaya/hubert-large-arabic-transcribe achieves 8.01% WER on CV-18.0, versus 32.94% on MASC clean, 50.16% on MASC noisy, 37.51% on MGB-2, and 67.82% on SADA; excluding CV-18.0 raises its average WER from 39.29% to about 47.1%, dropping it below whisper-medium. whitefox123/w2v-bert-2.0-ft also shows a disproportionately low CV-18.0 WER (24.18% vs 35.93% on MASC clean and 78.02% on SADA). Because the paper does not report the training data for these community models, leakage cannot be ruled out. Please quantify the overlap (e.g., deduplicate CV-18.0 test against previous CV releases used in training, or report the training corpus compositions) and show the leaderboard with and without CV-18.0, or flag CV-18.0 results for CV-trained models as potentially biased.","section":"Section 2.2, Section 3.5, Table 1"},{"comment":"MGB-2 is stated to be over 70% MSA, yet the Average WER gives MGB-2 the same weight as the other four test sets. This tilts the 'general' ranking toward MSA-dominated performance and away from a balanced multi-dialect assessment. Please report the dialect distribution of every test set and provide a sensitivity analysis (e.g., per-dialect WERs, dialect-weighted averaging, or one average excluding MGB-2) to justify the unweighted five-way average as a 'general' performance measure.","section":"Section 2.2, Section 3.5, Table 1"},{"comment":"For Seamless M4T, the authors run three separate inferences with different Arabic dialect configuration tags and report only the best result. This is a model-specific selection that is not applied to any other model in the benchmark, so Seamless's rank-4 position is not directly comparable with the zero-shot scores of the other models. Please report the WER for all three configurations or justify a fixed, principled configuration for leaderboard reporting.","section":"Section 2.1, Table 1"},{"comment":"The leaderboard ranks are point estimates with no confidence intervals or significance tests. Several adjacent ranks differ by less than 1% WER (e.g., ranks 5 and 6: 33.30% vs 34.04%; ranks 8 and 9: 39.29% vs 39.60%), so the stability of the ordering is unclear. Please add bootstrap confidence intervals or pairwise significance tests for at least the top and mid-table comparisons, or explicitly state that adjacent small differences should not be interpreted as reliable ordering.","section":"Table 1"}],"minor_comments":[{"comment":"The column header '#Mem model' appears twice; based on the caption, the second column should be labeled '#Mem infer' to distinguish model-loading memory from per-sample inference memory.","section":"Table 5"},{"comment":"Table 1 uses commas as decimal separators (e.g., '25,71%') while the text uses periods (e.g., '39.29%'); please make the decimal separator consistent throughout.","section":"Table 1"},{"comment":"The HuggingFace and GitHub URLs for the leaderboard and code should be checked for line-break corruption in the PDF; the GitHub URL in the text appears to be broken across lines.","section":"Section 3.5"},{"comment":"The phrase 'largest Arabic ASR benchmark to date' in Sections 1 and 4 is used without defining 'largest' (number of models, number of datasets, or total test hours); please state the criterion explicitly.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid benchmark contribution and the authors are transparent about several limitations, but the Common Voice leakage concern is central to the ranking claim and needs a quantitative treatment before publication. The MGB-2 weighting and Seamless best-of-three selection also need either justification or adjustment. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful benchmark paper, and you should know that the mid-table rankings have a real leakage problem that the authors acknowledge but don't quantify. The stress-test note is right about asafaya/hubert-large-arabic-transcribe: 8.01% WER on Common Voice 18.0 versus 67.82% on SADA is not a normal generalization gap. The model was fine-tuned on an earlier Common Voice release, and Common Voice reshuffles splits between releases. The paper says in Section 2.2 that this \"will not significantly impact the overall benchmark\" without showing any numbers. That's the load-bearing weakness.\n\nWhat the paper does well: it's a practical, broad reference. Fourteen models, five multi-dialect test sets, plus robustness, speaker, and efficiency analyses. The code and leaderboard are open, and the normalization is stated. The rankings are internally consistent, and the paper doesn't overclaim beyond \"largest Arabic ASR benchmark to date,\" which seems fair.\n\nThe soft spots are exactly where the reader put them. No confidence intervals or significance tests—with WER differences of a few points between adjacent models, you can't tell if rank 3 really beats rank 4. The MGB-2 test set is over 70% MSA, which the authors flag, but the Average WER weights it equally with the others. That's a reporting choice, not a flaw per se, but it should be in the table.\n\nOn the leakage: the stress-test math checks out. If you drop CV-18.0 for the hubert model, its average rises from 39.29% to roughly 47%, which would move it from rank 8 into the middle of the pack. The whitefox123, jonatasgrosman, and speechbrain models show the same pattern in less extreme form. The top of the table is probably safe—the Conformer and Whisper models were not trained on Common Voice—so the headline result survives. But the middle is not trustworthy as-is.\n\nThis is a paper for the Arabic ASR community and for anyone building evaluation frameworks for multilingual speech. It's infrastructure, not a scientific breakthrough, and that's fine. It deserves a serious referee: the contribution is real, the code is out, and the problems are fixable rather than fatal. I'd send it to review with a request to quantify the leakage, add error bars or significance tests, and consider per-dataset averages as a supplementary table.","headline":"Useful, well-scoped Arabic ASR benchmark that deserves referee time, but the unquantified Common Voice leakage means the mid-table rankings shouldn't be treated as final.","tokens_in":12027,"tokens_out":2691,"would_cite":false,"duration_ms":22444,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces the Open Universal Arabic ASR Leaderboard, a continuous five-dataset benchmark that ranks open Arabic speech models by average zero-shot WER.","keywords":["Arabic ASR","multidialect speech recognition","ASR benchmark","leaderboard","zero-shot evaluation","Word Error Rate","speaker robustness","Conformer-CTC"],"falsifier":"Compute utterance-level text overlap between the Common Voice 18.0 test split and the training data of any evaluated model that was fine-tuned on earlier Common Voice versions; if there is substantial overlap, re-run the ranking without Common Voice and see whether the Conformer-CTC plus language model still holds first place.","tokens_in":10967,"feed_emoji":"🎙️","tokens_out":8598,"duration_ms":70213,"temperature":0.7,"pith_summary":"This paper establishes a common reference for open-source Arabic speech recognition by building a multi-dialect benchmark leaderboard. It evaluates fourteen models zero-shot on five multi-dialect test sets—SADA, Common Voice 18.0, MASC test-clean, MASC test-noisy, and MGB-2—and ranks them by average Word Error Rate. The top-ranked system is the Conformer-CTC-large-Arabic model combined with a 4-gram language model, at 25.71% average WER, ahead of the Whisper large series and Seamless M4T. The same evaluation also measures robustness to noise and music, dialect-specific behavior, age and gender differences, and inference speed and memory, so the leaderboard doubles as a production-oriented survey.","feed_headline":"Conformer-CTC tops the first large Arabic ASR leaderboard","feed_subtitle":"Fourteen open models ranked zero-shot on five multidialect Arabic test sets, plus robustness and speed.","key_machinery":"The named object is the Open Universal Arabic ASR Leaderboard, a continuously updated ranking built from a multi-dataset zero-shot evaluation protocol. Fourteen open models transcribe five exclusively multi-dialectal Arabic test sets without any adaptation, and a text-normalization pipeline removes punctuation and diacritics, normalizes Hamza and Madda, and unifies numerals before WER and CER are computed. The average WER across the five sets is the single ranking number that carries the paper's argument, while the per-dataset columns expose dialect-specific strengths and weaknesses.","core_discovery":"The paper's central claim is that a single continuous leaderboard can fairly rank general Arabic ASR models, and that when ranked this way the current open-source landscape has a clear order. The Conformer-CTC-large-Arabic model, decoded with its released 4-gram language model, leads with 25.71% average WER and 10.02% average CER across the five zero-shot test sets; greedy decoding places it second, still ahead of Whisper large-v3, Seamless M4T, and the Whisper large-v2/v1 variants. The authors find that performance tracks the amount of labeled Arabic training data more than model scale: the Conformer was trained on about 3,600 hours, the Whisper models on about 739 hours, and the lowest-ranked self-supervised models on under 100 hours. Across dialects, every model does best on Modern Standard Arabic and clearly worse on Egyptian and Khaliji speech, and most models show higher error rates for elderly and female speakers.","pith_inferences":["A natural stress test the paper does not run: drop Common Voice 18.0 from the average and recompute the ranking, since the paper itself flags possible test leakage from Common Voice's reshuffled splits; if the top order shifts, the benchmark's headline ranking is partly an artifact of that leakage.","The per-dataset columns show that a model like the HuBERT fine-tune is much stronger on MASC clean than on MGB-2, so a practitioner targeting one dialect should not rely on the average rank alone; the leaderboard can support dialect-specific subsets as the community adds them.","If the leaderboard is adopted as a standard, it would be cheap to add native dialect labels (already present in SADA) as a first-class column, turning the robustness analysis into a per-dialect benchmark rather than a single average."],"forward_implications":["Arabic ASR practitioners get a single yardstick: new open models can be added to the same five test sets and compared directly against the published ranking.","The leaderboard's ranking suggests that investing in labeled Arabic speech data (the Conformer's 3,600 hours) pays off more than relying on large multilingual pretraining (Whisper's 739 Arabic hours).","The consistent MSA-over-dialect gap indicates that dialect coverage in training data, not model architecture, is the main obstacle to universal Arabic ASR.","The efficiency measurements give a practical trade-off: self-supervised models decode fastest and use least memory, while the top-ranking Conformer-CTC is still small and fast; Whisper large models are several times slower.","Because the benchmark is designed as a continuous project, the same evaluation framework can absorb future models and datasets without changing the comparison protocol."],"supporting_citations":[{"why":"Supplies the MGB-2 multi-dialect broadcast test set used in the average WER.","marker":"[3]"},{"why":"Supplies the SADA test set and the data for robustness, dialect, and speaker adaptation analyses.","marker":"[4]"},{"why":"Supplies the MASC test-clean and test-noisy multi-region subsets.","marker":"[5]"},{"why":"Supplies the Whisper model family that fills the middle of the ranking.","marker":"[12]"},{"why":"Supplies the Common Voice 18.0 test set and is the source of the acknowledged training-data leakage risk.","marker":"[18]"},{"why":"The existing speech leaderboard that motivates the continuous benchmark format and the minimal-load efficiency setup.","marker":"[23]"},{"why":"Supports the training-data scale claim for the top-ranked Conformer model via ASRSet 3.0.","marker":"[26]"},{"why":"Supplies the text-normalization recipe used before WER and CER computation.","marker":"[31]"}],"fun_headline_variants":["Conformer-CTC wins first open Arabic ASR leaderboard","Arabic ASR: More training data beats bigger models","Open leaderboard ranks 14 Arabic ASR models zero-shot","Conformer-CTC tops Arabic ASR, but dialects and age matter","First open Arabic ASR benchmark: Conformer-CTC leads"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings are not materially distorted by training-data overlap with the evaluation sets, particularly Common Voice, whose test splits change between releases.","fun_headline_variants_meta":{"raw":{"variants":["Conformer-CTC wins first open Arabic ASR leaderboard","Arabic ASR: More training data beats bigger models","Open leaderboard ranks 14 Arabic ASR models zero-shot","Conformer-CTC tops Arabic ASR, but dialects and age matter","First open Arabic ASR benchmark: Conformer-CTC leads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001115,"raw_usage":{"total_tokens":4604,"prompt_tokens":869,"completion_tokens":3735,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":3649}},"tokens_in":485,"tokens_out":3735,"duration_ms":26319,"temperature":1.0,"reasoning_tokens":3649,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:47:51.119399+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute utterance-level text overlap between the Common Voice 18.0 test split and the training data of any evaluated model that was fine-tuned on earlier Common Voice versions; if there is substantial overlap, re-run the ranking without Common Voice and see whether the Conformer-CTC plus language model still holds first place.","supporting_citations":[{"cited_title":"Masc: Massive arabic speech corpus,","cited_arxiv_id":null,"evidence_quote":"Supplies the MASC test-clean and test-noisy multi-region subsets."},{"cited_title":"Stateful conformer with cache-based inference for stream ing au- tomatic speech recognition,","cited_arxiv_id":null,"evidence_quote":"Supplies the text-normalization recipe used before WER and CER computation."},{"cited_title":"Fleurs: Few-shot learnin g evaluation of universal representations of speech,","cited_arxiv_id":null,"evidence_quote":"Supports the training-data scale claim for the top-ranked Conformer model via ASRSet 3.0."},{"cited_title":"Overall Leaderboard The overall leaderboard is shown in Table 1","cited_arxiv_id":null,"evidence_quote":"Supplies the MGB-2 multi-dialect broadcast test set used in the average WER."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SADA test set and the data for robustness, dialect, and speaker adaptation analyses."}],"review_version":1}