{"id":"caeb4397-85f2-4ef1-b25d-4cf15f62468e","arxiv_id":"2505.11572","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"ASR-FAIRBENCH introduces a live leaderboard and a Fairness-Adjusted ASR Score (FAAS) that combines word error rate with a demographic fairness penalty, applied to five speech recognition models.","lead":"The authors built an online leaderboard that scores speech recognition systems on both accuracy and fairness, using a 10% sample of Meta's Fair-Speech dataset and a new Fairness-Adjusted ASR Score. The tool gives developers and buyers a single number that penalizes models that perform worse for some demographic groups than for others.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FAAS results cannot be verified from the manuscript: Eq. (2) is not reproducibly defined and no per-model fairness scores or p-values are reported, so the central ranking claim is unsupported.","rationale":"I read the paper as introducing a reproducible fairness-aware ASR benchmark and using it to support a specific set of empirical rankings. For that central claim to hold, the FAAS metric must be well-defined, computable, and supported by reported results. The weakest place in the manuscript is not the sampling argument alone, though that is also weak: Table 1 reports only marginal entropies, not joint cell counts, so the 10% stratified sample could still produce sparse demographic combinations for mixed-effects Poisson regression. The more fundamental problem is that the fairness score cannot be verified even in principle from the submitted text. Equation (2) is malformed and unexplained, and no table or statistical summary connects the five benchmarked models to the stated FAAS values or to the claim of significant disparities. This is a verification failure, not a disagreement with the broader fairness-benchmarking consensus. The GitHub and HuggingFace artifacts might contain a working implementation, but the manuscript itself does not support the central numbers. The reader's REJECT verdict is therefore appropriate, and my concern does not change it. Machine-checked proofs are absent, and the parameter count of 4 does not mitigate the under-specification of the metric.","tokens_in":3074,"tokens_out":2819,"duration_ms":30854,"concrete_test":"Ask the authors to provide a self-contained derivation of Eq. (2) from Eq. (1), specifying all symbols (β0, βg, β_logRef, \\bar X) and the Poisson offset/likelihood, and show that Eqs. (3)–(7) follow algebraically. Then recompute FAAS for the five benchmarked models from that formula and report a complete table with per-attribute p-values and confidence intervals. If Eq. (2) cannot be derived or the recomputed values do not reproduce the stated ranking, the central claim fails. As a secondary check, run the full pipeline on all 26,471 utterances and compare FAAS ranks with the 10% sample to test whether sparse demographic cells change the leadership ordering.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central outputs—Whisper-medium FAAS 29.41, Whisper-tiny's better fairness despite higher WER, and the claim that fairness can outweigh slight WER gains—are presented only as statements in the conclusion. No table or figure reports the underlying values, confidence intervals, or p-values. More fundamentally, the metric itself is under-specified. Eq. (1) states a mixed-effects Poisson model for WER but does not define the random effects, likelihood, or offset needed to reproduce the disparity ratios. Eq. (2), the predicted WER used for the fairness score, is malformed in the typeset manuscript: it defines \\WER_g with an expression involving an unidentified \\bar X, a β_logRef term, and a denominator 'e \\bar X − 1', and the notation is not explained. Because raw fairness scores (Eq. 3), category scores (Eq. 6), and FAAS (Eq. 7) all depend on Eq. (2), the entire fairness ranking is not checkable from the paper as written. Directing the reader to the HuggingFace space does not replace a self-contained derivation. The central empirical claim of the submitted paper is that the reported disparities are real and the rankings are as stated, but the manuscript supplies neither a usable formula nor the supporting numerical evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ASR-FAIRBENCH, a real-time leaderboard that evaluates both accuracy (WER) and equity in speech recognition, using a 10% stratified sample of the Fair-Speech dataset. The authors propose a mixed-effects Poisson regression to estimate demographic disparities, derive a raw fairness score from predicted WERs, apply a p-value-based penalty, and combine fairness with WER into a Fairness-Adjusted ASR Score (FAAS). They claim Whisper-medium achieves the highest FAAS (29.41), that Whisper-tiny is fairer despite higher WER, and that fairness can outweigh slight WER improvements in some comparisons. The paper also describes a web platform with interactive leaderboards and visualizations.","tokens_in":3327,"tokens_out":4639,"duration_ms":45637,"significance":"The intended contribution is timely and relevant: a benchmark that explicitly accounts for demographic equity in ASR would fill a real gap, and the authors provide a live leaderboard and reproducible code. If the metric were rigorously defined and the empirical results fully reported, this could be a useful community resource. However, in its current form the central scientific claims are not verifiable: the predicted-WER equation is malformed, the fairness penalty is ad hoc, and no numerical results or statistical summaries support the reported rankings. The paper therefore currently functions more as a platform demo than as a self-contained research contribution.","major_comments":[{"comment":"The formula for predicted WER, which underpins the raw fairness score, is malformed and undefined: the symbols \\bar X, β_logRef, and the denominator e \\bar X − 1 are not introduced, and the expression is inconsistent with the regression in Eq. (1). Since Eqs. (3), (6), and (7) all depend on Eq. (2), the FAAS score cannot be computed or reproduced from the paper as written. A pointer to a HuggingFace space does not replace a self-contained derivation.","section":"Section 2.2.2, Eq. (2)"},{"comment":"The central empirical claims—Whisper-medium FAAS of 29.41, Whisper-tiny's superior fairness despite higher WER, and the Wav2Vec-large vs. Hubert-large comparison—are stated only in the conclusion, with no results table, per-model fairness scores, confidence intervals, or p-values anywhere in the manuscript. The screenshot in Figure 1 is not a substitute for reporting the underlying numbers. Without this evidence, the reader cannot assess whether the observed disparities are real or an artifact of the methodology.","section":"Section 4 and Figure 1"},{"comment":"The multiplicative penalty factor p/0.05 is invoked without any justification or sensitivity analysis. Because the p-value depends on sample size, this penalty can make the fairness score arbitrarily small for large datasets even when the group disparity is practically negligible, and it will tend to zero for any statistically significant effect as N grows. The choice of 0.05 as the threshold and the specific functional form need a principled derivation or at least a robustness check; currently the penalty is ad hoc and could dominate the ranking.","section":"Section 2.2.3, Eq. (5)"},{"comment":"The validation of the 10% stratified subsample is limited to reporting per-attribute entropy values. Matching marginal entropy for each attribute does not guarantee preservation of the joint demographic distribution, nor does it ensure sufficient observations in every cell of the mixed-effects Poisson model. The paper provides no analysis of cell sizes, sparsity, or standard errors for the disparity ratios, so the stability of the β1 estimates and the LRT p-values is unknown. If any demographic group is sparsely represented, the resulting FAAS scores could be driven by noise rather than by genuine model behavior.","section":"Section 2.1, Table 1"},{"comment":"The fairness score is computed from the fitted values of the same mixed-effects Poisson regression used to estimate demographic disparities, making the reported 'fairness' a direct transformation of the model's own predictions. This is not logically circular, but it means any misspecification of the Poisson model—such as unmodeled overdispersion, omitted covariates, or an incorrect random-effects structure—directly transfers into the fairness score. The manuscript does not report model diagnostics, goodness-of-fit measures, or a comparison of the Poisson model with alternative count or continuous-error models, so the robustness of the fairness ranking is unestablished.","section":"Section 2.2.2 and 2.2.3"}],"minor_comments":[{"comment":"The WER definition should parenthesize the numerator as (S+D+I)/N to avoid ambiguity.","section":"Section 2.2.1"},{"comment":"The introduction mentions a five-tier classification from 'severely biased' to 'exemplarily fair' for overall and attribute-specific assessments, but this scheme is never described in the methodology or used in the results. Either define the tier boundaries and report the resulting tiers, or remove the claim.","section":"Section 1"},{"comment":"The URL in reference [5] is malformed: it begins with 'urlhttps://' and contains a duplicated path ('huggingface.co/spaces/huggingface.co/spaces').","section":"Section 5, Ref. [5]"},{"comment":"The min-max scaling in the raw fairness score is not specified as to whether the minimum and maximum are taken across groups within a category, across categories, or across all models, which affects the comparability of category scores.","section":"Section 2.2.2, Eq. (3)"},{"comment":"The paper uses an inconsistent mix of notation, such as \\WER vs. WER and βlogRef vs. β1; the authors should unify the notation and define every symbol at its first appearance.","section":"General"}],"recommendation":"reject","confidential_remarks":"The paper has a useful platform contribution, but as a scientific article it is severely under-specified: the main metric is not reproducible from the equations, and the headline results are not backed by any reported data. The issues are not merely cosmetic; they require rewriting the methodology, adding a full evaluation section with statistical details, and rethinking the ad-hoc penalty. This goes beyond a major revision into the territory of a new submission. The contribution might fit better as a demo or dataset/benchmark paper at a workshop, where the platform itself is the deliverable, rather than as a full research paper in a serious journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. The FAAS leaderboard concept is genuinely useful: a single number that combines WER with a fairness penalty, backed by a live, reproducible benchmark for ASR equity. That fills a real niche. The second thing is that the headline numbers—Whisper-medium at 29.41, Whisper-tiny fairer despite higher WER—are not verifiable from the manuscript. There is no results table, no p-values, no confidence intervals, and Eq. (2), which defines predicted WER, is malformed as typeset. The stress-test note is correct: the fairness score chain depends on that equation, so the rankings are not checkable from the paper alone.\n\nWhat the paper does well: the mixed-effects Poisson regression approach to estimating demographic disparity ratios is sound and is an existing technique from earlier bias-measurement work, properly cited. The 10% stratified sampling is a pragmatic choice, and the entropy comparisons in Table 1 show per-attribute distributions are roughly preserved, even though the joint distribution is not checked. Building a public leaderboard with code and a HuggingFace space is a concrete, useful contribution that goes beyond many fairness papers.\n\nSoft spots, in order of severity. First, the missing empirical backbone: a benchmarking paper must report per-model WER, disparity ratios, p-values, and per-category scores. This one reports only final FAAS numbers in the conclusion. That's a load-bearing omission. Second, Eq. (2) is not reproducible: \\bar X is undefined, the β_logRef term appears from nowhere, and the denominator 'e \\bar X − 1' doesn't parse. Since Eq. (3), (6), and (7) all feed off it, the whole metric is ambiguous. Third, the p/0.05 penalty in Eq. (5) is arbitrary and uncalibrated; it changes scores substantially and has no justification. Fourth, the 10% sample validation ignores intersections of demographic attributes, leaving sparse cells that could destabilize the mixed-effects regression. These are not minor quibbles—they undercut the claim that the rankings are real.\n\nWho is this for? People building ASR systems for diverse users and researchers working on fairness evaluation metrics. They will appreciate the platform and the direction, but they cannot trust the specific rankings until the details are fixed. The paper deserves a serious referee because the topic is important and the platform is a real artifact, but the current draft should probably be rejected at review unless the authors supply the missing numbers, fix the equations, and justify the penalty. That's a heavy revision, not a polish.","headline":"Useful fairness leaderboard idea, but the headline rankings are not supported by the manuscript's equations or reported data.","tokens_in":3877,"tokens_out":3448,"would_cite":false,"duration_ms":31674,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces a real-time leaderboard that scores speech recognition models on demographic fairness as well as accuracy, and shows that Whisper-medium leads with a Fairness-Adjusted ASR Score of 29.41.","keywords":["ASR fairness","speech recognition","leaderboard","mixed-effects Poisson regression","Word Error Rate","demographic disparity","FAAS","Fair-Speech dataset"],"falsifier":"Re-run the full FAAS pipeline on the complete 26,471-utterance Fair-Speech corpus (or on several fresh stratified draws) and compare disparity-ratio confidence intervals and model rankings. If Whisper-medium's top FAAS of 29.41 or Whisper-tiny's fairness lead reverses, the leaderboard's reported rankings are sample artifacts rather than stable model properties.","tokens_in":2831,"feed_emoji":"🎙️","tokens_out":4428,"duration_ms":42145,"temperature":0.7,"pith_summary":"The paper introduces ASR-FAIRBENCH, a real-time leaderboard that scores automatic speech recognition systems on both accuracy and demographic equity. Its central claim is that fairness can be quantified from a small stratified sample of the Fair-Speech dataset using mixed-effects Poisson regression, then folded with Word Error Rate into a single Fairness-Adjusted ASR Score (FAAS). If this is right, benchmark rankings change: Whisper-medium tops FAAS at 29.41, Whisper-tiny is fairer despite higher WER, and fairness gaps can outweigh small accuracy gains. The platform is offered as a public audit tool for model submissions.","feed_headline":"Speech AI gets a fairness score, and Whisper-medium tops it","feed_subtitle":"ASR-FAIRBENCH folds demographic disparity into one number; Whisper-tiny is fairer but less accurate.","key_machinery":"The central object is the Fairness-Adjusted ASR Score (FAAS), built from a mixed-effects Poisson regression of word error counts with demographic attributes as fixed effects and speaker-level random effects. The regression produces disparity ratios $e^{\\beta_1}$; predicted WERs per group are min-max scaled to 0–100 and weighted by group prevalence; a likelihood-ratio test decides which disparities are penalized; and the log-ratio $\\text{FAAS} = 10 \\log_{10}(\\text{Overall Score} / \\text{WER})$ combines fairness and accuracy into one number. This lets a single score rank models by both competence and equity.","core_discovery":"On the paper's own terms: a 10% stratified sample of 26,471 Fair-Speech utterances preserves per-attribute entropy while cutting evaluation data by 90%; a mixed-effects Poisson regression on word error counts estimates disparity ratios per demographic attribute; statistically significant disparities ($p < 0.05$) penalize category scores by $p/0.05$; and the weighted overall fairness score enters $\\text{FAAS} = 10 \\log_{10}(\\text{Overall Score} / \\text{WER})$. Applied to five models, this yields a leaderboard where Whisper-medium scores 29.41, Whisper-tiny shows better overall fairness despite higher WER, and fine-tuned Wav2Vec and HuBERT models lag in fairness despite efficiency.","pith_inferences":["Marginal entropy matching does not guarantee joint-distribution preservation; the authors' sampling decision deserves a direct check by comparing full-data and sampled-data rankings.","The FAAS formulation could be ported to other speech tasks (speaker verification, speech translation) or to intersectional groups (gender × ethnicity), where disparities may be larger than any single attribute shows.","A public leaderboard with a single signed score creates an incentive to optimize for FAAS; whether that drives genuine equity gains or mere statistical tailoring is a testable outcome of adoption.","The penalty $p/0.05$ is one of many possible schedules; sensitivity analysis over penalty shapes would show whether top rankings are robust."],"forward_implications":["A model with lower WER can rank below a fairer model with slightly higher WER, so accuracy-only leaderboards can mislead purchasers and developers.","Researchers can submit any ASR model and receive an automated fairness audit within minutes, making equity measurement a routine part of model release.","The five-tier classification (from severely biased to exemplarily fair) gives attribute-specific diagnostics, showing which demographic category drives a model's penalty.","Whisper-family models currently lead FAAS; Whisper-tiny's better fairness at higher WER suggests model scale and fairness do not move together.","The $p < 0.05$ penalty schedule means only statistically supported disparities affect scores, preventing noise from moving rankings."],"supporting_citations":[{"why":"Establishes that dialect, gender, and race affect caption accuracy, the premise that demographic performance gaps exist.","marker":"[1]"},{"why":"Quantifies bias in automatic speech recognition and supplies measurement approaches that the fairness regression extends.","marker":"[2]"},{"why":"An example of an accuracy-only leaderboard that the benchmark positions itself against.","marker":"[4]"},{"why":"The open ASR leaderboard tradition focused on WER alone, the baseline ASR-FAIRBENCH adds fairness to.","marker":"[5]"},{"why":"Documents racial disparities in ASR, motivating the need for a fairness-adjusted benchmark.","marker":"[6]"},{"why":"Provides the Fair-Speech dataset whose stratified sample drives all fairness scores in the paper.","marker":"[7]"}],"fun_headline_variants":["ASR-FAIRBENCH: one score for speech AI equity","Whisper-medium leads new fairness-adjusted ASR leaderboard","Benchmarking speech AI: accuracy meets demographic fairness","Speech AI fairness scorecard: Whisper-medium wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a 10% stratified sample whose per-attribute entropy matches the full dataset also preserves the joint demographic distribution and leaves enough speakers per group for the regression and significance tests to be reliable; if not, fairness rankings could be artifacts of sparse demographic cells.","fun_headline_variants_meta":{"raw":{"variants":["ASR-FAIRBENCH: one score for speech AI equity","Whisper-medium leads new fairness-adjusted ASR leaderboard","Benchmarking speech AI: accuracy meets demographic fairness","Speech AI fairness scorecard: Whisper-medium wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000298,"raw_usage":{"total_tokens":1670,"prompt_tokens":834,"completion_tokens":836,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":767}},"tokens_in":450,"tokens_out":836,"duration_ms":8249,"temperature":1.0,"reasoning_tokens":767,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:56:58.820987+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the full FAAS pipeline on the complete 26,471-utterance Fair-Speech corpus (or on several fresh stratified draws) and compare disparity-ratio confidence intervals and model rankings. If Whisper-medium's top FAAS of 29.41 or Whisper-tiny's fairness lead reverses, the leaderboard's reported rankings are sample artifacts rather than stable model properties.","supporting_citations":[{"cited_title":"However, despite impressive strides in overall accuracy, these systems of- ten exhibit significant performance disparities across diverse demographic groups","cited_arxiv_id":null,"evidence_quote":"Establishes that dialect, gender, and race affect caption accuracy, the premise that demographic performance gaps exist."},{"cited_title":"Despite higher 1https://huggingface.co/spaces/satyamr196/ASR-FairBench 2https://github.com/SatyamR196/ASR-FairBench (a) ASR-F AIRBENCHleaderboard (b) Summarized results view","cited_arxiv_id":null,"evidence_quote":"An example of an accuracy-only leaderboard that the benchmark positions itself against."},{"cited_title":"Effects of talker dialect, gender & race on accuracy of bing speech and youtube automatic captions","cited_arxiv_id":null,"evidence_quote":"The open ASR leaderboard tradition focused on WER alone, the baseline ASR-FAIRBENCH adds fairness to."},{"cited_title":"A deep dive into the disparity of word error rates across thousands of nptel mooc videos,","cited_arxiv_id":null,"evidence_quote":"Provides the Fair-Speech dataset whose stratified sample drives all fairness scores in the paper."}],"review_version":1}