{"id":"c1af9ab1-c06a-43ac-af37-cf3dc6eb0881","arxiv_id":"1908.08717","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Women are under-represented in French broadcast corpora, and a French ASR system shows higher word error rates for women, but mainly for occasional, non-anchor speakers in spontaneous speech.","lead":"This paper measures how often women appear and speak in four major French TV and radio datasets used to train speech recognition systems, and tests whether the imbalance hurts recognition accuracy for women. It finds women are under-represented and that the accuracy gap appears mainly for occasional speakers and spontaneous speech.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline female/male WER gap may be a show-composition artifact; the causal claim is not identifiable from the aggregated observational comparison.","rationale":"The Pith Reader correctly identifies non-independence of speaker-episode WER values as a statistical flaw. That concern is real but mostly affects p-value precision; even with clustering, the large effect size might remain significant. The more load-bearing issue is bias rather than precision: the aggregated gender comparison is not adjusted for show, speech type, or accent, and the paper itself acknowledges inconsistent per-show gender trends in Section 4.2.1. If the overall gap is a composition effect, the central empirical assertion as summarized by the reader is misleading, not merely over-confident. The reader's rationale also notes the unsupported causal wording, which aligns with this concern. I therefore keep the CONDITIONAL verdict: the descriptive corpus analysis is useful, but the causal claim and the headline gender-gap interpretation require either a show-stratified re-analysis or a controlled training-data experiment before they can be accepted at face value.","tokens_in":10190,"tokens_out":9701,"duration_ms":109380,"concrete_test":"Fit a mixed-effects model to the existing speaker-episode WER values with fixed effects for gender, role, and speech type, and random intercepts for show and speaker; then test the gender coefficient. If the estimated gender effect collapses or becomes non-significant after conditioning on show, the aggregate 42.9% vs 34.3% gap is a show-composition artifact and the paper's 'driven by Punctual/spontaneous' claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the training-data gender imbalance causes lower ASR accuracy for women. The evidence is a single ASR system evaluated on 12 broadcast shows, with WER aggregated over speaker-episodes. Section 4.2.1 reports that within-show gender differences are inconsistent (some shows favor women), which is exactly the signature of a composition effect: women may be over-represented in the harder spontaneous shows, so the pooled 42.9%-vs-34.3% gap need not reflect any systematic female-speech disadvantage. The subsequent analyses by role and speech type (Sections 4.2.3 and 4.2.4) still pool across shows, so they do not remove this confound. Section 5 also lists a relevant robustness check, decoding without speaker-adapted features, as a future step rather than a reported result. Without either a show-adjusted analysis or a gender-balanced training control, the abstract's 'causes performance to decrease' wording is not identified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies gender representation in four widely used French broadcast corpora (ESTER1, ESTER2, ETAPE, REPERE) and examines whether the gender imbalance in the training data is associated with the performance of an ASR system. The authors first report that women account for 33.16% of speakers and 22.57% of speech time in the training data, and that this under-representation is stronger among Anchor speakers than among Punctual speakers. They then evaluate a Kaldi HMM-DNN ASR system trained on roughly 100 hours of prepared speech, computing episode-level WER on a 70-hour evaluation set stratified by gender, speaker role, and speech type. The headline results are a pooled female/male WER of 42.9% vs 34.3% (p<0.001), a significant gap among Punctual speakers (49.04% vs 38.56%), a significant gap for spontaneous speech (61.29% vs 46.51%), and no significant gap among Anchors or for prepared speech. The abstract and conclusion interpret these results causally, stating that the disparity in available data causes performance to decrease on women.","tokens_in":10388,"tokens_out":4829,"duration_ms":48818,"significance":"If the empirical findings hold, this is a useful contribution to the growing literature on fairness in speech and language technology. The descriptive sex-ratio results in Tables 3-5 are straightforward and credible, and they corroborate external reports on women's under-representation in French media. The introduction of the Anchor/Punctual role dichotomy is a simple and reproducible analytic choice that adds a genuinely useful stratification for interpreting ASR performance differences. The paper also makes an honest attempt to discuss speaker adaptation as a potential mechanism. However, the inferential and causal claims are currently stronger than the evidence supports: the statistical tests ignore within-speaker correlation, the pooled gender comparison is not adjusted for show composition despite evidence of between-show heterogeneity in Figure 1, and the proposed causal mechanism is left for future work. With clustered or mixed-effects analysis, show-adjusted estimates, and appropriately hedged causal language, the paper would be a solid empirical contribution.","major_comments":[{"comment":"The Wilcoxon rank-sum tests in Section 4.2 treat each episode-level WER value as independent, but Section 3.3.2 states that a given speaker can contribute several WER values, one per show occurrence. Under the null, WERs from the same speaker are likely correlated, which makes the reported p-values (e.g., p<0.001 for the pooled female/male comparison) anti-conservative. Because the repeated-speaker structure is especially pronounced for Anchor speakers and frequent Punctual speakers, this could change the role-gender conclusions. Please add a cluster-robust test, a permutation test that resamples speakers, or a mixed-effects model with speaker (and show) as random effects, and report how the significance of the key comparisons changes.","section":"Section 3.3.2 and Section 4.2.1"},{"comment":"The paper itself reports that within-show gender differences are inconsistent across shows, with some shows favoring women. This is exactly the signature of a show-composition effect: women may be over-represented in harder spontaneous shows, so the pooled 42.9%-vs-34.3% gap need not reflect any systematic female-speech disadvantage. The subsequent analyses by role (Section 4.2.3) and speech type (Section 4.2.4) still pool across shows, so they do not remove this confound. Please provide show-adjusted estimates, for example by including show as a fixed or random effect or by computing within-show gender contrasts, and state whether the gender effect survives such adjustment.","section":"Section 4.2.1 and Figure 1"},{"comment":"The causal interpretation in the abstract and conclusion ('causes performance to decrease') is not supported by the experimental design. Section 5 proposes that less fMLLR adaptation data for women causes the larger gap for Punctual speakers, but it explicitly defers the confirming experiment ('A way to confirm our hypothesis would be to reproduce our analysis on WER values obtained without using speaker adapted features'), and no gender-balanced training control is reported. As it stands, the study demonstrates associations, not causation. Please either report the decoding-without-adaptation result and/or a gender-balanced training condition, or revise the abstract, Section 5, and conclusion to use association language such as 'is associated with' or 'is correlated with'.","section":"Section 5"}],"minor_comments":[{"comment":"The abstract's phrase 'causes performance to decrease on women' and the conclusion's wording 'produces gender bias performance' overstate what the observational comparison can establish; even if the statistical issues are resolved, the causal claim should be softened unless a controlled training experiment is reported.","section":"Abstract and Conclusion"},{"comment":"The final sentence of Section 2.2 ends with a bare citation '[13]' after 'an ASR system trained on these data'; this citation seems intended to point to the system described in Section 3.3.1, but as written it is not integrated into a grammatical sentence and should be moved or rephrased.","section":"Section 2.2"},{"comment":"For prepared speech, the text reports p=0.005 immediately after saying 'WER scores are similar between men and women.' Since Section 3.3.2 fixes alpha=0.001, this p-value is not significant at the stated threshold, and the sentence should explicitly say that the difference is not significant at alpha=0.001 to avoid confusion.","section":"Section 4.2.4"},{"comment":"The conclusion states a 'WER increase of 24% for women compared to men,' but the reported pooled values 42.9% and 34.3% correspond to a relative increase of about 25.1% (and 8.6 percentage points absolute); please reconcile the numbers.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper relies on the authors' own ASR system from [13] for all performance measurements; this is not circular because the system outputs are inputs to the analysis, but the generalization of the causal claim should be explicitly limited to this single system architecture and training recipe. The descriptive contribution is solid and likely of interest to the speech-and-language fairness community, but the inferential and causal framing needs substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the Anchor/Punctual speaker role taxonomy and the finding that the WER gender gap disappears for Anchors and for prepared speech. That refines the fairness discussion in ASR: bias is not uniform across speakers, and the role/speech-type interaction is worth knowing. The descriptive statistics on gender representation in ESTER/ETAPE/REPER are credible, and the paper deserves credit for laying out the data and the evaluation design clearly. The 1% threshold for roles is transparent and analytic, not tuned to outcomes, so the circularity burden is low.\n\nThe soft spots are real but not fatal to the descriptive contribution. The Wilcoxon tests in Section 4.2 treat every episode-level WER as independent, even though Section 3.3.2 says a speaker can appear in multiple shows. That likely makes the p-values anti-conservative, and the more serious issue is that the pooled gender comparison (42.9% vs 34.3%) may be a show-composition artifact. The paper itself notes in Figure 1 that within-show gender differences are inconsistent, some favoring women. That is exactly what you would expect if women are over-represented in harder spontaneous shows, not necessarily because the ASR system is intrinsically worse on female voices. The role and speech-type splits still pool across shows, so they do not remove the confound. The causal wording in the abstract and conclusion ('causes performance to decrease on women') is therefore overclaimed.\n\nThat said, the authors are honest about the limitation; Section 5 explicitly lists decoding without speaker-adapted features as future work rather than claiming they have shown it. So the paper is not misleading on close reading, it is just earlier than its language suggests.\n\nFor its intended audience, this is a useful empirical contribution to the fairness-in-ASR discussion. The role taxonomy could be reused, and the descriptive statistics are a solid reference point. I would not cite the causal claim uncritically, but I would cite the role-aware WER results. The statistical weaknesses are fixable with cluster-robust or mixed-effects models and a show-adjusted analysis, and the paper would be stronger for it.\n\nRecommendation: this deserves a serious referee, but the referee should push for softened causal language and cluster-aware statistics. Send it to review.","headline":"Useful role-based descriptive study of gender imbalance in French broadcast corpora, but the causal claim about ASR performance is not supported by the observational analysis.","tokens_in":741,"tokens_out":902,"would_cite":true,"duration_ms":20005,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A French broadcast-trained ASR system produces significantly higher word error rates for women than for men, with the gap driven by occasional speakers and spontaneous speech.","keywords":["gender bias","automatic speech recognition","French broadcast corpora","word error rate","speaker role","speech type","Kaldi","broadcast news"],"falsifier":"Decode the same evaluation set without the speaker-adaptation step and compare the gender WER gaps: the paper's explanation predicts the Punctual-speaker gap should shrink or disappear when adaptation data is no longer an advantage. Alternatively, retrain the acoustic model on a gender-balanced subset of the same shows; if the women-versus-men gap among Punctual speakers persists at a similar size, gender imbalance in training data is not the main driver.","tokens_in":10019,"feed_emoji":"🎙️","tokens_out":6002,"duration_ms":53706,"temperature":0.7,"pith_summary":"The paper argues that four widely used French broadcast corpora underrepresent women in both speaker count and speech time, and that an ASR system trained on them performs measurably worse for women than for men. On a 70-hour evaluation set, women receive an average word error rate of 42.9% versus 34.3% for men, a difference reported as statistically significant at $p<0.001$. The gap is not uniform: it concentrates among Punctual (non-anchor) speakers and in spontaneous speech, while Anchors and prepared speech show no significant gender difference. The stakes are practical: these corpora remain common training material for French ASR, so a bias inside them lands directly in deployed systems.","feed_headline":"French ASR trained on broadcast data errs more on women's speech","feed_subtitle":"Women's word error rate is 42.9% versus 34.3% for men, with the gap among occasional speakers and spontaneous talk.","key_machinery":"The key analytical device is the speaker-role categorization: a speaker is an Anchor when he or she exceeds 1% of a show's speech turns and 1% of its total speech time, and Punctual when below both thresholds. Crossing this role with gender isolates who carries the performance gap. The paper also uses Wilcoxon rank-sum tests on episode-level WER values, and explains the pattern through speaker adaptation in the decoding pipeline: fMLLR-adapted features give speakers with more data a stronger adaptation, so Anchors recover from acoustic mismatch while Punctual speakers, who are disproportionately women, do not.","core_discovery":"The central empirical claim is that an ASR system trained on roughly 100 hours of gender-unbalanced French broadcast speech exhibits a gender bias in word error rate. Averaged over the evaluation set, women show 42.9% WER against 34.3% for men, with medians of 29% and 25% respectively. The paper then shows this overall gap is carried by specific subpopulations: among Punctual speakers, average WER is 49.04% for women and 38.56% for men; among Anchors, the difference is not significant ($p=0.173$). Similarly, on spontaneous speech women average 61.29% WER versus 46.51% for men, while on prepared speech the gap is not significant. The authors conclude that gender imbalance in training data translates into biased ASR performance, but that speaker role and speech spontaneity are entangled factors that determine where the bias appears.","pith_inferences":["As an extension of the paper's logic, the show-level role definition means the same person could be Anchor in one show and Punctual in another; testing whether stable voice characteristics or show-level factors drive the gap would refine the causal story.","The paper's own proposed confirmation is to decode without speaker adaptation; if the Punctual-speaker gender gap persists under that condition, the explanation would shift from adaptation-data quantity to the acoustic model itself.","The same role-based methodology could be applied to other underrepresented groups in broadcast data, such as dialectal, non-native, or older speakers, where unequal show presence may create analogous WER gaps.","A practical extension is to stratify benchmark reporting by speaker role and gender, so future French ASR work would routinely expose rather than average away such performance disparities."],"forward_implications":["If the claim holds, ASR systems trained on existing French broadcast corpora carry a measurable performance penalty for women's speech, so evaluations reporting only aggregate WER will hide a systematic error skew.","The Anchor versus Punctual distinction predicts that the gender gap will appear mainly in data with many non-professional, occasional speakers rather than in news-reader-style speech.","The results imply that rebalancing training data or ensuring sufficient adaptation data for female speakers are concrete remedies for reducing gender bias in French ASR.","Because spontaneous speech amplifies the gap, the bias may grow worse as ASR expands toward conversational and entertainment content without rebalanced training data."],"supporting_citations":[{"why":"Describes the Kaldi HMM-DNN ASR system whose performance is evaluated in this paper.","marker":"[13]"},{"why":"One of the four French broadcast corpora used as training and evaluation data.","marker":"[14]"},{"why":"Second ESTER corpus, another source of the training and evaluation data.","marker":"[15]"},{"why":"ETAPE corpus supplies the spontaneous and debate-style speech that grounds the speech-type analysis.","marker":"[20]"},{"why":"REPERE corpus is one of the four French broadcast corpora used in the study.","marker":"[17]"},{"why":"Earlier work on whether speech recognizers prefer female speakers, which this paper contrasts and extends.","marker":"[2]"},{"why":"The Kaldi speech recognition toolkit used to build the ASR system.","marker":"[25]"},{"why":"MLLR speaker adaptation provides the mechanism the paper invokes to explain why Anchors show no gender gap.","marker":"[21]"},{"why":"Tatman's YouTube captioning bias results that this paper's findings corroborate on a larger French data set.","marker":"[29]"},{"why":"The Wilcoxon rank-sum test, the statistical method used for all gender and role comparisons.","marker":"[23]"}],"fun_headline_variants":["French ASR: women's word error 42.9% vs men's 34.3%","Women's speech trips up French broadcast ASR","Gender gap in French ASR: 42.9% vs 34.3% WER","ASR trained on French broadcast skews against women","Under-represented women in French broadcast data skew ASR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The significance tests treat each episode-level word error rate as an independent observation even though the same speaker can appear in several episodes; if those values are correlated within a speaker, the reported $p$-values could be too small.","fun_headline_variants_meta":{"raw":{"variants":["French ASR: women's word error 42.9% vs men's 34.3%","Women's speech trips up French broadcast ASR","Gender gap in French ASR: 42.9% vs 34.3% WER","ASR trained on French broadcast skews against women","Under-represented women in French broadcast data skew ASR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001116,"raw_usage":{"total_tokens":4616,"prompt_tokens":887,"completion_tokens":3729,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":3632}},"tokens_in":503,"tokens_out":3729,"duration_ms":24807,"temperature":1.0,"reasoning_tokens":3632,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:30:40.159744+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Decode the same evaluation set without the speaker-adaptation step and compare the gender WER gaps: the paper's explanation predicts the Punctual-speaker gap should shrink or disappear when adaptation data is no longer an advantage. Alternatively, retrain the acoustic model on a gender-balanced subset of the same shows; if the women-versus-men gap among Punctual speakers persists at a similar size, gender imbalance in training data is not the main driver.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the Kaldi HMM-DNN ASR system whose performance is evaluated in this paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the four French broadcast corpora used as training and evaluation data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Second ESTER corpus, another source of the training and evaluation data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ETAPE corpus supplies the spontaneous and debate-style speech that grounds the speech-type analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"REPERE corpus is one of the four French broadcast corpora used in the study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier work on whether speech recognizers prefer female speakers, which this paper contrasts and extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Kaldi speech recognition toolkit used to build the ASR system."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MLLR speaker adaptation provides the mechanism the paper invokes to explain why Anchors show no gender gap."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Tatman's YouTube captioning bias results that this paper's findings corroborate on a larger French data set."},{"cited_title":"Mann and Donald R","cited_arxiv_id":null,"evidence_quote":"The Wilcoxon rank-sum test, the statistical method used for all gender and role comparisons."}],"review_version":1}