{"id":"654b5bb9-c451-4764-9942-878089a61a83","arxiv_id":"2506.22372","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM-based three-class gender labeling agrees with human annotations better than the lexical NFaiRR score, and the proposed CWEx metric combines neutral exposure with male-female exposure disparity for ranking fairness evaluation.","lead":"This paper tests whether large language models can detect gender bias in search result passages better than existing word-count methods, and it introduces a new fairness metric plus a hand-annotated dataset of 893 passages from MS MARCO. The authors report that LLM labels agree with human judgments more than the standard NFaiRR metric, but the new metric's advantage over existing ones is not directly validated.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CWEx's ranking-fairness claim rests on an unvalidated construct: document-level labels may encode genderedness rather than bias, and no ranking-level ground truth supports the fairness interpretation.","rationale":"The reader's weakest assumption is that document-level three-class labels are sufficient to evaluate ranking fairness, which is also the gap I identify. I go one step further: the paper's own annotation guideline provides direct evidence that the labels may encode 'explicitly talking about a specific gender' rather than bias, making the construct problem concrete rather than merely a missing validation. I did not focus on the reported '58.77% improvement' being an absolute kappa difference, or on the potential few-shot leakage from Grep-BiasIR examples, because those affect the magnitude of the claimed improvement rather than the validity of CWEx as a fairness metric. The released MSMGenderBias dataset and the demonstration that LLM labels agree with human labels better than NFaiRR's neutrality score are genuine contributions. The missing piece is a ranking-level validity check, which is addressable with a focused human study on paired rankings. If CWEx aligns with human fairness judgments, the metric's central claim is supported; if not, CWEx should be reframed as a descriptive exposure metric rather than a fairness metric. This is why the conditional verdict stands unchanged.","tokens_in":17939,"tokens_out":7904,"duration_ms":94084,"concrete_test":"Build a set of paired rankings from the released MSMGenderBias queries. For each query, create one ranking with relevant gender-specific documents (e.g., female-targeted health content for a female-oriented query) and one with equally relevant neutral documents, plus a relevance-controlled pair that holds exposure disparity fixed while swapping which gender is favored. Have fresh annotators judge which ranking is fairer using a definition independent of the paper's labels (e.g., relevance-weighted group representation). Compute CWEx for each ranking and measure rank correlation (Kendall's tau) with human fairness judgments. If CWEx disagrees with human judgments on gender-specific queries or on relevance-controlled pairs, the metric is not a valid fairness measure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CWEx 'effectively distinguishes gender bias in ranking' depends on an unvalidated construct. CWEx (Eq. 1) aggregates document-level LLM/human labels into group exposures and treats high neutral exposure plus low male-female exposure disparity as fair. The only validation reported is at the document level (Tables 3-5): LLM labels and NFaiRR's neutrality score are compared with human document labels. No experiment compares CWEx scores with human judgments of ranking fairness, or with rankings whose fairness properties are known. In addition, the annotation guideline in Figure 2 defines Male/Female as documents that 'explicitly talk about a person with a specific gender' or use gender terms 'more than the other gender'; by that rule, an informational biography that lists Alan Turing and Grace Hopper is labeled Male. This suggests the labels may track genderedness rather than bias, so the exposures entering CWEx may not correspond to unfairness at all. Without a ranking-level test, the claim that CWEx measures fairness, rather than the gender skew of retrieved content, is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using LLMs as document-level gender-bias detectors with three-class labels (neutral, male-biased, female-biased), introduces a new fairness metric called Class-wise Weighted Exposure (CWEx) that aggregates these labels into a ranked-list fairness score, and releases a new gender-bias annotation subset of MS MARCO called MSMGenderBias. The authors compare LLM labels with human labels on Grep-BiasIR and MSMGenderBias, reporting higher Cohen's kappa agreement than the NFaiRR neutrality score, and they compare ranking models with CWEx and NFaiRR. The core claims are that LLM-based detection is more accurate than lexical baselines and that CWEx provides a more detailed and human-aligned evaluation of fairness in ranked lists.","tokens_in":18128,"tokens_out":3855,"duration_ms":42932,"significance":"If established, the paper would make three useful contributions: a semantic, LLM-based alternative to lexical gender-bias detection for ranking evaluation; a new annotated dataset (MSMGenderBias) for the community; and a class-aware exposure metric that explicitly accounts for neutral, male, and female document categories. The document-level detection experiments are informative and the dataset release is a concrete asset. However, the paper's central ranking-level claim is not yet supported: CWEx is never validated against human judgments of ranking fairness or against rankings with known fairness properties. In addition, the reported kappa improvements are stated as percentages when they are absolute percentage-point differences, and the human annotation design is explicitly aligned with the LLM prompt, which weakens the independence of the human ground truth. These issues are load-bearing for the paper's headline claims, though they appear addressable with additional analysis and new experiments.","major_comments":[{"comment":"The claim of \"58.77% improvement\" on Grep-BiasIR and \"18.51% improvement\" on MSMGenderBias is stated as a percentage improvement, but the numbers are absolute percentage-point differences in Cohen's kappa. For Grep-BiasIR, the best LLM kappa is 0.8580 versus NFaiRR's 0.2703, a difference of 0.5877, which is a relative improvement of about 217%, not 58.77%. For MSMGenderBias, 0.5719 versus 0.3868 is a difference of 0.1851, a relative improvement of about 47.9%, not 18.51%. The abstract and Section 5.1 should report the kappa values directly and, if percentages are used, label them as absolute percentage-point differences or relative improvements consistently.","section":"Abstract and §5.1, Table 5"},{"comment":"The central claim that CWEx \"effectively distinguishes gender bias in ranking\" is not validated at the ranking level. Eq. (1) defines CWEx as a weighted combination of neutral exposure and male-female exposure disparity, but the experiments in Tables 7 and 8 only compare CWEx values across ranking models and against NFaiRR; there is no human judgment of ranking fairness, no simulated ranking with known fairness properties, and no external fairness benchmark. Without such a ranking-level test, the metric's interpretation as a fairness measure, rather than as a measure of gender skew in retrieved content, is unsupported. I recommend adding a controlled experiment with synthetic rankings whose fairness properties are known, or a human study that directly evaluates full ranked lists.","section":"§5.3, Tables 7 and 8; Eq. (1)"},{"comment":"The document-level labels appear to conflate genderedness with bias. The annotation guidelines in Figure 2 define Male/Female as documents that \"explicitly talk about a person with a specific gender\" or that include gender-related terms \"more than the other gender,\" and they label the example containing Alan Turing, Grace Hopper, and other computer scientists as Male despite the presence of a female scientist. This suggests the labels capture the dominant gender of the document's subject matter, not whether the document is biased. Because CWEx (Eq. 1) is built on these labels, a high CWEx score may simply indicate low gender skew in content, which is not the same as absence of bias. The authors should either align the label definition with a bias construct and provide evidence that the labels reflect bias, or explicitly reframe the metric as measuring representation skew.","section":"Figure 2 and §3.3"},{"comment":"The human annotation instructions were deliberately designed to align with the LLM prompt (Section 3.3: instructions \"designed to align with the instructions provided to the LLM\"), and the few-shot examples used in the prompts were selected from Grep-BiasIR, the same dataset used for evaluation (Section 5.1). This creates a risk of circularity: the high kappa agreement between LLM labels and human labels may be partly by construction, because humans were guided to apply the same criteria as the LLM. The paper should report the agreement using independently collected or pre-existing human labels, or at least quantify how much the aligned instructions inflate agreement. This issue directly affects the headline claim that LLM labels are better aligned with human judgment than NFaiRR's neutrality score.","section":"§3.3 and §5.1, Table 1"}],"minor_comments":[{"comment":"There are minor typographical errors in the author affiliations: \"Unviersity of Amsterdam\" and \"The Netherland\" should be corrected.","section":"Title page / author affiliations"},{"comment":"The sentence \"The proposed CWEx metric is constrained within the range of alpha to alpha−1\" is imprecise; the correct interval is [alpha−1, alpha]. Please rephrase for clarity.","section":"§3.2, Eq. (1)"},{"comment":"The paper selects the best prompt setting per model and compares that single best value to NFaiRR without reporting variance or significance tests. Because Cohen's kappa is computed on a finite sample, confidence intervals or a significance test would strengthen the comparison, especially for the smaller MSMGenderBias subset.","section":"Table 5"},{"comment":"The human evaluation section reports that \"the final classification of gender bias... was determined based on the majority vote among the annotators\" but also says that \"for the documents that each annotator selected one of the classes, we asked an expert annotator to annotate that document.\" This sentence is unclear: it is not specified when the expert annotation is used versus the majority vote, and the phrase \"each annotator selected one of the classes\" is ambiguous. Please clarify the adjudication procedure.","section":"§3.3"},{"comment":"The description of query sampling states that 20 queries were randomly selected from QS1 and QS2, and that BM25, BERT, MiniLM, and TinyBERT retrieved 446 and 447 passages respectively. It would be helpful to report how many unique queries and documents resulted after deduplication across retrieval models, since the statistics in Table 2 suggest some overlap.","section":"§4, Dataset"}],"recommendation":"major_revision","confidential_remarks":"The dataset release and the LLM-based detection experiments are solid contributions, and the paper is within scope for ICTIR. My main concern is that the headline ranking-fairness claim rests on an unvalidated metric and on labels that may measure genderedness rather than bias. The kappa percentage misreporting is a straightforward fix, but the ranking-level validation gap is substantive. I believe the authors can address it within a revision by adding a controlled ranking-level evaluation and by reframing the claims to match the evidence. I recommend major revision rather than rejection, because the central idea is promising and the deficiencies are addressable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful part of this paper is the dataset and the empirical comparison. MSMGenderBias, 893 MS MARCO passages with three-class gender labels, is a genuine addition to the IR fairness toolbox, and the finding that GPT-4o and Llama-3.1-8B-Instruct agree with human labels better than NFaiRR's lexical neutrality score (kappa 0.858 vs 0.270 on Grep-BiasIR) is worth having. The paper is transparent about prompts, annotation setup, and limitations, and it releases the data. No circularity burden in the metric itself - CWEx is a definition, not a fitted quantity.\n\nThe soft spots are real but not fatal. The biggest one is construct validity. The annotation guideline in Figure 2 labels any document that explicitly talks about a person of a specific gender as Male or Female. By that rule, the authors' own example 'Alan Turing, John von Neumann, Linus Torvalds, Geoffrey Hinton, Grace Hopper, and Mark Zuckerberg' is labeled Male. That is a list of famous computer scientists, not a biased document. So the labels are better described as genderedness than as bias. CWEx then aggregates these labels into group exposures and calls a ranking unfair if it skews male. Without any ranking-level ground truth - no human judgments of ranking fairness, no known-fair rankings - the claim that CWEx measures fairness is unsupported. The metric may just be measuring the gender skew of retrieved content.\n\nThe second issue is reporting. The abstract states 'improved alignment to human labels (58.77% for Grep-BiasIR, and 18.51% for MSMGenderBias)'. Those are absolute differences in kappa (0.8580 - 0.2703 = 0.5877), not relative improvements. The wording overstates the gain. A relative improvement would be 217%. This is easy to fix but should be fixed.\n\nMinor: the few-shot examples are drawn from Grep-BiasIR and then the same dataset is used for evaluation, a mild leakage risk. The human annotation instructions were also deliberately aligned with the LLM prompt, which makes the high agreement less surprising, though the comparison with NFaiRR is still fair.\n\nWho should read this: IR fairness researchers and anyone using LLMs as annotators. The dataset and the empirical LLM comparison are worth engaging with. The CWEx metric, as presented, is not yet a validated fairness measure. I would send this to peer review - it deserves a serious referee - but the authors should be asked to add a ranking-level validation of CWEx and correct the improvement percentages.","headline":"New gender-bias dataset and a useful LLM-vs-lexical comparison, but CWEx's fairness claim rests on labels that track genderedness, not bias - and the reported improvements are absolute kappa differences.","tokens_in":18709,"tokens_out":3353,"would_cite":true,"duration_ms":32114,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM gender labels beat word-count bias metric by up to 59%","keywords":["gender bias","fairness metrics","ranking fairness","large language models","bias detection","information retrieval","MSMGenderBias","exposure-based fairness"],"falsifier":"A user study would settle it: ask people to rate the fairness of many generated rankings and compare their ratings to CWEx. If rankings with high CWEx are not perceived as fairer than low-CWEx ones, or if for a query where no neutral relevant documents exist CWEx marks every ranking as unfair, the document-label-to-ranking link fails.","tokens_in":17698,"feed_emoji":"⚖️","tokens_out":4882,"duration_ms":48194,"temperature":0.7,"pith_summary":"This paper argues that large language models can detect gender bias in individual passages more reliably than the word-count methods underlying existing fairness metrics, and that the resulting three-class labels (neutral, male, female) can be turned into a ranking-level fairness metric. The new metric, Class-wise Weighted Exposure (CWEx), rewards rankings that put neutral documents high and penalizes large gaps between the exposure of male- and female-biased documents. On two datasets, LLM labels agreed with human annotators substantially better than the NFaiRR neutrality score, with Cohen's kappa improving by 58.77% on Grep-BiasIR and 18.51% on the new MSMGenderBias collection. If correct, this gives information retrieval practitioners a semantic, tunable way to measure and compare the gender fairness of ranked lists, plus a public benchmark for future work.","feed_headline":"LLM gender labels beat word-count bias metric by 59%","feed_subtitle":"A three-class exposure metric, CWEx, aligns with human judgments far more closely than NFaiRR's neutrality score.","key_machinery":"The central object is the Class-wise Weighted Exposure (CWEx) metric, defined as $\\mathrm{CWEx} = \\alpha \\cdot \\mathrm{Exposure}_{\\text{neutral}} - (1-\\alpha) \\cdot |\\mathrm{Exposure}_{\\text{male}} - \\mathrm{Exposure}_{\\text{female}}|$, where $\\mathrm{Exposure}_G$ is the summed position-bias weight $p(i) = 1/\\log_2(1+i)$ for documents of class $G$, normalized by the maximum possible exposure, and $\\alpha \\in [0,1]$ balances promoting neutral documents against reducing male-female disparity. The classification engine is few-shot prompting of LLMs with instructions covering gender-term frequency, balance of information, and lead representation. This machinery converts document-level bias categories into a single interpretable fairness score for a ranked list, and the $\\alpha$ parameter makes the trade-off explicit and domain-adjustable.","core_discovery":"The central claim is that gender bias in ranked lists is better measured by semantic document-level labels than by lexical term counts. The authors prompt GPT-4o, Llama-3.1-8B-Instruct, Llama-3.1-8B, Mixtral-8x7B-Instruct, and Qwen2.5-7B-Instruct to classify passages as male-biased, female-biased, or neutral, and show that these labels align with human judgments more closely than the neutrality score of NFaiRR: best Cohen's kappa is 0.858 versus 0.270 on Grep-BiasIR, and 0.572 versus 0.387 on MSMGenderBias. Building on these labels, they define CWEx, a metric that combines normalized neutral exposure with the male-female exposure gap, weighted by a parameter alpha that lets the evaluator trade off neutrality against gender parity. The paper also releases MSMGenderBias, a set of 893 MS MARCO passages labeled by crowdworkers.","pith_inferences":["The metric's premise that neutral exposure is inherently good may clash with domains where neutral documents are not relevant; a testable extension would weight neutral exposure by relevance before judging fairness.","Because LLMs themselves carry stereotype biases (the paper finds asymmetry between male- and female-stereotype detection), a single chosen annotator model could inject its own bias into the fairness score; one could ensemble several LLMs and measure disagreement.","The study's human evaluation covers only top-10 results from 20 queries per query set; scaling to full rankings would check whether CWEx remains stable and meaningful.","The three-class scheme treats 'mentions a gender' and 'biased toward a gender' as the same signal; a finer label set distinguishing representation from bias could change which rankings count as fair."],"forward_implications":["Fairness evaluation for ranking can shift from lexical term counting to semantic understanding, catching biases like 'his mother' and 'her son' that current word-based metrics miss.","CWEx gives practitioners a dial, alpha, to choose between prioritizing neutral language in top ranks and balancing exposure between male- and female-biased documents.","The same LLM-labeling pipeline extends to other protected attributes such as race, age, and ethnicity, and to non-binary gender groups by replacing the male-female disparity term with the max-min group disparity.","The released MSMGenderBias dataset provides a public benchmark for comparing future gender-bias detectors and fairness metrics.","LLMs can produce human-aligned bias labels at scale, reducing reliance on expensive manual annotation for building bias datasets."],"supporting_citations":[{"why":"Supplies Grep-BiasIR, the annotated IR bias dataset used for evaluation and as the source of prompt examples.","marker":"[23]"},{"why":"Defines NFaiRR and its neutrality score, the lexical baseline that CWEx is compared against.","marker":"[40]"},{"why":"MS MARCO, the passage collection from which the new MSMGenderBias annotations are drawn.","marker":"[35]"},{"why":"Provides the ARB metric and the QS1 query set used for sub-sampling documents for human evaluation.","marker":"[41]"},{"why":"Contributes the position-bias formulation p(i) that CWEx adopts for exposure weighting.","marker":"[2]"},{"why":"Pyserini toolkit used to run the BM25 retrieval baseline in the ranking experiments.","marker":"[29]"},{"why":"Sentence-Transformers library providing the cross-encoder re-rankers (BERT, MiniLM, TinyBERT).","marker":"[39]"},{"why":"Inspiration for the few-shot prompt design for using LLMs as annotators in IR evaluation.","marker":"[1]"}],"fun_headline_variants":["LLM labels boost gender-bias kappa from 0.27 to 0.86","CWEx: LLM-based metric aligns with human bias judgments","New fairness metric CWEx uses LLM labels to beat NFaiRR","LLM-driven metric outperforms lexical counters in gender bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole measure rests on the assumption that a document's three-way gender label, neutral, male, or female, captures what is unfair about a ranked list, so that a ranking is fair precisely when neutral documents get high exposure and male- and female-labeled documents get equal exposure.","fun_headline_variants_meta":{"raw":{"variants":["LLM labels boost gender-bias kappa from 0.27 to 0.86","CWEx: LLM-based metric aligns with human bias judgments","New fairness metric CWEx uses LLM labels to beat NFaiRR","LLM-driven metric outperforms lexical counters in gender bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001437,"raw_usage":{"total_tokens":5834,"prompt_tokens":1026,"completion_tokens":4808,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":4729}},"tokens_in":642,"tokens_out":4808,"duration_ms":40927,"temperature":1.0,"reasoning_tokens":4729,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:05:59.191916+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A user study would settle it: ask people to rate the fairness of many generated rankings and compare their ratings to CWEx. If rankings with high CWEx are not perceived as fairer than low-CWEx ones, or if for a query where no neutral relevant documents exist CWEx marks every ranking as unfair, the document-label-to-ranking link fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Grep-BiasIR, the annotated IR bias dataset used for evaluation and as the source of prompt examples."},{"cited_title":"Measuring Bias in a Ranked List using Term-based Representations","cited_arxiv_id":"2403.05975","evidence_quote":"Contributes the position-bias formulation p(i) that CWEx adopts for exposure weighting."}],"review_version":1}