{"id":"7dcc76d7-4c0f-48ea-8daa-4d92a40db068","arxiv_id":"2507.08866","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Label bias and proxy features drive algorithmic unfairness more than underrepresentation of protected groups in training data, and a new Data Bias Profile quantifies these risks.","lead":"The paper tests three well-known ways data can be biased: some groups get too few training examples, their labels are systematically wrong, and other features secretly encode their identity. It finds that wrong labels, especially when combined with such hidden proxies, cause more unfair model behavior than simply having too few examples of a group, and proposes a quantitative 'Data Bias Profile' to document these risks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Underrepresentation conclusion is based solely on uniform random subsampling; structured missing-feature-subspace underrepresentation is untested and may undermine the 'overemphasized' claim.","rationale":"Most load-bearing: the abstract and Section 4.3 make a strong, policy-relevant claim that underrepresentation is less important than commonly believed. This claim is the paper's primary novelty and the basis for recommendations in Table 7. The inference is valid only for the uniform-subsampling operationalization; the paper provides no evidence for structured underrepresentation, which is arguably the more common real-world failure mode (selection bias, non-response, missing intersectional cells). Because the claim is stated without this qualification, it is an overgeneralization. The reader's weakest_assumption identifies the same issue; I agree. The proposed test directly settles whether the conclusion is an artifact of the injection mechanism. If the test shows large disparities, the paper should temper the claim to 'uniform random underrepresentation has limited effect under these datasets/models' and add structured missingness as a future-work item. If the test shows no difference, the claim would be robust. No code is provided, which makes the test necessary for verification. The label-bias and proxy analyses are largely independent and appear well-supported; they would survive even if the underrepresentation claim is weakened. Hence the appropriate disposition remains CONDITIONAL (as the reader stated), with the condition being this structured-underrepresentation check and claim tempering. I therefore set verdict_should_be to UNCHANGED.","tokens_in":38369,"tokens_out":4775,"duration_ms":49090,"concrete_test":"Run a structured-underrepresentation ablation on Adult (gender) and Folktables: remove from training all disadvantaged-group examples in a predefined feature subspace (e.g., women with education ≥ Bachelor's, or a specific age/occupation cluster), matched to the same overall reduction in disadvantaged-group prevalence as u=0.8 and u=1 uniform subsampling; keep the test set unbiased. Retrain the same LR/RF/SVC models and report EO, DP, PQP and group-wise TPR. If structured removal yields materially larger fairness gaps than uniform subsampling at matched prevalence (e.g., ΔEO > 0.1), the 'underrepresentation is overemphasized' claim fails to generalize; if the gaps are comparable, the claim is supported. For a stronger test, also run the same protocol on synthetic data where the data-generating process has known group-specific feature overlap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 defines underrepresentation (Eq. 1) as uniform random subsampling of the disadvantaged group: Prσ'(s=d)=r·Prσ(s=d), leaving P(x|y,s=d) unchanged. The central claim that underrepresentation is 'overemphasized' (abstract, §4.3, §6) is inferred entirely from this injection mechanism. However, real-world underrepresentation frequently removes entire feature subspaces or intersectional strata (e.g., only certain occupations/age groups of women, certain dialects, certain skin tones), so the training-set feature distribution for the disadvantaged group is shifted, not merely smaller. Under such structured missingness, models can produce large group-wise performance disparities on unbiased test sets; the paper never tests this. Table 3 already shows statistically significant EO increases for Adult (gender) and NIH even under uniform removal, so the robustness of the conclusion is not strong even for the tested mechanism. The paper's Limitations section (§6) acknowledges other bias types and binary attributes but not this operationalization gap. Since the headline novelty is the underrepresentation finding, this is the most load-bearing weakness. The label-bias and proxy findings are more robust, but the combined claim as stated overgeneralizes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies three types of data bias—underrepresentation, label bias, and proxies—by injecting them into training and validation sets of seven datasets, then measuring fairness on unbiased test sets with three fairness metrics and several models. It reports that underrepresentation has only a minor effect on discrimination, while label bias has a large effect that is amplified by strong proxies. The paper then proposes three detection measures (RD, SD, sAUC) and combines them into a preliminary Data Bias Profile (DBP), demonstrated on popular fairness datasets as a proof of concept for documenting bias, predicting discrimination risk, and selecting fairness interventions. The discussion connects these findings to data governance requirements under the EU AI Act.","tokens_in":38594,"tokens_out":6522,"duration_ms":70800,"significance":"If the empirical ranking of biases is correct, this work would usefully reorder fairness practice toward label curation and proxy management rather than representation targets alone, and the DBP is a promising quantitative complement to qualitative dataset documentation. The paper has concrete strengths: a broad experimental matrix (seven datasets, multiple model families, three fairness metrics, ten repetitions with significance tests), evaluation on unbiased test sets, clearly specified injection protocols, and an explicit limitation discussion. The label-bias and proxy-amplification findings are well supported by the reported results. However, the headline underrepresentation claim and the validation of the DBP detection measures are not yet established at the level of generality claimed in the abstract and Section 6; both need substantial qualification or additional experiments.","major_comments":[{"comment":"The central claim that underrepresentation in training is overemphasized is inferred entirely from an injection mechanism that removes disadvantaged-group points uniformly at random. Real-world underrepresentation often removes structured subsets (feature subspaces, intersectional strata), which shifts P(x|y,s=d) rather than only reducing sample size, and this case is never tested. Even under the tested mechanism, the paper's own results do not uniformly show a minor impact: Adult (gender) EO rises from 0.08±0.02 to 0.21±0.07 for LR, and the appendix shows larger jumps for RF (0.10 to 0.30) and SVC (0.08 to 0.25), with PQP also significantly worsening. The Limitations section lists other bias types and binary attributes but does not acknowledge this operationalization gap. The abstract and Section 6 should either restrict the conclusion to uniform random subsampling or be backed by structured-missingness experiments.","section":"Section 4.2, Eq. (1); Section 4.3, Table 3; Appendix B.1, Tables 9 and 12"},{"comment":"The bias detection measures are validated on biases injected through the same mathematical quantities they measure: RD is exactly the prevalence gap manipulated by Eq. (1), sAUC is exactly the AUC of the proxy classifier used to define proxy strength in Eq. (3), and SD is an AUC-based separability gap that label flipping directly changes. The diagonal responses in Figure 4 are therefore partly by construction, so the experiments demonstrate internal consistency rather than the ability to detect real-world biases of these types. Since the practical value of the DBP depends on detection validity, the paper should provide an external validation or test the measures under bias mechanisms not defined by the same formulas, such as structured underrepresentation or naturally occurring group-dependent label noise.","section":"Section 5.1, Eqs. (10), (13), (14); Section 5.2, Figure 4"},{"comment":"The recommendation that including disadvantaged groups in training can be harmful under weak label bias goes beyond the evidence. The ΔEO values for f=0.2 are negative for most datasets, but several carry standard errors that overlap zero (e.g., Adult-gender -0.01±0.08, Crime -0.11±0.17, Folktables -0.01±0.06, German -0.09±0.17), and NIH and Compas show positive values. The interaction may be real, but the current wording ('hastily adding disadvantaged groups ... can cause more harm than good') is too strong for these point estimates. Please temper the claim or report the uncertainty more prominently.","section":"Section 4.4, Table 5; Section 6"}],"minor_comments":[{"comment":"The notation table defines u = r - 1, while the text and Eq. (1) define u = 1 - r as the underrepresentation factor. Please correct the table to avoid a sign inconsistency.","section":"Table 2 and Section 4.2, Eq. (1)"},{"comment":"The sentence 'as confirmed by the first column of Table 4 (f = 0)' appears to be a reference error; the relevant ΔEO values are in Table 5, column f = 0, not in Table 4.","section":"Section 4.4, paragraph on the joint effect"},{"comment":"Replacing u=1 and f=1 with 0.95 for the detection experiments is reasonable, but the figure axes and the appendix figures still display 1 at the maximum. Please align the axis labels with the actual values used.","section":"Section 5.2, Footnote 5 and Figure 4"},{"comment":"There are minor typos: 'disadvataged' in the Figure 2 caption should be 'disadvantaged', and 'Folkstables' in Figure 5 should be 'Folktables'.","section":"Figure 2 caption and Figure 5"},{"comment":"The NIH disease list contains 'mas', which should likely be 'mass', and the COMPAS description spells 'ProPulica' instead of 'ProPublica'. Please fix these typos.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper has clear strengths in experimental breadth and in the label-bias/proxy findings, and the DBP is an interesting proof of concept. The main risk is overstatement of the underrepresentation conclusion, which rests on a single subsampling mechanism and is partially contradicted by the paper's own significant results for Adult (gender), NIH, and several appendix tables. I would encourage the editor to require either a narrowing of the central claim or additional experiments with structured underrepresentation before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what you should know: this is a real empirical study, not a stunt. It compares three data biases across seven datasets, three fairness metrics, and several model classes, using unbiased test sets. The label-bias and proxy results are solid and useful. The headline claim about underrepresentation being overemphasized, though, is only tested under uniform random subsampling, and the paper's own data undercut it in places. Still worth a serious referee.\n\nWhat's new: the unified comparison itself, and the Data Bias Profile construct. Prior work looked at underrepresentation alone (Vetrò et al.) or representation metrics (Dominguez-Catena et al.). Showing that label bias and proxies interact—strong proxies amplify label-bias harm—is a genuinely useful result for practitioners. The authors are appropriately modest about the DBP being a proof of concept.\n\nSoft spots. First, the underrepresentation conclusion rests on Eq. 1, which removes disadvantaged-group points uniformly at random. Real underrepresentation often deletes feature subspaces or intersectional strata, which can produce much larger disparity. The paper doesn't test that, and the Limitations section doesn't flag it. The stress-test note gets this right. Second, the paper's own tables contradict the 'minor impact' framing: on Adult (gender), u=1 pushes EO from 0.08 to 0.21 (LR) and to 0.25-0.30 for RF/SVC (appendix). That's a real jump. The authors note the exceptions but still generalize. I'd call the conclusion 'mixed' rather than 'overemphasized.' Third, the detection measures are close to direct measurements of the injected bias—RD is the prevalence gap, sAUC is the proxy classifier's AUC, SD is a separability gap that flips directly alter. So Figure 4's diagonal responses are partly by construction. That weakens the 'detection' framing, though the measures may still be useful for documentation.\n\nMinor: no code, DBP shown on two datasets only, no thresholds.\n\nWho should read this: fairness researchers and anyone building data documentation for the EU AI Act. The label-bias and proxy analysis is worth citing; the underrepresentation conclusion should be cited with caution.\n\nRecommendation: send to peer review. It's a solid empirical core with an overbroad claim. The authors should temper the abstract, discuss the missingness mechanism, and release code.","headline":"Solid label-bias and proxy results, but the underrepresentation claim is only tested under uniform random subsampling and is undercut by the paper's own Adult (gender) numbers.","tokens_in":39110,"tokens_out":3012,"would_cite":true,"duration_ms":32032,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Underrepresentation of vulnerable groups in training data is a weaker driver of algorithmic discrimination than label bias and proxy features, and this paper introduces a Data Bias Profile to quantify each.","keywords":["algorithmic fairness","data bias","underrepresentation","label bias","proxy features","bias detection","Data Bias Profile","EU AI Act"],"falsifier":"Run the same bias-injection study with underrepresentation implemented as removal of a structurally defined subpopulation of the disadvantaged group, such as all instances sharing an intersectional feature combination, keeping test sets unbiased; if equal-opportunity gaps rise sharply with the underrepresentation factor, the paper's relative ranking of the three biases fails.","tokens_in":38145,"feed_emoji":"⚖️","tokens_out":7471,"duration_ms":80846,"temperature":0.7,"pith_summary":"This paper sets out to rank the data properties that actually drive algorithmic discrimination rather than take them on faith. It injects controlled doses of three biases — underrepresentation of a disadvantaged group, systematic label corruption against that group, and proxy features carrying information about the protected attribute — into five tabular and two medical imaging datasets, across several model types and three fairness metrics. The paper claims that underrepresentation in training data is overemphasized in fairness research, while label bias, amplified by strong proxies, is the more critical driver of unequal outcomes. To turn this into practice, it proposes detection statistics that need no unbiased reference data and packages them into a Data Bias Profile, a quantitative summary meant to guide dataset documentation and fairness interventions under anti-discrimination regulation.","feed_headline":"Underrepresentation is overrated in AI fairness","feed_subtitle":"Label flips and proxy features move fairness metrics far more than removing vulnerable groups.","key_machinery":"The carrying object is the Data Bias Profile (DBP), a quantitative summary that records three bias signals computed without access to an unbiased reference set: the Representation Difference, the difference between advantaged and disadvantaged group prevalence; the Separation Difference, the average of two cross-group AUC gaps measuring how much harder positive examples of the disadvantaged group are to rank than those of the advantaged group; and the proxy factor sAUC, the AUC of a classifier that attempts to predict the protected attribute from non-protected features. The paper shows each statistic responds mainly to its own injected bias, and that the profile's pattern anticipates both the level of model unfairness and whether a proxy-removal intervention will help.","core_discovery":"The paper's central claim is that the relative importance of data biases for algorithmic discrimination is inverted relative to conventional wisdom: removing up to 100% of a disadvantaged group from the training set leaves equal-opportunity gaps roughly constant in most settings, whereas flipping a share of that group's positive labels produces steep, consistent increases in unfairness, especially in datasets where non-protected features strongly predict the protected attribute. The authors interpret this as evidence that representation-driven interventions are overvalued and that label curation and proxy management deserve priority. To support this claim they evaluate models on unbiased test sets, inject biases only in training and validation, and show that label bias can be strong enough that including the disadvantaged group without fixing labels makes outcomes worse for that group.","pith_inferences":["If underrepresentation were imposed by deleting structured subpopulations rather than uniform random subsampling, the overemphasis conclusion could flip; testing that variant is a direct extension of the injection protocol.","The label-bias results reframe label noise as a fairness problem; the paper's injection protocol could serve as a benchmark for group-dependent label-noise correction methods.","A DBP with thresholds and multi-group support would be a stronger compliance tool for anti-discrimination audits; the paper explicitly leaves thresholds and multi-group extensions as open work.","The finding that label bias can make representation harmful suggests that data collection and labeling efforts should be evaluated jointly, not as separate pipeline stages."],"forward_implications":["Dataset balance should not be treated as the primary fairness fix; scarce annotated data from vulnerable groups is better spent on evaluation than on training.","Label quality is the first thing to audit: even a 20% flip of disadvantaged-group positive labels can widen equal-opportunity gaps significantly.","Proxy strength should guide intervention choice; removing features correlated with the protected attribute helps when sAUC is high and does little when it is low.","Including more disadvantaged-group samples without cleaning their labels can actively worsen outcomes for them, so representation efforts and label repair must go together.","For deployment documentation, a DBP can flag which bias signals a dataset carries and which fairness-enhancing intervention is likely to pay off."],"supporting_citations":[{"why":"The prior work the paper extends; it linked underrepresentation alone to discrimination risk, the exact emphasis the paper argues is overplayed.","marker":"Vetrò et al. 2021"},{"why":"Studies how fairness measures vary with protected-group ratios and class imbalance, providing the methodological backdrop for the underrepresentation experiments.","marker":"Brzezinski et al. 2024"},{"why":"Defines equality of opportunity, the primary fairness metric used to compare bias effects.","marker":"Hardt et al. 2016"},{"why":"Contributes the cross-dataset AUC formulation from which the Separation Difference label-bias detector is built.","marker":"Kallus and Zhou 2019"},{"why":"Exemplifies the received claim that missing representation causes generalization failure for underrepresented groups, which the paper tests and qualifies.","marker":"Shankar et al. 2017"},{"why":"Documents real-world accuracy disparities tied to underrepresentation, setting up the conventional wisdom the paper challenges.","marker":"Buolamwini and Gebru 2018"},{"why":"Supplies the lifecycle taxonomy that justifies treating underrepresentation, label bias, and proxies as the three core data biases.","marker":"Suresh and Guttag 2021"},{"why":"Real-world case of biased target labels in healthcare, motivating the focus on label bias as an especially critical driver.","marker":"Obermeyer et al. 2019"},{"why":"Policy standard that names bias drivers like representation and proxies, connecting the paper's ranking to compliance duties.","marker":"Schwartz et al. 2022"}],"fun_headline_variants":["Underrepresentation overrated: label bias drives AI unfairness","AI fairness: label bias matters more than group size","Label flips, not data removal, spike AI discrimination","For AI, biased labels outweigh missing vulnerable groups"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that underrepresentation is overemphasized assumes it takes the form of uniform random removal of disadvantaged instances; underrepresentation that removes entire feature regions or intersectional subgroups could hurt the group much more.","fun_headline_variants_meta":{"raw":{"variants":["Underrepresentation overrated: label bias drives AI unfairness","AI fairness: label bias matters more than group size","Label flips, not data removal, spike AI discrimination","For AI, biased labels outweigh missing vulnerable groups"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000331,"raw_usage":{"total_tokens":1817,"prompt_tokens":892,"completion_tokens":925,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":861}},"tokens_in":508,"tokens_out":925,"duration_ms":9065,"temperature":1.0,"reasoning_tokens":861,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:50:53.359840+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same bias-injection study with underrepresentation implemented as removal of a structurally defined subpopulation of the disadvantaged group, such as all instances sharing an intersectional feature combination, keeping test sets unbiased; if equal-opportunity gaps rise sharply with the underrepresentation factor, the paper's relative ranking of the three biases fails.","supporting_citations":[{"cited_title":"Properties of fairness measures in the context of varying class imbalance and protected group ratios","cited_arxiv_id":null,"evidence_quote":"Studies how fairness measures vary with protected-group ratios and class imbalance, providing the methodological backdrop for the underrepresentation experiments."},{"cited_title":"Equality of opportunity in supervised learning","cited_arxiv_id":null,"evidence_quote":"Defines equality of opportunity, the primary fairness metric used to compare bias effects."},{"cited_title":"The Fairness of Risk Scores Beyond Classification: Bipartite Ranking and the xAUC Metric","cited_arxiv_id":"1902.05826","evidence_quote":"Contributes the cross-dataset AUC formulation from which the Separation Difference label-bias detector is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Exemplifies the received claim that missing representation causes generalization failure for underrepresented groups, which the paper tests and qualifies."},{"cited_title":"Gender shades: Intersectional accuracy disparities in commercial gender classification","cited_arxiv_id":null,"evidence_quote":"Documents real-world accuracy disparities tied to underrepresentation, setting up the conventional wisdom the paper challenges."},{"cited_title":"Towards a standard for identifying and managing bias in artificial intelligence","cited_arxiv_id":null,"evidence_quote":"Policy standard that names bias drivers like representation and proxies, connecting the paper's ranking to compliance duties."}],"review_version":1}