{"id":"05323f0a-0ad0-47a0-88bd-ca47aed810f9","arxiv_id":"2501.15985","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An auditing framework that uses external population benchmarks to assess AI bias is proposed and illustrated on NYC hiring algorithms, without numerical application of its core metrics.","lead":"This paper proposes a demographic benchmarking framework that compares the demographic makeup of an AI system's affected population against external census data, to surface sampling, deployment, and structural bias. It illustrates the idea with public audits of New York City hiring tools, but never actually computes the framework's own new metrics in that illustration.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The framework's value claim rests on equating census demographics with the unbiased applicant pool, but Section 5's US-vs-NYC benchmark mismatch and the absence of any computation of the proposed metrics leave the central claim undemonstrated.","rationale":"The reader's weakest assumption correctly identifies the benchmark-population mismatch as a central vulnerability. I partially agree but would sharpen it: the problem is not only that US-wide proportions substitute for NYC proportions; it is that census-derived proportions are treated as the normative unbiased applicant pool even though the production population is self-selected and job-specific. This makes the framework's inference from demographic disparity to algorithmic bias unreliable. The paper also introduces metrics (DD, TDD, NDD, and positive-decision variants) but never applies them in the use case, so the central claim is supported only by qualitative reading of raw percentages. These considerations do not move the verdict because the reader's CONDITIONAL assessment already accounts for the need for a consistent benchmark population and an actual computation of the metrics. My concern is a more precise statement of that condition, not a new fatal flaw.","tokens_in":12170,"tokens_out":2429,"duration_ms":28110,"concrete_test":"Re-run the Section 5 analysis using a benchmark matched to the actual applicant pool rather than the U.S. census: use the NYC active-workforce distribution (the paper's own stated training-step population) and, if available, the aggregate applicant distribution across all LL144 audit reports, and compute DD, TDD, NDD, and positive-decision variants. If the flagged ADP Black and RippleMatch Asian deviations fall within the paper's stated fairness thresholds once the applicant pool is controlled, the empirical support for the framework collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that demographic benchmarking can assess model behavior without the training set—requires that the external benchmark be the correct counterfactual for an unbiased production population. Section 5 compares three AEDTs' applicant and positive-outcome distributions to U.S. census race-sex proportions, explicitly because candidate location is unknown, and interprets deviations (e.g., 'ADP exhibits a notable deviation in the Black category') as evidence of bias. But census active-workforce proportions are not the applicant pool for these platforms: applicant composition is shaped by job types, advertising channels, eligibility criteria, and self-selection. A deviation from census is therefore consistent with no algorithmic bias. Moreover, the framework's training-step benchmark (Section 4) would encode current occupational segregation into the 'expected' proportions, so it cannot distinguish structural bias from a benchmark that itself reflects structural bias. The paper never computes any of its own proposed metrics (Eqs. 8-13) on the Table 2 data, so the claim that demographic benchmarking 'proves valuable' is an assertion rather than a demonstrated result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Demographic Benchmarking Framework for auditing AI recommender systems. The framework defines a set of demographic disparity metrics (Eqs. 8-13) that compare an expected demographic distribution, derived from external census data, against the actual distribution in training data and in production data. The intended outputs are three bias indicators: sampling bias (training-phase mismatch), deployment bias (production-phase mismatch), and structural bias (situations where the benchmark itself requires adaptation). The paper reports a use case based on three automated employment decision tools (ADP, RippleMatch, Sheppard Mullin) audited under New York City Local Law 144, comparing their candidate and positive-outcome demographic distributions to U.S. census proportions. It then discusses the benefits of the framework for developers, policymakers, and auditors. The central claim, stated in Section 7, is that demographic benchmarking 'proves valuable' for assessing model behavior without access to the original training set.","tokens_in":1737,"tokens_out":1801,"duration_ms":51865,"significance":"The topic is timely and practically important: auditors often lack access to training data, and post-market monitoring of deployed systems is a recognized gap. The proposed metrics are simple and transparent, and they could, in principle, be used by auditors and regulators. The use case is grounded in public audit reports under a real regulatory framework (NYC Local Law 144), which gives the paper a concrete empirical anchor. However, as presented, the central claim is not demonstrated: Section 5 presents only qualitative comparisons and never computes the proposed metrics, and the benchmark population (U.S. active workforce) does not match the intended applicant pool for the audited systems. The concept of structural bias is also defined in a circular manner in Figure 1 and is not operationalized with any measurable quantity. If these issues were addressed, the framework could become a useful contribution to the fairness-auditing literature; in its current form, it reads more as a position statement than as a validated methodology.","major_comments":[{"comment":"The paper never computes any of the proposed metrics (Eqs. 8-13) on the data it presents. The qualitative observation that 'ADP exhibits a notable deviation in the Black category' is not supported by DD, TDD, NDD, or the positive-decision variants, nor is it compared to any acceptability threshold. Without these calculations, the claim in Section 7 that demographic benchmarking 'proves valuable' is asserted rather than demonstrated. Please compute and report the metrics on the Table 2 data, or explicitly reframe the use case as an illustrative workflow rather than an empirical validation.","section":"Section 5, Table 2"},{"comment":"The use case compares NYC AEDT audit data to U.S. census proportions, despite Local Law 144 being specific to New York City. The paper acknowledges the mismatch ('opaque audits regarding candidates' location data required us to compare results against the broader U.S. population distribution') but then interprets deviations from the U.S. benchmark as potentially indicative of bias. The applicant pool for these platforms is not the entire U.S. active workforce; it is shaped by job types, advertising channels, eligibility criteria, and self-selection. A deviation from U.S. census proportions is therefore not reliable evidence of algorithmic bias. The paper should either use a more appropriate benchmark, such as the NYC active workforce, or explicitly state that the results are not evidence of bias but only a demonstration of the framework's workflow.","section":"Section 5, Table 1 and text"},{"comment":"The concept of 'structural bias' is not operationalized. In Figure 1, structural bias is identified when the benchmark itself 'need[s] adaptation,' which is self-referential: it labels a situation as structural bias precisely when the framework's own reference distribution is judged inadequate. Section 7 describes structural bias as capturing 'the extent to which society may be inherently unfair,' but no metric, equation, or decision rule is provided for it. Please define structural bias in terms of measurable quantities (for example, through a specific pattern in the Eqs. 8-13 metrics or through a separate indicator) and explain how an auditor would distinguish structural bias from deployment bias or from an incorrect benchmark.","section":"Section 4, Figure 1"},{"comment":"The definition of the positive-decision benchmark P_i(D=1) in Eqs. 11-13 is unclear. The text states that 'D represents the true label in training and the model's output in production,' but the equations use P_i(D=1) as an expected proportion of positive decisions for group i, while R_i is the actual proportion in the dataset. It is not explained how P_i(D=1) is derived from external census data or what assumptions support it, since census demographics alone do not determine an expected positive-decision rate. Please clarify the source and meaning of P_i(D=1), as the interpretability of all positive-decision metrics depends on this definition.","section":"Section 4, Eqs. 11-13"}],"minor_comments":[{"comment":"The phrase 'NDD provides a more slight view' appears to be a typo; it likely should read 'a more balanced view' or 'a more nuanced view.'","section":"Section 4, Eq. 10 surrounding text"},{"comment":"The table note says 'All race categories presented exclude individuals identifying as Hispanic or Latino (nor Hispanic or Latino).' The parenthetical should read 'or Hispanic or Latino.' Additionally, the NYC and US percentage columns do not sum to 100, which should be acknowledged as rounding or stated explicitly.","section":"Table 1, note"},{"comment":"The 'Some Other Race' category is omitted from Table 2 because it is 'not required by law,' but the table rows then do not sum to 100% for any of the three tools. Please state this explicitly in the text or include the category in the table so readers can reconcile the totals.","section":"Section 5, Table 2"},{"comment":"The column headers in Table 2 ('ADPPositive % ADP') are unclear; format the headers consistently, for example 'ADP: Total %' and 'ADP: Positive %'.","section":"Table 2, header"},{"comment":"Reference [16] lists the year as '9998' and contains an 'In press' date of 11/2023; please correct the citation metadata. Also, the ACM reference format on the first page states '2018,' which is inconsistent with the 2025 arXiv date.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This appears to be an early working draft (the header explicitly says 'Unpublished working draft. Not for distribution'). The main weakness is not the framework's conceptual ambition but the lack of empirical demonstration: the proposed metrics are never computed in the use case, and the structural-bias notion is defined circularly. If the authors can compute the metrics on the Table 2 data, revisit the benchmark-matching issue, and operationalize structural bias, the paper could become a useful contribution to the algorithmic-auditing literature. As it stands, it is more of a position paper, and its central claim is unsupported by the presented evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one as a position statement from an auditing company, not as a validated method. The useful idea: when auditing a deployed model, compare the distribution of people affected by it to an external demographic reference rather than to the training data, which may itself be biased. That framing—separating sampling, deployment, and structural bias—is genuinely helpful for practitioners working under NYC LL144 or the EU AI Act, where training data is often unavailable.\n\nThe paper does some things well. It grounds the framework in concrete regulation, it is honest about the limitations of its own case study (it admits the NYC-US benchmark mismatch), and the formulas in Eqs. 8-13 are simple enough for auditors to implement without a research team. The positive-decision variants (DDP/TDDP/NDDP) are a small but useful extension that ties representation to model outputs rather than just inputs.\n\nThe soft spots are substantial, though. The biggest is that the Section 5 use case never computes a single one of the proposed metrics. The authors list percentages and then say things like 'ADP exhibits a notable deviation in the Black category,' but they don't calculate DD, TDD, or NDD for any of the three tools. The central claim that demographic benchmarking 'proves valuable' is therefore an assertion, not a demonstration. Second, the benchmark is shifted from New York City to the entire US because candidate location is unknown. Census active-workforce percentages for the US are not the applicant pool for a hiring platform; they are shaped by job types, advertising channels, and self-selection. A deviation from census is not, by itself, evidence of algorithmic bias. The paper acknowledges this, but then still interprets deviations as notable, which undercuts its own caution. Third, 'structural bias' is defined in Figure 1 as the case where the benchmark itself needs adaptation—that is circular, and the prose definition ('population lacks the necessary variability') isn't operationalized. Finally, the fairness thresholds (-0.1 to 0.1, 0.8 to 1.2) are asserted without justification.\n\nNone of this makes the paper worthless. It is a clear, honest statement of a practical auditing approach and a direct response to a regulatory gap. It just doesn't yet show that the approach works.\n\nI'd send it to a serious referee: the framework is testable, the regulatory relevance is real, and a revision that actually computes the metrics on the NY data, fixes the benchmark population, and defines structural bias non-circularly could turn this into a useful paper. For a reading group it's a decent 'what does an audit actually need?' discussion piece, though not a technical result.\n\nRecommendation: engage with it as a proposal, but don't accept it as evidence until the metrics are actually run.","headline":"A plausible framing for external demographic benchmarks in AI audits, but the paper's own case study never computes the metrics and the structural-bias concept is circular.","tokens_in":12870,"tokens_out":2291,"would_cite":false,"duration_ms":20925,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes demographic benchmarking as an audit method that detects sampling, deployment, and structural bias by comparing a model's observed population with an external census-derived reference, without needing the training set.","keywords":["Responsible AI","fairness","bias detection","socio-technical","demographic benchmarking","algorithmic auditing","AI lifecycle","Local Law 144"],"falsifier":"Run the framework on a deployment where the true ground-truth applicant pool is known, such as a hiring tool with internal records of all applicants. If a model whose applicant pool exactly matches the census benchmark still produces outcomes that Local Law 144-style disparate-impact tests flag as discriminatory, or a model whose pool differs widely from the benchmark still produces individually fair decisions, then the demographic disparity measures are not, by themselves, evidence of bias. Repeating the test with actual city-level location data would also settle whether the US-wide benchmark is a valid substitute for New York City.","tokens_in":11982,"feed_emoji":"⚖️","tokens_out":7020,"duration_ms":65401,"temperature":0.7,"pith_summary":"This paper argues that AI bias auditing has a blind spot: standard bias metrics compare groups only within the data the model already has, so if the training data is skewed, the audit inherits the skew. To close that gap, the authors propose demographic benchmarking: using an external, census-derived reference for who should be in the population affected by a model, then measuring how far the model's training data and live production data diverge from that reference. The central claim, tested on hiring tools covered by New York City Local Law 144, is that this external comparison can reveal sampling bias, deployment bias, and a newly named 'structural bias' without access to the original training set. If the framework works as described, auditors gain a bias signal that does not depend on true labels or on trusting the training distribution, and regulators get a quantitative population-representation target to write into law.","feed_headline":"Census data can expose AI bias even when training data is off-limits","feed_subtitle":"New framework compares a model's candidate pool to census demographics to flag sampling, deployment, and structural bias.","key_machinery":"The core machinery is the Demographic Benchmarking Framework and its disparity metrics. Demographic disparity (DD) is the difference between an expected proportion $P_i$ (from the external benchmark) and the actual proportion $R_i$ observed in a dataset for demographic group $i$; total demographic disparity (TDD) sums the absolute group discrepancies, and normalized demographic disparity (NDD) divides each discrepancy by its expected proportion before averaging. Two further metrics, DDP, TDDP, and NDDP, measure the same quantities using only the group shares of positive decisions $P_i(D=1)$, which lets auditors monitor whether a model's favourable outputs follow the expected distribution without needing true labels in production. The framework assigns these metrics to two lifecycle moments—training (to catch sampling bias) and production (to catch deployment bias)—and the residual mismatch that persists even when the benchmark is adjusted is labelled structural bias.","core_discovery":"The paper's central claim is that demographic benchmarking—comparing a model's observed population (in training or production) against an externally defined expected demographic distribution—is a valid and useful way to assess model behaviour and performance even when the original training set is unavailable. In the New York hiring case, the authors build a benchmark from U.S. Census 2020 data restricted to the active workforce (ages 16 and older), compare the candidate pools and positive-outcome groups of three automated employment decision tools to that benchmark, and show that each tool deviates from the reference in distinct ways. They interpret those deviations as evidence of sampling, deployment, or structural bias, and they argue the same external reference can be used during training to build balanced datasets and after deployment as a continuous drift monitor.","pith_inferences":["Editorial inference: the paper gives equations for demographic disparity but does not specify a procedure for choosing the acceptable ranges for TDD or NDD; a natural extension is to derive those ranges from the variance of the census estimates or from regulatory fairness thresholds.","Editorial inference: the NYC example substitutes US-wide census proportions for New York City population because the audit reports lack location data, so the measured 'deployment bias' could partly be a location mismatch; re-running the same audit on data with city-level or borough-level location tags would separate benchmark error from true deployment bias.","Editorial inference: because benchmarks are built from historical census data, they can encode residential or occupational segregation; a testable extension is to compare NDD against benchmarks built from a qualified-applicant pool (e.g., people with the required credentials) rather than the resident population, which would show whether census-resident proportions are the right fairness reference "],"forward_implications":["Auditors can detect bias in models whose training data is proprietary or sealed, because the comparison is against census data rather than against the training set.","Developers can use the benchmark as a dataset-construction target, building training sets whose demographic mix matches the intended real-world population.","Regulators can set explicit, quantitative representation thresholds, such as requiring a production candidate pool to fall within a stated NDD of a local census benchmark, instead of relying on vague data-quality requirements.","Post-deployment monitoring can flag drift when the live user or candidate population moves away from the benchmark, even when no ground-truth labels exist.","The framework gives policymakers a way to see structural bias—cases where the benchmark itself reveals that society's distribution is unequal—and to target equity interventions at the groups most underrepresented."],"supporting_citations":[{"why":"Supplies the fairness-metric toolkit and threshold conventions that the paper builds on when defining acceptable ranges for demographic disparity.","marker":"[8]"},{"why":"Establishes the canonical differential-accuracy audit showing that underrepresentation in training data leads to discriminatory outcomes, motivating an external evaluation reference.","marker":"[9]"},{"why":"Shows that standard fairness metrics are mathematically incompatible, which motivates turning to an external population benchmark instead of relying solely on in-data fairness criteria.","marker":"[10]"},{"why":"Analyzes automated hiring algorithms and shapes the regulatory context of New York City Local Law 144 that the paper's use case sits in.","marker":"[27]"},{"why":"Defines an end-to-end internal algorithmic auditing framework that the paper extends by adding a demographic benchmark for production monitoring.","marker":"[28]"},{"why":"Provides group-fairness audit metrics and a tooling approach that the paper adapts to measure demographic representation rather than only outcome disparities.","marker":"[29]"}],"fun_headline_variants":["Census benchmarks unmask AI bias without training data","Using census data to measure AI fairness at scale","Demographic benchmarks expose hidden AI bias","AI bias detected via external census reference","Benchmarking AI against census to spot structural bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that the census-derived demographic distribution is the correct normative reference for a fair model, and in the New York case that US-wide proportions can stand in for New York City's population, so if either the benchmark population differs from the true intended applicant pool or the location proxy misrepresents it, the measured disparities do not reliably indicate bias.","fun_headline_variants_meta":{"raw":{"variants":["Census benchmarks unmask AI bias without training data","Using census data to measure AI fairness at scale","Demographic benchmarks expose hidden AI bias","AI bias detected via external census reference","Benchmarking AI against census to spot structural bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000594,"raw_usage":{"total_tokens":2789,"prompt_tokens":962,"completion_tokens":1827,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":1758}},"tokens_in":578,"tokens_out":1827,"duration_ms":12722,"temperature":1.0,"reasoning_tokens":1758,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:49:02.704736+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the framework on a deployment where the true ground-truth applicant pool is known, such as a hiring tool with internal records of all applicants. If a model whose applicant pool exactly matches the census benchmark still produces outcomes that Local Law 144-style disparate-impact tests flag as discriminatory, or a model whose pool differs widely from the benchmark still produces individually fair decisions, then the demographic disparity measures are not, by themselves, evidence of bias. Repeating the test with actual city-level location data would also settle whether the US-wide benchmark is a valid substitute for New York City.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the canonical differential-accuracy audit showing that underrepresentation in training data leads to discriminatory outcomes, motivating an external evaluation reference."}],"review_version":1}