{"id":"c68af5c4-f01b-41a9-b563-e8bed1b99e9e","arxiv_id":"2502.00428","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Simulated algorithm audits show that synthetic data and small or incomplete audit samples can make group-parity metrics unreliable, while differentially private aggregate statistics generally remain reliable.","lead":"By simulating third-party audits of a recidivism model and a healthcare coverage model, this paper measures how limited data access changes the reliability of group fairness metrics. It finds that small samples, missing predictors, and especially synthetic data can hide real disparities, while differentially private summary statistics stay accurate at typical privacy settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The high-disparity synthetic-data result may be an artifact of how disparity was constructed by reassigning protected attributes, not a general property of synthetic data.","rationale":"The paper is a careful simulation with a clear, policy-relevant message. It has real strengths: two realistic datasets, multiple access scenarios, multiple synthetic generators, and sensitivity checks across several parity metrics. The reader's conditional verdict is reasonable. My stress-test focuses on the strongest claim in Section 4.4. The high-disparity case in Section 3.5.3 is created by reassigning 95% of positive predictions to the underprivileged group. This makes the protected attribute a near-deterministic function of the model's positive predictions, with no corresponding feature signal. Synthetic generators approximate the joint feature distribution and therefore cannot be expected to reproduce a disparity stored only in the group label. That means the central 'invisibilization' result may be an artifact of the construction. The low-disparity real-model cases show poor overlap too, but those disparities are small, and calling them erased 'disparities' depends on treating small baseline effects as meaningful Type 2 errors; the paper does not report type-1/type-2 rates. A re-run with disparity induced through realistic feature-outcome relationships would settle whether the conclusion generalizes. The reader's own rationale notes that the high-disparity case is artificially constructed, but does not tie this to the central synthetic-data claim; hence partial agreement. This concern does not impugn the rest of the paper, but it sharpens the scope of the strongest recommendation and justifies the existing conditional verdict.","tokens_in":30912,"tokens_out":6512,"duration_ms":64501,"concrete_test":"Re-run the synthetic-data experiment (Sections 3.2.5 and 3.5.3) with high disparity induced by a realistic mechanism: keep the original features, sample the outcome with a strong group-dependent offset (e.g., add +1.0 to the log-odds of the positive outcome for the underprivileged group), and train the audited XGBoost model on that data. Then generate synthetic audit samples with the same four generators and compute the Table 5 overlap and the type-1/type-2/reverse error rates. If synthetic estimates again collapse to parity, the strong claim stands; if they track the true disparity, the current high-disparity result is an artifact of the protected-attribute reassignment.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's strongest claim (Section 4.4) is that synthetic data systematically invisibilizes group disparities, so synthetic data should not replace real data in fairness evaluations. The key high-disparity evidence is constructed artificially in Section 3.5.3: to create high disparity, the authors 'reassign 95% of those who receive a positive prediction to the underprivileged group, and reassign the rest to the privileged group.' This makes the protected attribute a near-deterministic function of the model's positive predictions. Synthetic generators are trained to approximate the joint distribution of features, protected attribute, and outcome; a disparity encoded only in the protected-attribute label, with no corresponding feature signal, is structurally unlikely to be reproduced by any generator that learns from the feature distribution. The resulting 'invisibilization' may therefore be an artifact of the construction rather than a general property of synthetic data for audits of real biased models, where disparity typically flows through features correlated with group membership. The paper's own real models have only low disparity, so they do not provide an independent test of the strong claim. The low-disparity results show poor overlap, but those baseline disparities are small, and calling them erased 'disparities' depends on treating small effects as meaningful Type 2 errors; the paper does not report the actual type-1/type-2/reverse error rates. The conclusion 'synthetic data should not replace real data' is thus broader than the evidence supports.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates how limited data access affects the reliability of quantitative algorithm audits that estimate group parity metrics. Through simulations on two public datasets (NIJ Recidivism and ACS Public Coverage), the authors compare three access scenarios: aggregate confusion matrices only (A), individual-level data with model predictions (B), and individual-level data without model predictions (C). They examine five quality-loss factors: sample size reduction, feature removal, disparate missing values, differentially private aggregation, and synthetic data generation. The main claims are that data minimization and anonymization can substantially increase error rates, that differentially private confusion matrices are generally reliable, and that synthetic data systematically invisibilizes disparities and should not replace real data in fairness evaluations. The paper also discusses regulatory and HCI implications.","tokens_in":31126,"tokens_out":4953,"duration_ms":46307,"significance":"If the strong claims are supported, the paper makes a valuable contribution to the algorithmic auditing literature by providing empirical evidence on how common data-sharing practices can undermine audit integrity. The experimental design is a notable strength: 100 train/audit splits, 500 bootstrap repetitions, multiple model classes, multiple parity metrics, and two realistic public datasets. The paper also engages seriously with the legal and HCI dimensions of auditor access. However, the significance hinges on the validity of the headline conclusions about differentially private aggregates and synthetic data, which are currently not fully supported by the reported statistics.","major_comments":[{"comment":"The statement that with n=1,000 a privacy budget of epsilon=0.5 or above yields estimations that are 'highly unlikely to lead to interpretation errors' is contradicted by Table 4. At n=1,000, the proportion of metric values within the baseline 95% CI is 0.08 for ACS AOD high disparity at epsilon=0.5, 0.20 at epsilon=1, and 0.41 and 0.63 for NIJ SPD high and low disparity at epsilon=0.5. These values are far below the 0.70 threshold the authors themselves use to indicate high overlap. This claim is load-bearing for the recommendation that differentially private confusion matrices are well-suited for public releases, and it should be revised or supported with direct error-rate reporting.","section":"Section 4.1, Table 4"},{"comment":"The high-disparity condition is constructed by reassigning 95% of positive predictions to the underprivileged group, making the protected attribute a near-deterministic function of the model output. Synthetic generators trained on the joint distribution of features and outcomes have no feature-based signal for this disparity and are structurally unlikely to reproduce it. The observed 'invisibilization' of disparities in the high-disparity condition is therefore potentially an artifact of this construction rather than a general property of synthetic data for audits of real biased models, where disparity typically flows through features correlated with group membership. The paper's own trained models have only low disparity, so they do not independently test the strong claim that synthetic data 'has a strong tendency to invisibilize disparities' (Section 4.4). I recommend an additional experiment where high disparity is induced through feature-based mechanisms, such as strengthening group-correlated coefficients or applying group-specific base rates.","section":"Section 3.5.3 and 4.2.4"},{"comment":"The paper defines Type 1, Type 2, and reverse errors based on whether the baseline and experimental confidence intervals share the same configuration, but it never reports the actual frequencies of these error types. The only quantitative measure reported is the proportion of metric values within the baseline CI (e.g., Tables 4-11), which is not the same as an error rate. Consequently, claims such as 'highly unlikely to lead to interpretation errors' (Section 4.1) and 'synthetic data generation has a strong tendency to invisibilize disparities, often leading to Type 2 errors' (Section 4.4) are not directly supported by the reported statistics. The authors should report the fraction of splits where each error type occurs for the headline conditions, at least for the main results in Section 4.","section":"Section 3.4 and throughout Results"},{"comment":"The simulations assume the audit dataset contains ground-truth outcomes and protected characteristics for every individual. Section 5.5 acknowledges this excludes applications where sensitive features cannot be collected, but the abstract and policy recommendations (e.g., Section 5.2) draw broad conclusions about data access and anonymization without consistently carrying this caveat. Prior work cited by the authors (Kallus et al. [78]) shows that proxies for protected attributes are unreliable for disparity assessment. The measured error rates for missing features, synthetic data, and sample size therefore do not automatically transfer to settings without protected characteristics. The paper should either explicitly restrict its conclusions to the assumed setting or include an additional analysis where protected attributes must be inferred.","section":"Section 3.1 and 5.5"}],"minor_comments":[{"comment":"The claim that Access Scenario A offers 'high metric accuracy' should be qualified by sample size and epsilon, since Table 4 shows low overlap for small n even at moderate epsilon values.","section":"Table 4 and Section 4.1"},{"comment":"The methods section states that PrivBayes is used with the DPART library but does not report the privacy budget used for PrivBayes; Table 10 lists two epsilon values (1 and 5). Please state the epsilon choices in the methods.","section":"Section 3.2.5"},{"comment":"The text says 'results generalized across these metrics,' but the appendix only shows results for the ACS dataset across three metrics; the claim for the NIJ dataset is not substantiated in the appendix.","section":"Section 3.6 and Appendix A.2"},{"comment":"The cumulative feature-removal plots are described as 'at the F14 point, features 14 to 18 are missing,' but the text also refers to '5-7 low importance features' for NIJ. Please clarify the correspondence between the x-axis labels and the number of removed features.","section":"Figure 4"},{"comment":"The high-disparity reassignment procedure should specify whether it is applied only to the audit set or also to the training set, and whether the ground-truth labels are recomputed after reassignment. This is important for interpreting the synthetic data results.","section":"Section 3.5.3"},{"comment":"The statement that intersectional assessments 'would require even more granular and higher quality data' is speculative; consider marking it explicitly as a conjecture or providing reference to supporting evidence.","section":"Section 5.5"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional verdict is appropriate. The strongest concern—that the synthetic-data result may be an artifact of the artificial disparity construction—lands and should be addressed in revision. The DP reliability claim is also contradicted by the paper's own Table 4. The empirical setup is otherwise solid and the paper has the potential to be a useful contribution to the auditing literature, but the reporting gaps in error rates and the scope caveat need to be handled before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core contribution is the joint comparison of three audit access levels across five data-quality factors on two realistic prediction tasks. That is genuinely new, and the experimental design is careful: 100 train/audit splits, bootstrap CIs, multiple metrics and model classes. The DP confusion-matrix result (reliable except at tiny epsilon and small n) and the feature-removal results are useful and credible.\n\nThe central claim—that synthetic data systematically erases group disparities—is important and broadly consistent with prior work, but the strong version in Section 4.4 is not fully supported. The high-disparity case is constructed by reassigning protected attributes based on the model's predictions, making the protected label a near-deterministic function of the outcome. Synthetic generators trained on the joint distribution will struggle to reproduce that kind of label, so the \"invisibilization\" may be partly an artifact of the construction. The real models only have low disparity, so the paper lacks an independent high-disparity test with naturally encoded disparity. That said, the low-disparity results also show poor overlap, so the general warning about synthetic data is not baseless—it just needs to be stated with more nuance.\n\nTwo reporting gaps weaken the paper. First, the paper never reports actual Type 1/Type 2/reverse error rates, only CI overlap proportions. The DP small-sample claim (\"highly unlikely to lead to interpretation errors\") is asserted without the direct evidence. Second, the synthetic-data procedure in Access Scenario C is under-specified: it is unclear whether the replicated model is trained on synthetic data or real data. Also, no code or data artifacts are provided, which is a real reproducibility shortfall for an empirical simulation.\n\nThese are addressable issues that do not overturn the main message. The paper deserves a serious referee, but the revision should clarify the synthetic pipeline, report error rates, and temper the \"synthetic data should not replace real data\" conclusion to match the evidence. I would bring it to reading group and would cite it as the first multi-factor comparison of audit access levels.","headline":"First systematic comparison of audit access levels; the synthetic-data finding is real but the strong version overreaches.","tokens_in":31686,"tokens_out":2578,"would_cite":true,"duration_ms":26974,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Synthetic data erases the disparities that algorithm audits are designed to detect, and should not replace real data in fairness evaluations.","keywords":["algorithm auditing","group parity metrics","synthetic data","differential privacy","data minimization","fairness evaluation","data access","privacy-enhancing technologies"],"falsifier":"Run the same audit protocol with a synthetic-data generator not in the paper's set, such as a modern diffusion-based tabular model, on the NIJ and ACS datasets; if the synthetic samples reproduce the baseline parity metrics' confidence intervals at high overlap rates for both low- and high-disparity cases, the claim that synthetic data systematically erases disparities would be falsified for that generator class.","tokens_in":30671,"feed_emoji":"🔍","tokens_out":6471,"duration_ms":60180,"temperature":0.7,"pith_summary":"This paper tries to establish which data-sharing practices let an outside auditor reliably measure whether an algorithm treats demographic groups equally, and which practices produce misleading verdicts. The authors simulate audits of two real prediction models—recidivism and public health coverage—under three levels of access, degrading the audit data the way real organizations do: smaller samples, missing features, unevenly missing values, differentially private noise, and synthetic data. The central finding is that data minimization and anonymization can push fairness metrics so far from the truth that auditors declare parity where bias exists, with synthetic data the worst offender: every generator tested erases group disparities. The paper concludes that synthetic data should not replace real data in fairness evaluations, while differentially private confusion matrices can stay reliable at privacy budgets that are realistic in practice.","feed_headline":"Synthetic data hides the disparities audits are meant to catch","feed_subtitle":"In simulations, privacy-focused data sharing produces false 'no bias' verdicts; differential privacy on aggregates stays reliable.","key_machinery":"The machinery is a controlled audit simulation. The authors train an XGBoost model on 70% of a real dataset, hold out 30% as the audit set, compute baseline group parity metrics (Statistical Parity Difference for recidivism, Average Odds Difference for health coverage) with bootstrapped 95% confidence intervals, and then re-run the same metrics after degrading the audit data in five ways: subsampling, feature removal, disparate missingness, differentially private noise on aggregate confusion matrices, and synthetic data generated by Gaussian Copula, CT-GAN, Copula-GAN, and PrivBayes. Reliability is measured as the proportion of experimental metric values that fall inside the baseline confidence interval, plus a classification of interpretation errors (Type 1, Type 2, and reverse errors). This design lets the authors attribute shifts in audit conclusions to specific data-sharing practices rather than to metric or model choice.","core_discovery":"Simulating an auditor computing group parity metrics on two real-world prediction tasks (recidivism and public health coverage), the paper finds that privacy-protective data-sharing practices systematically degrade audit reliability in different ways. A differentially private confusion matrix—an aggregate—produces accurate parity estimates at privacy budgets of ε ≥ 0.5 for samples above roughly 1,000 people, meaning strong privacy protection and reliable audit metrics are compatible under Scenario A access. Individual-level data with model outputs (Scenario B) and without model outputs (Scenario C) become unreliable when the sample drops below about 1,000, when key predictive features are missing, or when missing values are concentrated in the underprivileged group; even 1% disparate missingness reduced the proportion of estimates within the baseline 95% confidence interval to 72% under Scenario B and 25% under Scenario C. The strongest and most actionable claim is that synthetic data has a systematic tendency to invisibilize disparities: metrics computed on synthetic samples consistently indicate parity regardless of the true disparity level, producing Type 2 errors. The paper concludes that synthetic data is highly misleading for auditing and should not replace real data in fairness evaluations.","pith_inferences":["A natural extension is to test whether the invisibilization result holds for newer generative tabular models such as diffusion-based synthesizers; if some generators preserve disparities, the paper's blanket conclusion would need to be scoped by generator class.","The results imply that regulatory proposals for remote data science or sandbox access should prefer differentially private aggregates over synthetic data releases, since the former preserve audit signal while the latter systematically destroy it.","Because intersectional subgroups are smaller than the two-group samples studied here, the sample-size thresholds found in this paper imply that intersectional audits will be even more fragile under data minimization, strengthening the case for privileged access to such assessments.","The Type 2 error result suggests a concrete disclosure rule: data holders who release synthetic or minimized data should be required to document exactly what was removed or generated, so auditors can discount the data rather than be misled."],"forward_implications":["Regulators and platforms that suggest synthetic data as a privacy solution for auditors should treat it as unfit for quantitative fairness evaluations unless a generator is shown to preserve group disparities.","Differentially private confusion matrices are a viable low-disclosure transparency mechanism: organizations could release them by default for public accountability without endangering audit reliability.","Auditors with individual-level data should require disclosure of sample size, the list of model predictors, and group-wise missingness rates before trusting a parity estimate.","Audits that replicate a model (Scenario C) are as reliable as direct score access only if the audit sample is roughly 160% larger for the recidivism case, and they collapse under missing features or synthetic data, so direct access to model outputs should be prioritized.","Even small amounts of missing data concentrated in an underprivileged group can reverse an audit conclusion, which makes missingness diagnostics a necessary part of standard audit practice."],"supporting_citations":[{"why":"Shows that sample size strongly affects the reliability of groupwise fairness metric estimates, motivating the subsampling experiment's error thresholds.","marker":"[77]"},{"why":"Prescribes confidence intervals rather than point estimates for parity metrics, which the paper adopts as its baseline ground truth.","marker":"[19]"},{"why":"Warns that privacy-preserving data sharing mechanisms, including synthetic data, often fail to preserve utility, a claim the paper's synthetic-data results substantiate.","marker":"[120]"},{"why":"Documents errors introduced when publishing and using synthetic data, supporting the paper's invisibilization finding.","marker":"[129]"},{"why":"Demonstrates that proxies for protected demographic characteristics are unreliable for disparity assessments, which anchors the paper's assumption that audit data must include the protected characteristic.","marker":"[78]"},{"why":"Provides the taxonomy of record-level and aggregation-based privacy techniques that structures the paper's three access scenarios.","marker":"[58]"},{"why":"Shows that missing values bias fairness estimations, the prior result the disparate-incompleteness experiment extends to audit settings.","marker":"[134]"},{"why":"Supplies PrivBayes, one of the four synthetic data generators tested, whose outputs always indicated parity.","marker":"[133]"},{"why":"Provides the ACS Public Coverage dataset and prediction task used as one of the two audit case studies.","marker":"[44]"}],"fun_headline_variants":["Synthetic data hides the very disparities audits seek","Audit reliability suffers with synthetic or tiny samples","Differential privacy keeps audits accurate, synthetic does not","Real data needed: synthetic data masks algorithmic bias","Small sample audits: error rates soar, bias hidden"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The simulations assume the auditor has the ground-truth outcome and the protected characteristic for every person in the audit dataset; if real audit data lacks these fields, the measured error rates and the synthetic-data warning do not directly apply.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic data hides the very disparities audits seek","Audit reliability suffers with synthetic or tiny samples","Differential privacy keeps audits accurate, synthetic does not","Real data needed: synthetic data masks algorithmic bias","Small sample audits: error rates soar, bias hidden"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000423,"raw_usage":{"total_tokens":2159,"prompt_tokens":920,"completion_tokens":1239,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":1176}},"tokens_in":536,"tokens_out":1239,"duration_ms":9942,"temperature":1.0,"reasoning_tokens":1176,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T19:02:54.726558+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same audit protocol with a synthetic-data generator not in the paper's set, such as a modern diffusion-based tabular model, on the NIJ and ACS datasets; if the synthetic samples reproduce the baseline parity metrics' confidence intervals at high overlap rates for both low- and high-disparity cases, the claim that synthetic data systematically erases disparities would be falsified for that generator class.","supporting_citations":[{"cited_title":"Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference","cited_arxiv_id":null,"evidence_quote":"Shows that sample size strongly affects the reliability of groupwise fairness metric estimates, motivating the subsampling experiment's error thresholds."},{"cited_title":"Why the search for a privacy-preserving data sharing mechanism is failing","cited_arxiv_id":null,"evidence_quote":"Warns that privacy-preserving data sharing mechanisms, including synthetic data, often fail to preserve utility, a claim the paper's synthetic-data results substantiate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents errors introduced when publishing and using synthetic data, supporting the paper's invisibilization finding."},{"cited_title":"Assessing Algorithmic Fairness with Unobserved Protected Class Using Data Combination","cited_arxiv_id":null,"evidence_quote":"Demonstrates that proxies for protected demographic characteristics are unreliable for disparity assessments, which anchors the paper's assumption that audit data must include the protected characteristic."},{"cited_title":"Anonymization: The imperfect science of using data while preserving privacy","cited_arxiv_id":null,"evidence_quote":"Provides the taxonomy of record-level and aggregation-based privacy techniques that structures the paper's three access scenarios."},{"cited_title":"Assessing Fairness in the Presence of Missing Data","cited_arxiv_id":null,"evidence_quote":"Shows that missing values bias fairness estimations, the prior result the disparate-incompleteness experiment extends to audit settings."},{"cited_title":"M., Srivastava, D., and Xiao, X","cited_arxiv_id":null,"evidence_quote":"Supplies PrivBayes, one of the four synthetic data generators tested, whose outputs always indicated parity."}],"review_version":1}