External Evaluation of Discrimination Mitigation Efforts in Meta's Ad Delivery

Aleksandra Korolova; Basileal Imana; John Heidemann; Zeyu Shen

REVIEW 4 major objections 6 minor 82 references

External Evaluation of Discrimination Mitigation Efforts in Meta's Ad Delivery

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that Meta's Variance Reduction System satisfies the settlement's impression-variance metric but reaches fewer unique users and raises advertiser cost per person reached, and that a simple budget-splitting alternative…

desk verdict First independent black-box audit of Meta's VRS; settlement-terms critique is the strongest part, while the reach/cost findings are plausible but rest on an imperfectly controlled design. read the letter →

arxiv 2506.16560 v1 pith:2L56DUUR submitted 2025-06-19 cs.CY

Basileal Imana , Zeyu Shen , John Heidemann , Aleksandra Korolova This is my paper · ORCID

classification cs.CY

keywords algorithmicdiscriminationaddeliveryvariancereductionsystemfairnessmetricsexternalauditsettlementcomplianceblack-boxmethodologyfairness-utilitytrade-off

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the 2022 settlement between Meta and the U.S. Department of Justice actually improves access to housing and other opportunity ads, and whether the Variance Reduction System (VRS) Meta built to comply is a good way to do it. The authors argue the settlement's chosen metrics are too weak: they count impressions rather than unique individuals, treat small and large ads equally in coverage targets, and permit leveling down, reducing everyone's exposure to hit variance targets. Using paired real-world ad campaigns run by ordinary advertisers, they find VRS does reduce variance by the settlement's own formula, but mean reach falls by about 10% and cost per 1,000 people reached rises by about 12%, so for a fixed budget fewer unique users see opportunity ads. They then test a transparent alternative, splitting a campaign's budget equally across demographic subgroups, and find it increases reach for all groups while lowering cost, concluding the utility loss of VRS is an implementation choice, not an inevitability.

What carries the argument

The load-bearing object is the settlement's variance metric, defined as half the summed absolute difference between a demographic group's eligible ratio and its delivery ratio, and the VRS bid multiplier that Meta's machine-learning module applies to steer delivery toward the eligible ratio. The paper also relies on its black-box paired-campaign design: the same ad creative and budget run once with VRS enabled and once without, on disjoint balanced custom audiences, so any difference in delivery is attributable to VRS. Its eligible-ratio baseline is estimated externally as the mean delivery ratio across all VRS-enabled ads, justified by the law of large numbers under the assumption that VRS works.

What would settle it

Run the same paired VRS/no-VRS campaigns but compute variance against an independently obtained eligible ratio, for example the demographic impression distribution across all advertisers in the same geographic areas from public ad-library data or a compliance report, and check whether VRS still reduces variance below the 10% threshold; if it does not, the paper's central empirical claim fails. A second decisive test would compare reach and cost under VRS with reach and cost under budget-splitting at larger budgets and longer durations, since the paper's 24-hour, $20 campaigns may not reflect steady-state delivery.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the settlement's compliance framework and Meta's implementation of it decouple 'fairness' as measured by impression variance from access to opportunities as experienced by individuals. The paper shows three structural gaps in the settlement: variance is defined over impressions rather than reach, coverage thresholds count ads rather than people and therefore allow the platform to leave the largest campaigns unregulated, and nothing forbids leveling down. Its field experiments with 36 paired ad campaigns show VRS-enabled ads reduce variance below the 10% threshold for race in all 18 race experiments, but mean reach drops by 9.82% and cost per 1,000 reached rises by 12.02%, with the leveling-down pattern appearing in several replications. The paper further claims its budget-splitting strategy dominated VRS in 36 paired experiments, increasing reach for Black, White, female, and male users while reducing cost, which it reads as evidence that the expense and reduced exposure of VRS are artifacts of Meta's implementation.

Load-bearing premise

The paper's estimate of the eligible ratio, the baseline against which VRS's variance reduction is judged, is not measured from the platform's eligible-audience data; it is the mean of the delivery ratios of the VRS-enabled ads themselves, so if VRS's delivery is itself skewed, the measured reduction in variance may be an artifact of the baseline.

Editorial extensions

If this is right

Under the settlement's own compliance metrics, VRS does reduce variance for housing-tagged ads, including below 5% in 15 of 18 race experiments, so the mechanism is not inert.
For a fixed advertiser budget, VRS reduces the number of unique users who see opportunity ads, with mean reach down 9.82% and cost per 1,000 reached up 12.02%, implying compliance costs are passed to advertisers and users.
Voluntary expansion of VRS to employment and credit ads does not achieve the same variance reduction as housing: variance stayed above 10%, comparable to no-VRS delivery, so voluntary extension should not be assumed equivalent.
Coverage targets based on the share of ads allow the platform to exclude the largest campaigns; using public political-ad data, excluding the largest 19% of ads would exempt about 78.9% of impressions, far more than the 19% a random exclusion would exempt.
Budget-splitting, with an equal budget per demographic subgroup and separate campaigns, increases reach for all groups and lowers cost relative to VRS, showing that higher cost and lower reach are not inherent to fairness interventions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The eligible-ratio estimation is the main circularity risk: because the baseline is derived from VRS's own delivery outcomes, an independent baseline from platform-side aggregate impression data would be the cleanest confirmation that the measured variance reduction is real rather than an artifact.
Reframing compliance metrics in terms of unique reach would align the settlement with its stated goal of opportunity access, and would make VRS's reduced reach directly noncompliant rather than merely undesirable.
The budget-splitting result suggests a concrete, testable design for platforms: expose demographic budget-allocation controls to advertisers, letting fairness constraints be set explicitly instead of through an opaque bid multiplier.
A natural extension of the paper's findings is that future settlements should specify non-degradation constraints on reach and cost alongside variance targets, otherwise compliant systems can still reduce access to opportunities.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version: this paper is the first independent black-box evaluation of Meta's Variance Reduction System, and it's worth reading for the settlement-terms analysis alone. The empirical reach/cost findings are plausible and consistent across 36 runs, but the experimental design leaves a real confound unquantified, and the variance-reduction claim leans on a self-estimated baseline.

What's actually new: the authors toggle VRS on and off by declaring non-housing ads as housing, which is a clever low-cost trick that lets external auditors isolate a deployed mitigation system. They are the first to measure VRS's effect on unique reach and advertiser cost in the wild. The settlement critique in Section 3 is sharp and stands on its own: impressions-versus-reach, the coverage slack that lets Meta exclude the largest campaigns, and the leveling-down risk are each argued clearly and quantified with Ad Library data. The budget-splitting comparison in Section 5 is also genuinely useful as an existence proof that the cost increase is not inevitable.

The soft spots are real but proportionate. The paired design runs VRS and no-VRS copies on disjoint audience partitions to avoid self-competition; that means each pair is not a controlled contrast, since audiences differ in competition, match rates, and activity. The authors acknowledge this in Section 4.1.3, but they do not quantify it, and there are no confidence intervals or significance tests on the -9.82% reach and +12.02% CPP figures. The same design issue carries into the budget-splitting comparison, which runs four separate campaigns. On top of that, the eligible-ratio baseline used for the variance-reduction claim is not measured from Meta's data; it is the mean delivery ratio of the VRS-enabled ads themselves. That creates a circularity problem for the specific claim that VRS reduces variance against the true eligible ratio. The authors are transparent about the limitation, but it is the weakest link in the empirical chain.

None of this sinks the paper. The settlement-terms analysis does not depend on the experiments at all, and the reach/cost direction is consistent enough across many independent runs that I would be surprised if it flipped entirely. The citation pattern is honest, with proper credit to prior paired-ad audits and to Gelauff et al. for budget splitting. No invented entities, no hidden parameters beyond the per-group eligible ratio estimates.

Who is this for: regulators, platform auditors, and fairness researchers who want to know whether a landmark settlement's chosen metric actually helps people. It deserves a serious referee. I would send it to review, and I would ask for uncertainty quantification, a robustness check on the eligible-ratio baseline, and either a same-audience cohort or a formal model of the audience-partition confound before treating the cost/reach numbers as settled.

Referee Report

4 major / 6 minor

Summary. The paper analyzes the 2022 Meta/DoJ settlement and Meta's Variance Reduction System (VRS) for housing ad delivery. It makes three contributions: (1) a policy analysis arguing that the settlement's impression-based variance metric, coverage rule, and lack of a non-leveling-down constraint permit implementations that do not increase individual access to opportunity; (2) a black-box experimental evaluation comparing otherwise identical ads run with and without VRS, reporting that VRS reduces demographic variance but also reduces mean reach by 9.82% and raises cost per 1,000 reach (CPP) by 12.02%, thereby passing compliance costs to advertisers; and (3) a comparison with a budget-splitting alternative that, in the authors' experiments, increases reach for all demographic groups and lowers CPP relative to VRS. The authors also examine VRS in employment and credit domains, finding weaker variance reduction than in housing. Data from the experiments are made public.

Significance. If the empirical findings are correct, this is the first fully independent, advertiser-accessible external audit of VRS, and it provides timely evidence for regulators and civil-rights stakeholders about how settlement metrics interact with real delivery outcomes. The paper's policy analysis of the settlement terms is thoughtful and largely independent of the experiments. The black-box methodology and public data release are valuable for reproducibility and for future audits of ad delivery systems. However, the central empirical claims about VRS's effect on reach and cost currently rest on a confounded paired design and on a circular baseline for the variance evaluation, and the reported aggregate effects lack confidence intervals or significance tests. These issues are load-bearing for the paper's main conclusions, so the manuscript needs substantial revision before the empirical claims can be considered established.

major comments (4)

[§4.1.3, §4.2.3] The paired design runs VRS-enabled and no-VRS ads on disjoint audience partitions to avoid self-competition, so each point in Figure 5 compares outcomes measured on different sets of users. Differences in match rates, auction competition, and user activity between partitions could produce the reported -9.82% reach and +12.02% CPP changes even if VRS had no effect. The paper acknowledges this confound but does not quantify it. Please provide per-pair differences with standard errors or confidence intervals, and ideally a mixed-effects model that includes audience partition as a random effect or a same-audience crossed design. Without such analysis, the central claim that VRS passes compliance costs to advertisers is not supported by the data as reported.
[§4.1.5, §4.2.1] The eligible-ratio baseline used to evaluate VRS is estimated as the mean of the delivery ratios of the VRS-enabled ads themselves, justified by assuming that VRS works and by the law of large numbers. The subsequent claim that VRS reduces variance is therefore partly circular: the baseline is derived from the very system under evaluation. The post-matching audience-size data described in Appendix D provide a more independent basis for the eligible ratio, but the main text does not use them. Please re-estimate eligibility using the audience-match API fractions, or at minimum perform a sensitivity analysis over a plausible range of eligible ratios, and report whether the variance-reduction conclusion survives.
[§4.1.1, §4.2.3] The experiments trigger VRS by declaring non-housing ads as housing ads, and the six creatives include hair products and golfing, which are not economic-opportunity ads. The global reach and CPP results in §4.2.3 are averaged over all 36 experiments, including these non-opportunity creatives, but the paper's policy conclusion is about 'fewer exposures to opportunities for individuals.' The effect of VRS on reach and cost may differ for education, insurance, and financial ads, which are the actual opportunity categories. Please report the reach and CPP results separately for the opportunity ads (EA, EB, IA, FA) and for the non-opportunity ads, and discuss whether the aggregate conclusion holds for opportunity ads alone.
[§5.2] The budget-splitting comparison in Figure 6 also uses separate campaigns on separate audience partitions for the VRS arm and the split arm, with no confidence intervals or significance tests. The claim that budget-splitting 'outperforms VRS' by increasing reach for all groups and reducing CPP is therefore subject to the same audience-confounding concern as the main paired comparison. Please provide per-pair comparisons with uncertainty quantification and, if possible, run the VRS and split arms on the same or crossed audience partitions.

minor comments (6)

[Abstract] The first sentence contains a typo: 'resulted is a first-of-its-kind change' should be 'resulted in a first-of-its-kind change.'
[§2.4] The name 'Pesysakhovich et al.' should be 'Peysakhovich et al.' to match reference [59].
[Figure 4] The labels 'vrs-credit1', 'vrs-credit2', etc. on the y-axis of Figure 4a are not defined in the caption or text; please explain what the axis represents and define 'Declared as.'
[Figures 5 and 6] The scatterplots show individual points but no indication of variability across the three audience replications; adding paired differences with error bars or a small-multiples per-creative breakdown would make the aggregate claims easier to interpret.
[Appendix B] The decision to omit 'flipped' audiences to save costs weakens the location-based race proxy; prior work cited by the authors used flipped audiences as a control. At minimum, please state explicitly that DMA-specific confounds remain possible and discuss how this could affect the race-specific results.
[§4.2.3 and Errata] The errata note indicates that the published FAccT version contained a miscalculation for the Male demographic attribute in Figures 5b and 5c. Please confirm that all numbers in the text, including the 9.82% and 12.02% aggregates, correspond to the corrected figures.

Circularity Check

1 steps flagged · score 5.0 of 10

Variance-reduction verification uses a baseline estimated from the VRS ads themselves, making the central variance claim partially self-referential; reach and cost findings remain independent.

fitted input called prediction [§4.1.5 (Estimating Eligible Ratio) and §4.2.1 (Does VRS Reduce Variance with Respect to Eligible Ratio?)]
"We estimate the eligible ratio used by Meta for our ads by taking the mean of all delivery ratios we observe for each attribute across all VRS-enabled ads we run... Assuming that VRS works as intended and by the law of large numbers, the mean of the delivery ratios is a reasonable estimate."

The 'eligible ratio' baseline is not an external ground truth; it is defined as the mean delivery ratio of the VRS-enabled ads being evaluated. Since the variance metric is the distance from each ad's delivery ratio to this baseline, the VRS group is compared against a reference point constructed from its own outputs, while the no-VRS group is compared against that same VRS-derived reference. The paper's justification for this construction is the assumption that VRS works as intended—the very claim the experiment is meant to establish. Thus the later statement that 'VRS does reduce variance with respect to our estimate of the eligible ratios' is partially self-referential: the baseline shifts with the system under test, so the verification is not independent.

full rationale

The paper's main empirical contributions—VRS raises CPP and lowers reach (§4.2.3), and budget-splitting outperforms VRS (§5)—are direct comparisons between VRS and no-VRS (or VRS and split) campaigns run under the same budget and creative. These do not derive their baseline from the system under test and are not circular. The variance-reduction claim in §4.2.1, however, is partially circular: §4.1.5 defines the eligible ratio as the mean of delivery ratios across the VRS-enabled ads, justified only by assuming VRS works as intended. The variance metric then measures distance from that VRS-derived baseline, so 'VRS reduces variance' is evaluated against a reference constructed from VRS's own outputs, and the no-VRS arm is compared to a baseline it had no role in shaping. This biases the comparison in VRS's favor and makes the verification self-referential rather than an independent check of Meta's compliance metric. The same estimated baseline is reused in §4.2.2 to conclude employment/credit VRS is less effective, inheriting the same issue. Because the paper's headline claim that VRS reduces variance is central but not fully forced (per-ad deviations from the mean remain), and because the cost/reach and budget-splitting results are independent, the overall circularity is moderate: score 5.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rely on a handful of platform-behavior assumptions and one data-derived baseline. The most consequential is the self-estimated eligible ratio in §4.1.5, which creates a circularity burden for the variance-reduction claim. The cost and reach comparisons have no free parameters but share the platform-reporting assumption.

free parameters (1)

Per-group eligible ratio estimates (Black, White, Male, Female) = 0.42, 0.58, 0.45, 0.55
Estimated in §4.1.5 as the mean delivery ratios of VRS-enabled ads; used as the baseline for all variance calculations. This is a data-derived number, not an independently measured eligible ratio, and it directly affects the claim that VRS brings variance below compliance thresholds.

assumptions (5)

domain assumption The eligible ratio for an ad can be estimated by the mean delivery ratio across VRS-enabled ads because VRS works as intended and the law of large numbers applies.
Invoked in §4.1.5; all variance computations use this self-estimated baseline. If VRS systematically misses the true eligible audience, the variance-reduction result is an artifact.
domain assumption Declaring a non-housing ad as housing activates VRS and Meta will not reject the ad.
The entire VRS/no-VRS contrast in §4.1.1 relies on this inference from Meta's and DoJ's public statements; it is not verified with Meta.
domain assumption Running VRS and no-VRS copies on separate audience partitions leaves VRS status as the only meaningful difference.
Acknowledged in §4.1.3 as a potential confound from different competition from other advertisers; no flipped-audience control was run.
domain assumption The heavy-tailed spend and impression distribution of US political ads generalizes to housing and opportunity ads.
Used in §3.2 to quantify impressions that could escape VRS under selective application; the paper cites Meta data but does not use actual housing-ad distributions.
domain assumption Meta's reported reach and cost metrics accurately reflect unique users reached and true advertiser costs.
All reach and CPP comparisons in §4.2.3 and §5.2 depend on platform-reported metrics that the advertiser cannot independently verify.

how reviews work

0 comments

Cite this review

Pith. "Pith review of External Evaluation of Discrimination Mitigation Efforts in Meta's Ad Delivery." pith.science (2026). https://pith.science/paper/2L56DUUR

@misc{pith2026250616560,
  author       = {Pith},
  title        = {Pith review of: External Evaluation of Discrimination Mitigation Efforts in Meta's Ad Delivery},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2L56DUUR}},
  note         = {Machine review of arXiv:2506.16560}
}

read the original abstract

The 2022 settlement between Meta and the U.S. Department of Justice to resolve allegations of discriminatory advertising resulted is a first-of-its-kind change to Meta's ad delivery system aimed to address algorithmic discrimination in its housing ad delivery. In this work, we explore direct and indirect effects of both the settlement's choice of terms and the Variance Reduction System (VRS) implemented by Meta on the actual reduction in discrimination. We first show that the settlement terms allow for an implementation that does not meaningfully improve access to opportunities for individuals. The settlement measures impact of ad delivery in terms of impressions, instead of unique individuals reached by an ad; it allows the platform to level down access, reducing disparities by decreasing the overall access to opportunities; and it allows the platform to selectively apply VRS to only small advertisers. We then conduct experiments to evaluate VRS with real-world ads, and show that while VRS does reduce variance, it also raises advertiser costs (measured per-individuals-reached), therefore decreasing user exposure to opportunity ads for a given ad budget. VRS thus passes the cost of decreasing variance to advertisers. Finally, we explore an alternative approach to achieve the settlement goals, that is significantly more intuitive and transparent than VRS. We show our approach outperforms VRS by both increasing ad exposure for users from all groups and reducing cost to advertisers, thus demonstrating that the increase in cost to advertisers when implementing the settlement is not inevitable. Our methodologies use a black-box approach that relies on capabilities available to any regular advertiser, rather than on privileged access to data, allowing others to reproduce or extend our work.

Figures

Figures reproduced from arXiv: 2506.16560 by the authors.

**Figure 2.** Example screenshots of our ad creatives. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Comparison of variance with and without VRS. VRS [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Comparison of VRS’s performance on housing, the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Comparison of reach metric and cost per 1,000 reach (CPP) with and without VRS. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Reach and CPP by demographic group. The top figures show, compared to VRS, the budget-splitting method increases [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

**Figure 7.** Figure 7: Distribution of ad spend and number of impressions [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 8.** Figure 8: Delivery ratio by race and gender for all VRS [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

82 extracted references · 46 canonical work pages

[1]

Rachad Alao, Miranda Bogen, Jingang Miao, Ilya Mironov, and Jonathan Tannen. 2021. How Meta is working to assess fairness in relation to race in the U.S. across its products and systems. Technical Report https: //ai.facebook.com/research/publications/how-meta-is-working-to-assess- fairness-in-relation-to-race-in-the-us-across-its-products-and-systems

2021
[2]

John Albert. 2023. Not a solution: Meta’s new AI system to contain discriminatory ads. https://algorithmwatch.org/en/meta-discriminatory-ads/

2023
[3]

Muhammad Ali, Piotr Sapiezynski, Miranda Bogen, Aleksandra Korolova, Alan Mislove, and Aaron Rieke. 2019. Discrimination through Optimization: How Facebook’s Ad Delivery Can Lead to Biased Outcomes. In Proceedings of the ACM Conference on Computer-Supported Cooperative Work and Social Computing, Vol. 3. ACM, 1–30. https://dl.acm.org/doi/10.1145/3359301

doi:10.1145/3359301 2019
[4]

Muhammad Ali, Piotr Sapiezynski, Aleksandra Korolova, Alan Mislove, and Aaron Rieke. 2021. Ad Delivery Algorithms: The Hidden Arbiters of Political Messaging. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining (Virtual Event, Israel) (WSDM ’21). Association for Computing Machinery, New York, NY, USA, 13–21. doi:10.1145/3...

arXiv 2021
[5]

Julia Angwin and Terry Paris Jr. 2016. Facebook Lets Advertisers Exclude Users by Race – ProPublica. Retrieved February 18, 2021 from https://www.propublica. org/article/facebook-lets-advertisers-exclude-users-by-race

work page 2016
[6]

Carolyn Ashurst and Adrian Weller. 2023. Fairness Without Demographic Data: A Survey of Approaches. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (Boston, MA, USA) (EAAMO ’23). Association for Computing Machinery, New York, NY, USA, Article 14, 12 pages. doi:10.1145/3617694.3623234

arXiv 2023
[7]

Brian Asquith, Brad Hershbein, Tracy Kugler, Shane Reed, Steven Ruggles, Jonathan Schroeder, Steve Yesiltepe, and David Van Riper. 2022. Assessing the Impact of Differential Privacy on Measures of Population and Racial Resi- dential Segregation. Harvard Data Science Review Special Issue 2 (jun 24 2022). https://hdsr.mitpress.mit.edu/pub/1rsg867y

work page 2022
[8]

Roy L. Austin. 2023. An Update on Our Ads Fairness Efforts. https://about.fb. com/news/2023/01/an-update-on-our-ads-fairness-efforts/

work page 2023

Show all 82 references

[9]

Joachim Baumann, Piotr Sapiezynski, Christoph Heitz, and Aniko Hannak. 2024. Fairness in Online Ad Delivery. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (Rio de Janeiro, Brazil) (FAccT ’24). Association for Computing Machinery, New Y...

2024
[10]

Zoë Ruha Bell, Shafi Goldwasser, Michael P Kim, and Jean-Luc Watson. 2024. Certifying Private Probabilistic Mechanisms. In Annual International Cryptology Conference. Springer, 348–386

2024
[11]

Emily Black, John Logan Koepke, Pauline Kim, Solon Barocas, and Mingwei Hsu. 2024. Less Discriminatory Algorithms. Georgetown Law Journal 113, 1 (2024). doi:10.2139/ssrn.4590481 Washington University in St. Louis Legal Studies Research Paper Forthcoming

2024 doi
[12]

Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, et al. 2024. Black-box access is insufficient for rigorous ai audits. In Proceedings of the 2024 ACM Conference on Fa...

2024
[13]

Elisa Celis, Anay Mehrotra, and Nisheeth K

L. Elisa Celis, Anay Mehrotra, and Nisheeth K. Vishnoi. 2019. Toward Controlling Discrimination in Online Ad Auctions. In International Conference on Machine Learning. https://api.semanticscholar.org/CorpusID:162168628

2019
[14]

Meta Business Help Center. [n. d.]. About Advantage campaign budget. https: //www.facebook.com/business/help/153514848493595?id=629338044106215
[15]

Shuchi Chawla, Christina Ilvento, and Meena Jagadeesan. 2020. Multi-category fairness in sponsored search auctions. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 348–358

2020
[16]

Shuchi Chawla and Meena Jagadeesan. 2022. Individual fairness in advertis- ing auctions through inverse proportionality. In 13th Innovations in Theoretical Computer Science Conference (ITCS 2022) . Schloss Dagstuhl–Leibniz-Zentrum für Informatik, 42–1

2022
[17]

Rachel Cummings, Varun Gupta, Dhamma Kimpara, and Jamie Morgenstern
[18]

Amit Datta, Michael Carl Tschantz, and Anupam Datta. 2015. Automated Experi- ments on Ad Privacy Settings. Proceedings on Privacy Enhancing Technologies 1 (2015). doi:10.1515/popets-2015-0007

2015 doi
[19]

Yashar Deldjoo, Dietmar Jannach, Alejandro Bellogín, Alessandro Difonzo, and Dario Zanzonelli. 2023. Fairness in Recommender Systems: Research Landscape and Future Directions. User Modeling and User-Adapted Interaction 33 (2023), 1–50. doi:10.1007/s11257-023-09364-z

2023 doi
[20]

Cynthia Dwork and Christina Ilvento. 2019. Fairness Under Composition. Schloss Dagstuhl – Leibniz-Zentrum für Informatik. doi:10.4230/LIPICS.ITCS.2019.33

2019 doi
[21]

Cynthia Dwork, Nitin Kohli, and Deirdre Mulligan. 2019. Differential privacy in practice: Expose your epsilons! Journal of Privacy and Confidentiality 9, 2 (2019)

2019
[22]

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Cali- brating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3 . Springer, 265–284

2006
[23]

Cynthia Dwork, Adam Smith, Thomas Steinke, and Jonathan Ullman. 2017. Ex- posed! A Survey of Attacks on Private Data. Annual Review of Statistics and Its Application 4, Volume 4, 2017 (2017), 61–84. doi:10.1146/annurev-statistics- 060116-054123

2017 doi
[24]

Elliott, Peter A

Marc N. Elliott, Peter A. Morrison, Allen Fremont, Daniel F. McCaffrey, Philip Pantoja, and Nicole Lurie. 2009. Using the Census Bureau’s surname list to improve estimates of race/ethnicity and associated disparities. In Health Services and Outcomes Research Methodology . 69–8...

2009 doi
[25]

Jessie Finocchiaro, Roland Maio, Faidra Monachou, Gourab K Patro, Manish Raghavan, Ana-Andreea Stoica, and Stratis Tsirtsis. 2021. Bridging machine learning and mechanism design towards algorithmic fairness. In Proceedings of the 2021 ACM conference on fairness, accountability...

2021
[26]

Ferdinando Fioretto, Diptangshu Sen, and Juba Ziani. 2025. Differentially Pri- vate Graph Data Release: Inefficiencies & Unfairness. In The 28th International Conference on Artificial Intelligence and Statistics

2025
[27]

Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary, Evan P

Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary, Evan P. Hamilton, and Derek Roth. 2019. A comparative study of fairness-enhancing interventions in machine learning. In Proceedings of the Conference on Fairness, Accountability, and Transpare...

2019
[28]

Francis Fukuyama, Barak Richman, Ashish Goel, Marietje Schaake, R Katz, and Douglas Melamed. 2020. Report of the working group on platform scale.Program on Democracy and the Internet, Stanford University, October 6 (2020), 29

2020
[29]

Lodewijk Gelauff, Ashish Goel, Kamesh Munagala, and Sravya Yandamuri. 2020. Advertising for demographically fair outcomes. arXiv preprint arXiv:2006.03983 (2020)

2020 arXiv
[30]

Avijit Ghosh, Ritam Dutt, and Christo Wilson. 2021. When Fair Ranking Meets Uncertain Inference. InProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, Canada) (SIGIR ’21). Association for Computing Mac...

2021
[31]

Avijit Ghosh, Pablo Kvitca, and Christo Wilson. 2023. When Fair Classification Meets Noisy Protected Attributes. In Proceedings of the 2023 AAAI/ACM Confer- ence on AI, Ethics, and Society (Montréal, QC, Canada) (AIES ’23). Association for Computing Machinery, New York, NY, US...

2023 doi
[32]

Guidehouse. 2024. VRS Compliance Metrics Verification. https://www.justice. gov/crt/media/1362086/dl

2024
[33]

Guidehouse. 2024. VRS Compliance Metrics Verification. https://www.justice. gov/crt/media/1385866/dl

2024
[34]

Lily Hu and Yiling Chen. 2020. Fair classification and social welfare. InProceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20). Association for Computing Machinery, New York, NY, USA, 535–545. doi:10.1145/3351095.3372857

2020
[35]

Basileal Imana, Aleksandra Korolova, and John Heidemann. 2021. Auditing for Discrimination in Algorithms Delivering Job Ads. In Proceedings of the Web Conference 2021 (Ljubljana, Slovenia) (WWW ’21). Association for Computing Machinery, New York, NY, USA, 3767–3778. doi:10.114...

2021
[36]

Basileal Imana, Aleksandra Korolova, and John Heidemann. 2023. Having your Privacy Cake and Eating it Too: Platform-supported Auditing of Social Media Algorithms for Public Interest. Proc. ACM Hum.-Comput. Interact. 7, CSCW1, Article 134 (April 2023), 33 pages. doi:10.1145/3579610

2023 doi
[37]

Basileal Imana, Aleksandra Korolova, and John Heidemann. 2024. Auditing for Racial Discrimination in the Delivery of Education Ads. In The 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’24) (Rio de FAccT ’25, June 23–26, 2025, Athens, Greece Basileal...

2024
[38]

Basileal Imana, Aleksandra Korolova, and John Heidemann. 2025. Auditing for Bias in Ad Delivery Using Inferred Demographic Attributes. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’25). Association for Computing Machinery, Athe...

2025
[39]

External Evaluation of Discrimination Mitigation Efforts in Meta’s Ad Delivery

Basileal Imana, Zeyu Shen, John Heidemann, and Aleksandra Korolova. 2025. Dataset for experiments in “External Evaluation of Discrimination Mitigation Efforts in Meta’s Ad Delivery”. https://ant.isi.edu/datasets/addelivery-vrs/

2025
[40]

Kim, Aleksandra Korolova, Guy N

Michael P. Kim, Aleksandra Korolova, Guy N. Rothblum, and Gal Yona. 2020. Preference-informed fairness. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20). Association for Computing Machinery, New York, NY, USA, 54...

2020
[41]

Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2024. The challenge of understanding what users want: Inconsistent preferences and engagement optimization. Management science 70, 9 (2024), 6336–6355

2024
[42]

Lam, Ayush Pandit, Colin H

Michelle S. Lam, Ayush Pandit, Colin H. Kalicki, Rachit Gupta, Poonam Sahoo, and Danaë Metaxa. 2023. Sociotechnical Audits: Broadening the Algorithm Auditing Lens to Investigate Targeted Advertising. Proc. ACM Hum.-Comput. Interact. 7, CSCW2, Article 360 (Oct. 2023), 37 pages....

2023 doi
[43]

Anja Lambrecht and Catherine Tucker. 2019. Algorithmic bias? An empirical study of apparent gender-based discrimination in the display of STEM career ads. Management Science 65, 7 (2019), 2966–2981

2019
[44]

Benjamin Laufer, Manish Raghavan, and Solon Barocas. 2025. What Consti- tutes a Less Discriminatory Algorithm?. In Proceedings of the 2025 Symposium on Computer Science and Law (Munich, Germany) (CSLA W ’25). Association for Com- puting Machinery, New York, NY, USA, 136–151. d...

2025
[45]

Laura Murphy and Associates. 2020. Facebook’s Civil Rights Audit – Final Report. https://about.fb.com/wp-content/uploads/2020/07/Civil-Rights-Audit- Final-Report.pdf

2020
[46]

Jeffrey W Lockhart, Molly M King, and Christin Munsch. 2023. Name-based demographic inference and the unequal distribution of misrecognition. Nature Human Behaviour (2023), 1–12

2023
[47]

Manning, Prabhakar Raghavan, and Hinrich Schütze

Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. In- troduction to Information Retrieval . Cambridge University Press, New York, NY, USA. https://nlp.stanford.edu/IR-book/

2008
[48]

Elisa Celis

Anay Mehrotra and L. Elisa Celis. 2021. Mitigating Bias in Set Selection with Noisy Protected Attributes. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT ’21). Association for Computing Machinery, New York,...

2021 doi
[49]

Anay Mehrotra and Nisheeth Vishnoi. 2022. Fair Ranking with Noisy Protected Attributes. In Advances in Neural Information Processing Systems , Vol. 35. Curran Associates, Inc., 31711–31725. https://proceedings.neurips.cc/paper_files/paper/ 2022/file/cdd0640218a27e9e2c0e52e324e...

2022
[50]

Meta. 2024. Upcoming expansion of special ad categories to include financial products and services in early 2025. Retrieved December 11, 2024 from https: //www.facebook.com/business/help/510724041294968

2024
[51]

Meta. 2025. Reach. Retrieved January 11, 2025 from https://www.facebook.com/ business/help/710746785663278?helpref=faq_content

2025
[52]

Meta Business Help Center. [n. d.]. About Meta Advantage. https://www. facebook.com/business/help/733979527611858
[53]

Meta Technical Report. 2023. Towards Fairness in Personalized Ads. Retrieved No- vember 24, 2023 from https://about.fb.com/wp-content/uploads/2023/01/Toward_ fairness_in_personalized_ads.pdf

2023
[54]

Brent Mittelstadt, Sandra Wachter, and Chris Russell. 2023. The Unfairness of Fair Machine Learning: Levelling down and strict egalitarianism by default. Michigan Technology Law Review (2023). https://arxiv.org/abs/2302.02404

2023 arXiv
[55]

Varun Nagaraj Rao and Aleksandra Korolova. 2023. Discrimination through Image Selection by Job Advertisers on Facebook. In ACM Conference on Fairness, Accountability, and Transparency (FAccT 2023). 1772–1788

2023
[56]

Milad Nasr and Michael Carl Tschantz. 2020. Bidding strategies with gender nondiscrimination constraints for online ad auctions. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20). Association for Computing Machine...

2020
[57]

National Fair Housing Alliance. 2019. Facebook Settlement: Civil Rights Advo- cates Settle Lawsuit with Facebook. Transforms Facebook’s Platform Impacting Millions of Users. https://nationalfairhousing.org/facebook-settlement/

2019
[58]

North Carolina State Board of Elections. [n. d.]. Voter History Data. https://dl.ncsbe.gov/index.html. Downloaded on March 12, 2024

2024
[59]

Alexander Peysakhovich, Christian Kroer, and Nicolas Usunier. 2023. Implement- ing Fairness Constraints in Markets Using Taxes and Subsidies. Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (2023). https://api.semanticscholar.org/CorpusID:257496315

2023
[60]

David Pujol, Ryan McKenna, Satya Kuppam, Michael Hay, Ashwin Machanava- jjhala, and Gerome Miklau. 2020. Fair decision making using privacy-protected data. In Proceedings of the 2020 Conference on Fairness, Accountability, and Trans- parency. 189–199

2020
[61]

Chand Rajendra-Nicolucci and Ethan Zuckerman. 2022. Forgetful Advertising: Imagining a More Responsible Digital Ad System. Yale JL & Tech. 24 (2022), 421

2022
[62]

Rodolfa, Hemank Lamba, and Rayid Ghani

Kit T. Rodolfa, Hemank Lamba, and Rayid Ghani. 2021. Empirical observation of negligible fairness–accuracy trade-offs in machine learning for public policy. Nature Machine Intelligence 3, 10 (Oct. 2021), 896–904. doi:10.1038/s42256-021- 00396-x

2021 doi
[63]

Piotr Sapiezynski, Avijit Ghosh, Levi Kaplan, Aaron Rieke, and Alan Mislove
[64]

Zeyu Shen, Lodewijk Gelauff, Ashish Goel, Aleksandra Korolova, and Kamesh Munagala. 2021. Robust Allocations with Diversity Constraints. In 35th Confer- ence on Neural Information Processing Systems (NeurIPS) , Vol. 34. 29684–29696

2021
[65]

Jonathan Stray, Alon Halevy, Parisa Assar, Dylan Hadfield-Menell, Craig Boutilier, Amar Ashar, Chloe Bakalar, Lex Beattie, Michael Ekstrand, Claire Leibowicz, et al
[66]

Jonathan Stray, Alon Halevy, Parisa Assar, Dylan Hadfield-Menell, Craig Boutilier, Amar Ashar, Chloe Bakalar, Lex Beattie, Michael Ekstrand, Claire Leibowicz, Connie Moon Sehat, Sara Johansen, Lianne Kerlin, David Vickrey, Spandana Singh, Sanne Vrijenhoek, Amy Zhang, McKane An...

2024
[67]

Latanya Sweeney. 2013. Discrimination in Online Ad Delivery: Google Ads, Black Names and White Names, Racial Discrimination, and Click Advertising. Queue 11, 3 (March 2013), 10–29. doi:10.1145/2460276.2460278

2013
[68]

Jun Tang, Aleksandra Korolova, Xiaolong Bai, Xueqiang Wang, and Xiaofeng Wang. 2017. Privacy Loss in Apple’s Implementation of Differential Privacy on MacOS 10.12. arXiv preprint arXiv:1709.02753 (2017)

2017 arXiv
[69]

Facebook’s Civil Rights Team. 2021. Meta’s Progress on Civil Rights Audit Com- mitments. https://about.fb.com/wp-content/uploads/2021/11/Metas-Progress- on-Civil-Rights-Audit-Commitments.pdf

2021
[70]

The US Department of Justice. 2022. Justice Department Secures Ground- breaking Settlement Agreement with Meta Platforms, Formerly Known as Facebook, to Resolve Allegations of Discriminatory Advertising. Retrieved June 29, 2022 from https://www.justice.gov/opa/pr/justice-depar...

2022
[71]

The US Department of Justice. 2024. United States v. Meta Platforms, Inc., f/k/a Facebook, Inc. (S.D.N.Y.). Retrieved Jan 6, 2025 from https://www.justice.gov/ crt/case/united-states-v-meta-platforms-inc-fka-facebook-inc-sdny

2024
[72]

Aditya Srinivas Timmaraju, Mehdi Mashayekhi, Mingliang Chen, Qi Zeng, Quintin Fettes, Wesley Cheung, Yihan Xiao, Manojkumar Rangasamy Kan- nadasan, Pushkar Tripathi, Sean Gahagan, Miranda Bogen, and Rob Roudani
[73]

Ariana Tobin and Jeremy B. Merrill. 2018. Facebook Is Letting Job Advertisers Target Only Men – ProPublica. Retrieved February 18, 2021 from https://www. propublica.org/article/facebook-is-letting-job-advertisers-target-only-men

2018
[74]

Mariia Vladimirova, Federico Pavone, and Eustache Diemert. 2024. Fairjob: A real-world dataset for fairness in online systems. Advances in Neural Information Processing Systems 37 (2024), 10442–10469

2024
[75]

Nils Wernerfelt, Anna Tuchman, Bradley Shapiro, and Robert Moakler. 2022. Estimating the Value of Offsite Data to Advertisers on Meta.University of Chicago, Becker Friedman Institute for Economics Working Paper 114 (2022)

2022
[76]

Wikipedia. [n. d.]. Power law. https://en.wikipedia.org/wiki/Power_law
[77]

Alexandra Wood, Micah Altman, Aaron Bembenek, Mark Bun, Marco Gaboardi, James Honaker, Kobbi Nissim, David R O’Brien, Thomas Steinke, and Salil Vadhan
[2018]

may have an impact on Variance and Coverage

Differential privacy: A primer for a non-technical audience. Vand. J. Ent. & Tech. L. 21 (2018), 209. External Evaluation of Discrimination Mitigation Efforts in Meta’s Ad Delivery FAccT ’25, June 23–26, 2025, Athens, Greece Appendix A Distribution of Impressions/Spend on Meta...

2018
[2019]

In Adjunct publication of the 27th conference on user modeling, adaptation and personalization

On the compatibility of privacy and fairness. In Adjunct publication of the 27th conference on user modeling, adaptation and personalization . 309–315
[2022]

Don’t See Color

Algorithms that "Don’t See Color": Measuring Biases in Lookalike and Special Ad Audiences (AIES ’22). Association for Computing Machinery, New York, NY, USA, 609–616. doi:10.1145/3514094.3534135
[2023]

In Proceedings of the 29th ACM SIGKDD Confer- ence on Knowledge Discovery and Data Mining (KDD ’23)

Towards Fairness in Personalized Ads Using Impression Variance Aware Reinforcement Learning. In Proceedings of the 29th ACM SIGKDD Confer- ence on Knowledge Discovery and Data Mining (KDD ’23) . ACM, 4937–4947. doi:10.1145/3580305.3599916
[2024]

ACM Transactions on Recommender Systems 2, 3 (2024), 1–57

Building human values into recommender systems: An interdisciplinary synthesis. ACM Transactions on Recommender Systems 2, 3 (2024), 1–57

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Rachad Alao, Miranda Bogen, Jingang Miao, Ilya Mironov, and Jonathan Tannen. 2021. How Meta is working to assess fairness in relation to race in the U.S. across its products and systems. Technical Report https: //ai.facebook.com/research/publications/how-meta-is-working-to-assess- fairness-in-relation-to-race-in-the-us-across-its-products-and-systems

2021

[2] [2]

John Albert. 2023. Not a solution: Meta’s new AI system to contain discriminatory ads. https://algorithmwatch.org/en/meta-discriminatory-ads/

2023

[3] [3]

Muhammad Ali, Piotr Sapiezynski, Miranda Bogen, Aleksandra Korolova, Alan Mislove, and Aaron Rieke. 2019. Discrimination through Optimization: How Facebook’s Ad Delivery Can Lead to Biased Outcomes. In Proceedings of the ACM Conference on Computer-Supported Cooperative Work and Social Computing, Vol. 3. ACM, 1–30. https://dl.acm.org/doi/10.1145/3359301

doi:10.1145/3359301 2019

[4] [4]

Muhammad Ali, Piotr Sapiezynski, Aleksandra Korolova, Alan Mislove, and Aaron Rieke. 2021. Ad Delivery Algorithms: The Hidden Arbiters of Political Messaging. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining (Virtual Event, Israel) (WSDM ’21). Association for Computing Machinery, New York, NY, USA, 13–21. doi:10.1145/3...

arXiv 2021

[5] [5]

Julia Angwin and Terry Paris Jr. 2016. Facebook Lets Advertisers Exclude Users by Race – ProPublica. Retrieved February 18, 2021 from https://www.propublica. org/article/facebook-lets-advertisers-exclude-users-by-race

work page 2016

[6] [6]

Carolyn Ashurst and Adrian Weller. 2023. Fairness Without Demographic Data: A Survey of Approaches. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (Boston, MA, USA) (EAAMO ’23). Association for Computing Machinery, New York, NY, USA, Article 14, 12 pages. doi:10.1145/3617694.3623234

arXiv 2023

[7] [7]

Brian Asquith, Brad Hershbein, Tracy Kugler, Shane Reed, Steven Ruggles, Jonathan Schroeder, Steve Yesiltepe, and David Van Riper. 2022. Assessing the Impact of Differential Privacy on Measures of Population and Racial Resi- dential Segregation. Harvard Data Science Review Special Issue 2 (jun 24 2022). https://hdsr.mitpress.mit.edu/pub/1rsg867y

work page 2022

[8] [8]

Roy L. Austin. 2023. An Update on Our Ads Fairness Efforts. https://about.fb. com/news/2023/01/an-update-on-our-ads-fairness-efforts/

work page 2023