Pith. sign in

REVIEW 1 major objections 5 minor 60 references

Counterfactual Risk Assessments, Evaluation, and Fairness

T0 review · 1 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Risk assessments should be judged by risk under a baseline intervention, not by outcomes already changed by historical decisions; doubly robust estimators make this possible, and observational fairness parity rarely carries over.

desk verdict The methodological core—DR estimators for counterfactual metrics and the balance conditions linking observational to counterfactual fairness—is solid and novel; read the child welfare claims as an illustration, not decisive evidence, since they depend on an untestable exchangeability assumption. read the letter →

arxiv 1909.00066 v3 pith:5EO4VGXQ submitted 2019-08-30 stat.ML cs.CYcs.LGstat.APstat.ME

Amanda Coston , Alan Mishler , Edward H. Kennedy , Alexandra Chouldechova This is my paper · ORCID
classification stat.MLcs.CYcs.LGstat.APstat.ME MSC 62D20
keywords counterfactualriskassessmentdoublyrobustestimationalgorithmicfairnesspotentialoutcomesdecisionsupportchildwelfarescreeningequalizedoddsbaserateparity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that risk assessment tools used in decision support, such as child welfare hotline screening, are typically trained and evaluated on observed outcomes, so they measure risk under whatever historical decisions were made rather than under the intervention the tool is meant to inform. The authors define counterfactual analogues of standard performance metrics (true positive rate, precision, false positive rate, calibration) in terms of the outcome under a baseline treatment, and provide doubly robust estimators for them. They prove that observational fairness parity implies counterfactual parity only under special balance conditions, so equalizing a standard metric such as base rate parity or equalized odds can, and in their synthetic experiments does, increase disparity in the counterfactual version. On a large child welfare hotline dataset, doubly robust evaluation reverses the usual verdict: a model of risk under screen-out is better calibrated and better at identifying cases later placed out of home or given services than the standard observational model. The paper's core point is that predictive accuracy and fairness are the wrong targets when decisions change outcomes; the right targets are counterfactual.

What carries the argument

The load-bearing object is the counterfactual risk model $E[Y^0 \mid X]$, the probability of the adverse outcome under the baseline treatment such as no investigation, together with its doubly robust estimator $$\widehat{DR}_{$Y^{0}$} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1 - T_i}{1 - \hat{\pi}(X_i)}(Y_i - \hat{s}_0(X_i)) + \hat{s}_0(X_i) \right],$$ which is consistent if either the propensity model or the outcome regression is correctly specified. The second engine is the family of balance conditions (balBP, balPP, balEO), explicit algebraic conditions on the joint distribution of potential outcomes, treatment, and group membership that are necessary and sufficient for observational parity to imply counterfactual parity. The doubly robust estimators for TPR, precision, FPR, and calibration each substitute the augmented outcome $\frac{1-T}{1-\hat{\pi}(X)}(Y - \hat{s}_0(X)) + \hat{s}_0(X)$ into the relevant expectation, which corrects for the fact that treated cases' observed outcomes were altered by the decision itself.

What would settle it

On a dataset where screening decisions were randomized, or where an exogenous shock changed screen-in rates, estimate the counterfactual metrics with the paper's doubly robust procedure and compare them to direct randomized estimates; disagreement beyond sampling error would falsify the identifying assumptions. Without such data, a sensitivity analysis that injects an unmeasured confounder correlated with both screen-in and re-referral, and asks how strong the association must be to erase the counterfactual model's advantage, would settle how load-bearing the exchangeability assumption is.

Watch

Extended reading notes

Core claim

The central claim is that in decision settings the quantity a risk assessment should estimate is the potential outcome under a baseline intervention, $E[Y^0 \mid X]$, rather than the observed outcome $E[Y \mid X]$, and that evaluation and fairness metrics should be defined against $Y^0$. The paper identifies each counterfactual metric under consistency, exchangeability, and weak positivity, and proposes doubly robust estimators that combine a plug-in outcome regression with an inverse-probability-weighted residual correction; under sample splitting and $n^{-1/4}$ nuisance convergence these are $\sqrt{n}$-consistent and asymptotically normal. The theoretical core is a set of three theorems: observational base rate parity implies counterfactual base rate parity if and only if a balance condition (balBP) holds, with analogous necessary-and-sufficient conditions for predictive parity (balPP) and equalized odds (balEO), and with independence conditions given as sufficient cases. The paper further shows empirically that two standard fairness-corrective procedures, reweighing for demographic parity and post-processing for equalized odds, produce counterfactual disparity where none existed before when treatment assignment was already biased. In the child welfare application, the doubly robust evaluation shows the counterfactual model is well-calibrated by race and that the observational model underestimates re-referral risk, conclusions the standard observational evaluation would invert.

Load-bearing premise

The empirical and real-data conclusions assume exchangeability, $Y^0 \perp T \mid X$: the measured features capture every variable that jointly affects whether a call is screened in and whether the family is re-referred within six months. The paper states this assumption is untestable; if unmeasured confounding is present, the doubly robust estimates are biased and the claim that the counterfactual model outperforms the observational model on child welfare data is not supported.

Editorial extensions

If this is right

  • Evaluating a risk model against observed outcomes overstates the performance of observational models, since treated cases' outcomes were partly determined by the intervention; replacing the observed outcome with the counterfactual outcome under the baseline changes which model is selected.
  • Doubly robust counterfactual evaluation is computable for all standard classification metrics, comes with confidence intervals, and is consistent when at least one of the propensity or outcome-regression models is correctly specified.
  • Observational fairness parity transfers to counterfactual parity only under balance conditions that require, roughly, no residual treatment bias within risk strata and no group differences in risk under treatment, conditions unlikely to hold in child welfare and criminal justice.
  • Fairness-corrective methods that equalize observed metrics can induce counterfactual disparity that harms the group historically less likely to receive beneficial treatment, because the correction bakes historical bias into the score.
  • In the child welfare data, the observational model trained on observed re-referrals performs worse than a random classifier at predicting downstream outcomes such as out-of-home placement and services, while the counterfactual model transfers to those related risk tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable audit follows directly from the theorems: estimate the propensity score and the potential-outcome distributions by group and check whether a balance condition such as $\mathrm{balBP}$ is approximately satisfied; if it is not, observational parity claims provide no evidence about counterfactual parity.
  • Because a baseline intervention must be chosen for every counterfactual metric, operationalizing this approach requires a policy judgment, usually 'no intervention,' which regulators and agencies would need to make explicit; part of the fairness debate thereby shifts from statistics to normative choice.
  • The same doubly robust machinery can evaluate not only risk models but the decisions themselves: ranking metrics for responsiveness-targeted interventions, continuous treatment doses, and off-policy comparisons of alternative screening thresholds are direct extensions the paper sketches but does not implement.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper argues that standard evaluation and fairness metrics for risk assessment instruments (RAIs) are distorted by historical treatment decisions, and it introduces counterfactual analogs targeting outcomes under a baseline intervention. It defines doubly robust (DR) estimators for counterfactual TPR, precision, FPR, and calibration (§3.3.3), proves three theorems characterizing when observational fairness (base rate parity, predictive parity, equalized odds) implies the corresponding counterfactual fairness (§4.1), and demonstrates the framework on a synthetic dataset with known potential outcomes and on Allegheny County child welfare hotline data (§3.4). The empirical sections show that DR evaluation reverses the model ranking seen under observational evaluation, and that fairness-corrective methods such as reweighing and post-processing can increase counterfactual disparity.

Significance. If the results hold, the paper provides a principled correction to the evaluation of risk assessments in decision-support settings, where the observed outcome is affected by the historical intervention. The theoretical contributions are strong: Appendix A gives clean identifications under consistency, exchangeability, and positivity; Appendix B gives algebraic proofs of the balance conditions; Theorems 1–3 are substantive and correctly derived. The synthetic experiments are particularly convincing because they evaluate against the true potential outcomes, and the code is publicly released. The real-data analysis is a useful illustration but is limited by the untestable exchangeability assumption; this does not undermine the methodological core, which stands on its own.

major comments (1)
  1. [§3.4.2, assumption (2)] The real-data conclusion that the counterfactual model outperforms the observational model under DR evaluation is identified only under the exchangeability assumption Y0 ⟂ T | X. The authors acknowledge that this assumption is untestable and argue it is plausibly satisfied, but the strength of the empirical claims (e.g., 'DR evaluation shows that the counterfactual model is well-calibrated and the observational model underestimates risk') goes beyond what can be supported without a sensitivity analysis. Because the child welfare demonstration is a stated contribution, I request a sensitivity analysis that quantifies how large unmeasured confounding would need to be to alter the model ranking, or, failing that, a clear statement in the conclusions that these results are conditional on exchangeability.
minor comments (5)
  1. [§1] In the sentence about 'pure predition' settings, 'predition' should be 'prediction'.
  2. [§4.1.3] The heading 'Theorem 3 (Eqalized Odds)' contains a typo; it should be 'Equalized Odds'.
  3. [Appendix D, Figure 9] The caption uses 'counterfatual' instead of 'counterfactual'.
  4. [§3.3.3] The calibration confidence interval is computed with the plug-in bin proportion in the denominator and treats the residual variance as known; the text should note that uncertainty in the estimated bin proportion is ignored, or cite a delta-method treatment.
  5. [§3.4.4, Table 1] The AUROC for the observational model on the placement task has a confidence interval of (0.46,0.49), which is below 0.5; a brief comment on whether this is statistically distinguishable from random would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the counterfactual targets, DR estimators, and fairness theorems are derived from stated assumptions and external statistical theory, not from the quantities they are used to predict.

full rationale

The paper's derivation chain is self-contained. The counterfactual targets in Equations 1-4 are identified in Appendix A from consistency and exchangeability (Assumptions 1-2, Section 3.2.2) using iterated expectations and the potential-outcomes algebra; no target quantity is used to define an input parameter. The doubly robust estimators in Equations 5-12 are standard augmented inverse-propensity-weighting (AIPW) constructions, whose validity is grounded in the external literature cited by the paper (e.g., Robins and Rotnitzky [39, 41] and Robins et al. [51]), not in a result derived within this paper. The synthetic-data evaluation uses a known external ground truth (both potential outcomes are generated and observed for each unit), and the simulation parameters c and k are varied in Appendix D rather than tuned to force the headline ranking. The fairness theorems in Section 4.1 are proved in Appendix B from consistency alone; the balance conditions are derived from probability expansions, not assumed as the conclusion. The real-data child welfare claim is explicitly conditional on exchangeability, which the paper flags as untestable: in Section 3.2.2 it states, 'This is an untestable assumption but it may be reasonable in the child welfare setting where the measured variables capture most of the information the call screeners use to make their decision.' That is an assumption-dependence limitation on the empirical ranking, not a circular reduction: unmeasured confounding would bias all DR estimates, but the estimators would not thereby become equal to their inputs by construction. The paper's self-citations (e.g., [23] for DR rates and [8] for the child welfare setting) provide background, standard method theory, or dataset context; they are not load-bearing for the central derivation, and no self-citation is invoked as a uniqueness theorem or as the sole justification for a defined quantity. Therefore no significant circularity is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central methodology is parameter-free: the DR estimators and the balance conditions in Theorems 1-3 are derived from the potential outcomes axioms, not fit to data. The only hand-set numbers are the simulation parameters (c, k) used to illustrate the phenomena; the qualitative conclusions are robust across the appendix variations. No new entities are postulated.

free parameters (3)
  • c (treatment effect scale, synthetic DGP) = 0.1 (main); 0.3, 0.5 in appendix
    Hand-chosen in the simulation to make treatment substantially reduce the probability of the adverse outcome. Varying it changes how much the counterfactual model beats the observational model, so the headline empirical gap is illustrative for this parameter.
  • k (treatment assignment bias, synthetic DGP) = 1.6 (main); 0.8, 1.0, 2.0 in appendix
    Hand-chosen to make group A=1 more likely to be treated; larger k increases treatment imbalance and the magnitude of the induced counterfactual disparity. Not estimated from data.
  • Decision threshold for post-processing = 0.5
    Used in Section 4.2.2 and Appendix E to compute FPR and FNR; standard choice, not tuned.
assumptions (4)
  • domain assumption Consistency: Y = T Y1 + (1-T) Y0, with no interference between units.
    Invoked in Section 3.2.2 assumption (1) and throughout Appendix B proofs; if interference or versions of treatment exist, the algebraic expansions (Eq. 19, 24) used in Theorems 1-3 fail.
  • domain assumption Exchangeability (no unmeasured confounding): Y0 ⟂ T | X.
    Assumption (2) in Section 3.2.2; required for identifying E[Y0|X] as E[Y|X,T=0] (Appendix A) and for all DR estimators in Section 3.3.3. Untestable; drives the child welfare conclusions.
  • domain assumption Weak positivity: P(π(X) < 1) = 1.
    Assumption (3) in Section 3.2.2; required so the IPW weights (1-T)/(1-π) are well-defined. Theorems 1-3 also assume P(T=0|y0,a) is nonzero.
  • standard math Nuisance estimation consistency and sample splitting for asymptotic normality
    Section 3.3.3 relies on n^{1/4} convergence of the propensity and outcome regression estimates and sample splitting to claim sqrt(n)-consistency and valid confidence intervals; standard results cited from Kennedy (2016) and van der Laan and Robins (2003).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Counterfactual Risk Assessments, Evaluation, and Fairness." pith.science (2026). https://pith.science/paper/5EO4VGXQ

@misc{pith2026190900066,
  author       = {Pith},
  title        = {Pith review of: Counterfactual Risk Assessments, Evaluation, and Fairness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5EO4VGXQ}},
  note         = {Machine review of arXiv:1909.00066}
}
read the original abstract

Algorithmic risk assessments are increasingly used to help humans make decisions in high-stakes settings, such as medicine, criminal justice and education. In each of these cases, the purpose of the risk assessment tool is to inform actions, such as medical treatments or release conditions, often with the aim of reducing the likelihood of an adverse event such as hospital readmission or recidivism. Problematically, most tools are trained and evaluated on historical data in which the outcomes observed depend on the historical decision-making policy. These tools thus reflect risk under the historical policy, rather than under the different decision options that the tool is intended to inform. Even when tools are constructed to predict risk under a specific decision, they are often improperly evaluated as predictors of the target outcome. Focusing on the evaluation task, in this paper we define counterfactual analogues of common predictive performance and algorithmic fairness metrics that we argue are better suited for the decision-making context. We introduce a new method for estimating the proposed metrics using doubly robust estimation. We provide theoretical results that show that only under strong conditions can fairness according to the standard metric and the counterfactual metric simultaneously hold. Consequently, fairness-promoting methods that target parity in a standard fairness metric may --- and as we show empirically, do --- induce greater imbalance in the counterfactual analogue. We provide empirical comparisons on both synthetic data and a real world child welfare dataset to demonstrate how the proposed method improves upon standard practice.

Figures

Figures reproduced from arXiv: 1909.00066 by the authors.

Figure 1
Figure 1. Synthetic data results. Each column is an evaluation method (described in § 3.3). Colors denote the learning method [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Child welfare results. Each column is an evaluation method (§ 3.3). Colors denote the learning method (§ 3.2). Obser [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 5
Figure 5. Recall for downstream child welfare decisions. At [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Child welfare risk distributions by treatment type [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 6
Figure 6. Figure 6: Counterfactual and observational base rates be [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Counterfactual ROC curves before and after post [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Synthetic data results. with parameters with [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Synthetic data results. with parameters with [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Synthetic data results. with parameters with [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Synthetic data results when including treatment decision as a feature with parameters with [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Synthetic data results when including treatment decision as a feature with parameters with [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Post-processing results on synthetic data with parameters [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: Post-processing results on synthetic data with parameters [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: Post-processing results on synthetic data with parameters [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]
Figure 16
Figure 16. Figure 16: Post-processing results on synthetic data with parameters [PITH_FULL_IMAGE:figures/full_fig_p027_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 46 canonical work pages

  1. [1]

    Ahmed M Alaa and Mihaela van der Schaar. 2017. Bayesian inference of indi- vidualized treatment effects using multi-task gaussian processes. In Advances in Neural Information Processing Systems . 3424–3432

    work page 2017

  2. [2]

    Michelle Alexander. 2011. The new jim crow. Ohio St. J. Crim. L. 9 (2011), 7

    work page 2011

  3. [3]

    Chelsea Barabas, Karthik Dinakar, Joichi Ito, Madars Virza, and Jonathan Zittrain

    work page

  4. [4]

    Solon Barocas and Andrew D Selbst. 2016. Big data’s disparate impact. Calif. L. Rev. 104 (2016), 671

    2016

  5. [5]

    Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. 2009. Building classifiers with independency constraints. In 2009 IEEE International Conference on Data Mining Workshops. IEEE, 13–18

    work page 2009

  6. [6]

    Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. 2015. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. InProceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . ACM, 1721–1730

    2015

  7. [7]

    Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data 5, 2 (2017), 153–163

    2017

  8. [8]

    Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountabil- ity and Transparency. 134–148

    work page 2018

Show all 60 references
  1. [9]

    Alexandra Chouldechova and Aaron Roth. 2018. The frontiers of fairness in machine learning. arXiv preprint arXiv:1810.08810 (2018)

    2018 arXiv

  2. [10]

    Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . ACM, 797–806

    2017

  3. [11]

    Maria De-Arteaga, Artur Dubrawski, and Alexandra Chouldechova. 2018. Learn- ing under selective labels in the presence of expert consistency. arXiv preprint arXiv:1807.00905 (2018)

    2018 arXiv

  4. [12]

    Alan J Dettlaff, Stephanie L Rivaux, Donald J Baumann, John D Fluke, Joan R Rycraft, and Joyce James. 2011. Disentangling substantiation: The influence of race, income, and risk on the substantiation decision in child welfare. Children and Youth Services Review 33, 9 (2011), 1630–1637

    2011

  5. [13]

    Julia Dressel and Hany Farid. 2018. The accuracy, fairness, and limits of predicting recidivism. Science advances 4, 1 (2018), eaao5580

    2018

  6. [14]

    Miroslav Dudík, John Langford, and Lihong Li. 2011. Doubly robust policy evaluation and learning. arXiv preprint arXiv:1103.4601 (2011)

    2011 arXiv

  7. [15]

    Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference . ACM, 214–226

    2012

  8. [16]

    Andrew Guthrie Ferguson. 2016. Policing predictive policing. Wash. UL Rev. 94 (2016), 1109

    2016

  9. [17]

    Moritz Hardt, Eric Price, Nati Srebro, et al . 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems . 3315– 3323

    2016

  10. [18]

    Nathan Kallus and Angela Zhou. 2018. Confounding-robust policy improvement. In Advances in Neural Information Processing Systems . 9269–9279

    2018

  11. [19]

    Nathan Kallus and Angela Zhou. 2018. Residual Unfairness in Fair Machine Learning from Prejudiced Data. In Proc. International Conference on Machine Learning. Stockholm, Sweden, 2439–2448

    2018

  12. [20]

    Faisal Kamiran and Toon Calders. 2012. Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems 33, 1 (2012), 1–33

    2012

  13. [21]

    Toshihiro Kamishima, Shotaro Akaho, and Jun Sakuma. 2011. Fairness-aware learning through regularization approach. In 2011 IEEE 11th International Confer- ence on Data Mining Workshops . IEEE, 643–650

    2011

  14. [22]

    Danielle Leah Kehl and Samuel Ari Kessler. 2017. Algorithms in the criminal justice system: Assessing the use of risk assessments in sentencing. (2017)

    2017

  15. [23]

    Edward H Kennedy. 2016. Semiparametric theory and empirical processes in causal inference. In Statistical causal inferences and their applications in public health research. Springer, 141–167

    2016

  16. [24]

    Edward H Kennedy, Zongming Ma, Matthew D McHugh, and Dylan S Small. 2017. Non-parametric methods for doubly robust estimation of continuous treatment effects. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 79, 4 (2017), 1229–1245

    2017

  17. [25]

    Edward H Kennedy, Wyndy L Wiitala, Rodney A Hayward, and Jeremy B Sussman

  18. [26]

    Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf. 2017. Avoiding discrimination through causal reasoning. In Advances in Neural Information Processing Systems . 656–666

    2017

  19. [27]

    Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Ziad Obermeyer. 2015. Prediction policy problems. American Economic Review 105, 5 (2015), 491–95

    2015

  20. [28]

    Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2016. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807 (2016)

    2016 arXiv

  21. [29]

    Amanda Kube, Sanmay Das, and Patrick J Fowler. 2019. Allocating interven- tions based on predicted outcomes: A case study on homelessness services. In Proceedings of the AAAI Conference on Artificial Intelligence

    2019

  22. [30]

    Matt Kusner, Chris Russell, Joshua Loftus, and Ricardo Silva. 2019. Making Decisions that Reduce Discriminatory Impacts. In International Conference on Machine Learning. 3591–3600

    2019

  23. [31]

    Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfac- tual fairness. In Advances in Neural Information Processing Systems . 4066–4076

    2017

  24. [32]

    David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. 2019. Fairness through causal awareness: Learning causal latent-variable models for biased data. In Proceedings of the Conference on Fairness, Accountability, and Transparency . ACM, 349–358

    2019

  25. [33]

    Marc Mauer. 2010. Justice for all-challenging racial disparities in the criminal justice system. Hum. Rts. 37 (2010), 14

    2010

  26. [34]

    Shira Mitchell, Eric Potash, and Solon Barocas. 2018. Prediction-based decisions and fairness: A catalogue of choices, assumptions, and definitions. arXiv preprint arXiv:1811.07867 (2018)

    2018 arXiv

  27. [35]

    Razieh Nabi and Ilya Shpitser. 2018. Fair inference on outcomes. In Thirty-Second AAAI Conference on Artificial Intelligence

    2018

  28. [36]

    J Neyman. 1923. Sur les applications de la theorie des probabilites aux experiences agricoles: essai des principes (Masters Thesis); Justification of applications of the calculus of probabilities to the solutions of certain questions in agricultural experimentation. Excerpts E...

    1923

  29. [37]

    Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger

  30. [38]

    Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. 2009. Dataset shift in machine learning . The MIT Press

    2009

  31. [39]

    James M Robins and Andrea Rotnitzky. 1995. Semiparametric efficiency in multivariate regression models with missing data. J. Amer. Statist. Assoc. 90, 429 (1995), 122–129

    1995

  32. [40]

    In Advances in Neural Information Processing Systems

    On fairness and calibration. In Advances in Neural Information Processing Systems. 5680–5689

  33. [41]

    James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. 1994. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association 89, 427 (1994), 846–866

    1994

  34. [42]

    Donald B Rubin. 2005. Causal inference using potential outcomes: Design, mod- eling, decisions. J. Amer. Statist. Assoc. 100, 469 (2005), 322–331

    2005

  35. [43]

    James M Robins and Andrea Rotnitzky. 2001. Inference for semiparametric models: Some questions and an answer-Comments

    2001

  36. [44]

    Peter Schulam and Suchi Saria. 2017. Reliable decision support using counterfac- tual models. In Advances in Neural Information Processing Systems . 1697–1708

    2017

  37. [45]

    Uri Shalit, Fredrik D Johansson, and David Sontag. 2017. Estimating individual treatment effect: generalization bounds and algorithms. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 . JMLR. org, 3076–3085

    2017

  38. [46]

    Carl-Erik Särndal, Bengt Swensson, and Jan H Wretman. 1989. The weighted residual technique for estimating the variance of the general regression estimator of the finite population total. Biometrika 76, 3 (1989), 527–537

    1989

  39. [47]

    Megan Stevenson. 2018. Assessing risk assessment in action. Minn. L. Rev. 103 (2018), 303

    2018

  40. [48]

    Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert MÞller. 2007. Covari- ate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research 8, May (2007), 985–1005

    2007

  41. [49]

    Vernon C Smith, Adam Lange, and Daniel R Huston. 2012. Predictive modeling to forecast student outcomes and drive effective interventions in online community college courses. Journal of Asynchronous Learning Networks 16, 3 (2012), 51–61

    2012

  42. [50]

    Department of Health & Human Services

    Administration U.S. Department of Health & Human Services. 2019. Child Maltreatment 2017. https://www.acf.hhs.gov/cb/research-data-technology/ statistics-research/child-maltreatment

    2019

  43. [51]

    Mark J Van der Laan, MJ Laan, and James M Robins. 2003. Unified methods for censored longitudinal data and causality . Springer Science & Business Media

    2003

  44. [52]

    Adith Swaminathan and Thorsten Joachims. 2015. Batch learning from logged bandit feedback through counterfactual risk minimization. Journal of Machine Learning Research 16, 1 (2015), 1731–1755

    2015

  45. [53]

    Yixin Wang, Dhanya Sridhar, and David M Blei. 2019. Equal Opportunity and Affirmative Action via Counterfactual Predictions.arXiv preprint arXiv:1905.10870 (2019)

    2019 arXiv

  46. [54]

    Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. 2015. Fairness constraints: Mechanisms for fair classification. arXiv preprint arXiv:1507.05259 (2015)

    2015 arXiv

  47. [55]

    Tyler J VanderWeele and Miguel A Hernan. 2013. Causal inference under multiple versions of treatment. Journal of causal inference 1, 1 (2013), 1–20

    2013

  48. [56]

    Junzhe Zhang and Elias Bareinboim. 2018. Equality of opportunity in classifica- tion: A causal approach. In Advances in Neural Information Processing Systems . 3671–3681

    2018

  49. [57]

    Junzhe Zhang and Elias Bareinboim. 2018. Fairness in decision-making – the causal explanation formula. In Thirty-Second AAAI Conference on Artificial Intel- ligence. 13 FAT* ’20, January 27–30, 2020, Barcelona, Spain Amanda Coston, Alan Mishler, Edward H. Kennedy, and Alexandr...

    2018

  50. [58]

    Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013. Learning fair representations. In International Conference on Machine Learning . 325–333. 12 Counterfactual Risk Assessments, Evaluation, and Fairness FAT* ’20, January 27–30, 2020, Barcelona, Spain

    2013

  51. [2013]

    Medical care 51, 3 (2013), 251

    Improved cardiovascular risk prediction using nonparametric regression and electronic health record data. Medical care 51, 3 (2013), 251

    2013

  52. [2017]

    arXiv preprint arXiv:1712.08238 (2017)

    Interventions over predictions: Reframing the ethical debate for actuarial risk assessment. arXiv preprint arXiv:1712.08238 (2017)

    2017 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.