Pith. sign in

REVIEW 4 major objections 5 minor 41 references

A scalable Bayesian double machine learning framework, with application to racial disproportionality assessment

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read This paper claims that a Bayesian empirical likelihood combined with double machine learning yields a coverage-valid posterior for high-dimensional semiparametric treatment effects, and that applying it to London stop-and-search data…

desk verdict The Bayesian DML wrapper is a plausible incremental method with decent simulations, but the London application's headline negative effect is likely built into the outcome definition, so the empirical claim should not be trusted as stated. read the letter →

arxiv 2502.07695 v3 pith:WBJFIIUS submitted 2025-02-11 stat.AP stat.ME

classification stat.APstat.ME MSC 62F1562G0562P25
keywords partiallylinearregressionBayesianempiricallikelihooddoublemachinelearningstopandsearchracialdisproportionalityhigh-dimensionalconfoundingposteriorcoveragecausalinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a Bayesian double machine learning procedure for estimating a treatment effect in a partially linear model with many confounders. It replaces the usual likelihood with a generalized empirical likelihood that reweights the data to satisfy the Neyman-orthogonal moment condition, and it uses sample splitting so machine-learned nuisance functions can be plugged in without contaminating the posterior. The paper argues that the resulting posterior is valid in the coverage sense of Monahan and Boos (1992): simulated H statistics are uniform. Applied to 2.6 million London stop-and-search records aggregated to 33 boroughs, the method finds a negative posterior mean of the effect of Black population share on the expressive-crime disproportionality index, ranging from about -0.55 to -0.31 across all nine method-ML combinations. The authors interpret this as evidence that accounting for Black population share alleviates disproportionality: boroughs with larger Black shares show lower stop-and-search disproportionality for expressive crimes.

What carries the argument

The load-bearing object is the Bayesian generalized empirical likelihood posterior combined with sample-splitting double machine learning. The treatment effect β solves E[ψ(Z;β)] = 0 with ψ = [D−π(X)][Y−βD−µ(X)], where π and µ are nuisance functions estimated out-of-fold by Lasso, random forest, or neural network. The likelihood is replaced by profile empirical-likelihood weights p_i that minimize a Cressie-Read divergence subject to the moment condition, and β is updated by MCMC. Validity is certified by the Monahan-Boos H statistic, which checks that the posterior cumulative distribution at the true β is uniform over prior-predictive draws.

What would settle it

Re-estimate the effect using a leave-one-out disproportionality index where each borough's rate is divided by the London average computed from the other 32 boroughs only; if the negative posterior mean disappears or flips sign, the headline result is an artifact of the denominator. A sharper test would substitute an individual-level outcome such as the probability of being stopped given presence, or add borough-level crime counts to the confounder set and re-run the analysis.

Watch

Extended reading notes

Core claim

The central claim is that a Bayesian posterior can be built around the double machine learning estimating equation and still be a proper posterior. The posterior uses weights p_i from a generalized empirical likelihood (EL, ETEL, or Hellinger distance) subject to the moment condition, giving π(β|z) ∝ π0(β) ∏ p_i; the paper demonstrates by simulation that the resulting intervals achieve nominal coverage when checked with the Monahan-Boos H statistic. On the applied side, the paper claims that 'accounting for the proportion of the Black population can help alleviate disproportionality' in expressive-crime stop and search. Across all specifications, the posterior mean for β is negative, between -0.55 and -0.31; the 95% credible intervals from random-forest and neural-network nuisance models exclude zero, while Lasso intervals are wider and contain zero.

Load-bearing premise

The causal reading rests on no unmeasured confounding at the borough level: after conditioning on the 31 observed covariates, boroughs with larger Black population shares must not differ systematically in unmeasured drivers of stop-and-search disproportionality, such as actual crime rates, police deployment, or reporting practices; in addition, the disproportionality index itself divides by a London average that includes each borough, so part of the negative association may be built into the outcome definition.

Editorial extensions

If this is right

  • If the coverage result holds beyond the simulation setting, researchers can report Bayesian credible intervals for high-dimensional semiparametric treatment effects without relying on asymptotic normal approximations.
  • The negative posterior mean implies that, within London, stop-and-search disproportionality for expressive crimes is lower in boroughs with larger Black population shares, a pattern the authors read as more aggressive policing of Black individuals in predominantly white areas.
  • The proposed algorithm is computationally about five times faster than the Gaussian-process benchmark (BDR-HD) in the n=50 simulation while matching its bias and RMSE, so it offers a scalable route for small-area Bayesian causal inference.
  • The framework directly extends the double machine learning toolkit to settings where policymakers hold informative priors, since the prior enters only through β and the nuisance functions remain nonparametric.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A leave-one-out recomputation of the DI, dividing each borough's rate by the London average excluding that borough, would directly test whether the negative coefficient is partly mechanical; the paper does not report that check.
  • Given n=33 boroughs and 31 covariates, the wide Lasso credible intervals that include zero are arguably the more honest uncertainty statement; the tighter random-forest intervals still depend on the same identifying assumption of no unmeasured confounding.
  • A natural out-of-sample prediction is that longitudinal changes in borough Black population share between Censuses should track decreases in the expressive-crime DI if the negative effect is causal; this is testable with future data.
  • The Monahan-Boos validation is performed under the simulation's known data-generating process; a misspecified nuisance model or an incorrectly specified moment condition could break the coverage guarantee, and the paper does not probe this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a Bayesian extension of double/debiased machine learning for the partially linear model Y=µ(X)+βD+U. The likelihood is replaced by a generalized empirical likelihood (EL, ETEL, Hellinger) subject to the Neyman moment condition, with K-fold sample splitting for nuisance estimation. The authors argue via Monahan-Boos H-statistic simulations that the resulting posterior has valid coverage, show simulation comparisons for binary and continuous treatments, and apply the method to 33 London boroughs with outcome the Black disproportionality index for expressive crimes and treatment the borough Black population share, reporting a negative posterior mean.

Significance. If the method's coverage calibration holds beyond the simulated DGP, this is a useful contribution: it gives applied researchers a Bayesian posterior for a semiparametric causal parameter while retaining DML's robustness to high-dimensional nuisance estimation, and the code is public. The empirical question is important and the negative association is policy-relevant. However, the empirical inference is currently undermined by the definition of the outcome, which appears to be mechanically coupled to the treatment, and by the very small number of boroughs relative to confounders.

major comments (4)
  1. [Section 2, Table 3] As written, the outcome DI_i is the borough rate of Black people in Stop and Search for expressive crime divided by the London rate. If 'rate' is stops per Black resident, then DI_i = (B_i/(d_i P_i)) / (Σ_j B_j / Σ_j d_j P_j), so the treatment variable d_i (percentage of Black population) appears in the denominator of the outcome. Under a colorblind null in which B_i/(d_iP_i) is constant, E[DI_i]=1, but small-d_i boroughs have enormous sampling noise, which mechanically produces a negative regression slope of DI on d_i. This is exactly the pattern in Table 4, and the observed maximum DI 12.58 in a borough with only 1.89% Black residents is consistent with the artifact. Please give the exact formula for DI; if it is the one above, redefine the outcome so that the treatment does not enter its construction, or provide a quantitative demonstration that the negative association is not a ratio artifact.
  2. [Section 5, Table 4] The application has n=33 boroughs and p=31 covariates; with two-fold sample splitting, each nuisance model is estimated on roughly 16 units. The posterior credible intervals are extremely sensitive to the ML estimator: Lasso intervals all include zero (-2.17,0.95; -2.21,0.99; -2.18,1.00), while random forest and neural network intervals exclude zero. The claim that 'all methods suggest a negative relationship' is based only on posterior means. The paper should report the instability across folds, add a spatial component or at least a spatial residual analysis, and show sensitivity to the informative prior N(0,2). As it stands, the empirical conclusion is not robust.
  3. [Section 3.5, Figure 3] The Monahan-Boos validation produces Kolmogorov-Smirnov p-values of 0.07 (ETEL) and 0.09 (EL) on 10,000 simulations. These are borderline rather than clear evidence of uniformity, so the statement that the method 'yields a valid posterior' overstates what is shown. Report coverage at several nominal levels (e.g., 50%, 80%, 95%) with Monte Carlo standard errors, and restrict the claim to the DGP studied; no theoretical proof of general posterior validity is given.
  4. [Section 3.1] The causal estimand β=E[∂Y(d)/∂d] is not well matched to the DI outcome because DI_i is defined relative to the London-wide average rate, which itself depends on all borough treatments. Changing d_i for one borough changes the denominator of DI_j for every j, so the stable-unit-treatment-value assumption is strained even before considering the denominator coupling in the first comment. Please clarify what hypothetical intervention the estimand corresponds to, or reframe the analysis as descriptive.
minor comments (5)
  1. [Throughout] The manuscript contains several typographical errors, including 'dispropotionality', 'Alogrithm', 'the the empirical Cressie-Read statistic', 'counfounders', and 'may raises'. A careful proofread is needed.
  2. [Algorithm 2, Step 7] The algorithm samples β^(j) from a product that uses p_i^(j) computed from β^(j-1); please clarify whether this is a Metropolis-within-Gibbs step and report acceptance rates or a convergence diagnostic.
  3. [Figure 2] The histograms appear to be of posterior means, while the red curve is labeled the standard normal density; the comparison target should be stated explicitly.
  4. [Section 2, Table 3] The observed DI values range from 1.66 to 12.58, with no borough below 1, even though the text describes DI<1 as below expectation; please discuss whether the DI is restricted by construction and what this implies for the interpretation.
  5. [Discussion] The sentence 'police are more aggressively policing Black individuals in areas that are predominantly white' is a causal claim that goes beyond the estimated association; consider softening or removing it.

Circularity Check

1 steps flagged · score 6.0 of 10

The application's negative DI-Black-share relationship is partly built into the outcome: the DI is a rate whose denominator is the treatment (Black population share).

  1. self definitional [Section 2 (outcome and treatment definitions); Section 5, Table 4]
    "Therefore, we define the outcome variable as the disproportionality index (DI) for expressive crime targeting Black people in each Borough (total 33 Boroughs) and it can be calculated as: DIi = rate of Black people in Stop and Search for expressive crime in Boroughi / rate of average Black people in Stop and Search for expressive crime in London. ... To evaluate the effect of disproportionality, we use the percentage of Black people in each borough as the treatment variable."

    The DI is a rate ratio. With London's Black stop share around 38% and Black population share 13.5%, the only reading consistent with DI values up to 12.58 (Table 3) is that the borough rate is Black expressive stops divided by the borough's Black population, so the treatment D_i (Black population share) is the denominator of Y_i: Y_i = [B_i/(D_i P_i)] / [sum_j B_j/(sum_j D_j P_j)]. Hence log Y_i contains -log D_i by construction; small-D boroughs have mechanically large, high-variance DI values. Regressing this outcome on D therefore yields a negative coefficient even under a colorblind null, exactly the sign reported in Table 4 (-0.55 to -0.31).

full rationale

The methodological development is not circular: the empirical likelihood weights are solved from the Neyman moment condition, the sample-splitting algorithm is standard DML, and the posterior-coverage claim is checked by the Monahan-Boos H-statistic simulation, an external criterion. The citations to the authors' own ETEL work (Luo et al., 2023) are background and not load-bearing. However, the paper's headline empirical finding is partially circular: the DI outcome is defined using the treatment (Black population share) in its denominator, so the negative posterior means in Section 5 are in part a mathematical property of the outcome construction. The Discussion flags residual spatial autocorrelation but not this denominator coupling. Because the method itself stands independently but the application's central conclusion is partially forced by definition, the circularity score is 6 rather than 0-2.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

Central claim rests on untestable causal assumptions at borough level, a hand-chosen informative prior, and the DI outcome definition. No new entities are postulated.

free parameters (1)
  • Prior hyperparameters for treatment effect beta in application = N(0,2)
    Authors assign informative prior N(0,2) in Section 5, reflecting an initial assumption of no effect; posterior intervals are sensitive to this choice, especially in small samples.
assumptions (5)
  • domain assumption No unmeasured confounding (NUC): given X, treatment D is independent of potential outcomes
    Section 3.1 lists NUC among core causal assumptions; with only 31 borough-level covariates and no sensitivity analysis, this is untestable and load-bearing for the causal interpretation.
  • domain assumption Consistency, SUTVA, and positivity
    Section 3.1 invokes these standard causal assumptions; SUTVA may be questionable at borough level due to spatial spillovers.
  • standard math Regularity conditions for double machine learning: nuisance estimates converge fast enough and Neyman orthogonality holds
    Section 3.5 relies on Chernozhukov et al. (2018) asymptotic normality; the paper does not verify these rates for n=33 and p=31.
  • standard math Existence of empirical likelihood weights: zero lies in the convex hull of the moment function values
    Section 3.3 cites Baggerly (1998) for uniqueness; in small samples with many moment constraints this may fail.
  • domain assumption The DI outcome is a valid measure of disproportionality
    Section 2 defines DI as borough Black stop rate divided by London average; validity depends on the rate definition and on the denominator not inducing mechanical correlation with the treatment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A scalable Bayesian double machine learning framework, with application to racial disproportionality assessment." pith.science (2026). https://pith.science/paper/WBJFIIUS

@misc{pith2026250207695,
  author       = {Pith},
  title        = {Pith review of: A scalable Bayesian double machine learning framework, with application to racial disproportionality assessment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WBJFIIUS}},
  note         = {Machine review of arXiv:2502.07695}
}
read the original abstract

Racial disproportionality in stop and search practices elicits substantial concerns about its societal and behavioral impacts. In London, Black individuals are about four times more likely to be stopped and searched than White individuals. Using data on stop and search events in London from January 2019 to December 2023, this paper aims to investigate disproportionality in the volume of stops for expressive crimes involving Black individuals compared to other ethnicities. We employ a semi-parametric partially linear structural regression method and introduce a Bayesian empirical likelihood procedure combined with double machine learning techniques to control for high-dimensional confounding and to accommodate strong prior assumptions. In addition, we show that the proposed procedure yields a valid posterior in terms of coverage. Applying this approach to the stop and search dataset, we find that racial disproportionality aimed at the Black community may be influenced by the borough racial composition when focusing on expressive crimes.

Figures

Figures reproduced from arXiv: 2502.07695 by the authors.

Figure 1
Figure 1. DI for Black People in Stop & Search for Expressive Reasons [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Comparison between double ML (DML) and Bayesian empirical likelihood methods using full-sample [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Histograms for H statistics using the Bayesian generalized empirical likelihood. P-values represents the Kolmogorov–Smirnov test for uniformity of H. 4 Simulation In this section, we examine the performance of the proposed Bayesian method described in the previous section. We consider the following models for our simulation studies. • EL, ETEL, HD: The Bayesian generalized empirical likelihood with λ = 0, −1, −1/2 r… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Stop & Search Subjects Ethnic Groups Proportions [PITH_FULL_IMAGE:figures/full_fig_p023_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 33 canonical work pages

  1. [1]

    Papadogeorgou, and F

    Antonelli, J., G. Papadogeorgou, and F. Dominici (2022). Causal inference in high dimensions: A marriage between B ayesian modeling and good frequentist properties. Biometrics\/ 78\/ (1), 100--114

  2. [2]

    Baggerly, K. A. (1998). Empirical likelihood as a goodness-of-fit measure. Biometrika\/ 85\/ (3), 535--547

  3. [3]

    Bowling, B. and C. Phillips (2007). Disproportionate and discriminatory: Reviewing the evidence on police stop and search. The Modern Law Review\/ 70\/ (6), 936--961

  4. [4]

    Chamberlain, G. and G. W. Imbens (2003). Nonparametric applications of B ayesian inference. Journal of Business & Economic Statistics\/ 21\/ (1), 12--18

  5. [5]

    Chetverikov, M

    Chernozhukov, V., D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal\/ 21\/ (1), C1--C68

  6. [6]

    Chernozhukov, V. and H. Hong (2003). An MCMC approach to classical estimation. Journal of Econometrics\/ 115\/ (2), 293--346

  7. [7]

    Shin, and A

    Chib, S., M. Shin, and A. Simoni (2018). Bayesian estimation and comparison of moment condition models. Journal of the American Statistical Association\/ 113\/ (524), 1656--1668

  8. [8]

    Gaebler, J., W. Cai, G. Basse, R. Shroff, S. Goel, and J. Hill (2022). A causal framework for observational studies of discrimination. Statistics and Public Policy\/ 9\/ (1), 26--48

Show all 41 references
  1. [9]

    Graham, D. J., E. J. McCoy, and D. A. Stephens (2016). Approximate B ayesian inference for doubly robust estimation. Bayesian Analysis\/ 11\/ (1), 47--69

  2. [10]

    Hahn, J. (1998). On the role of the propensity score in efficient semiparametric estimation of average treatment effects. Econometrica\/ , 315--331

  3. [11]

    State of policing: The annual assessment of policing in england and wales 2022

    HM Inspectorate of Constabulary and Fire & Rescue Services (2022). State of policing: The annual assessment of policing in england and wales 2022. HMICFRS\/

  4. [12]

    Beck, and J

    Huang, Z., B. Beck, and J. Antonelli (2024). Causal inference and racial bias in policing: New estimands and the importance of mobility data. arXiv:2409.08059\/

  5. [13]

    Johnson, and R

    Imbens, G., P. Johnson, and R. H. Spady (1998). Information theoretic approaches to inference in moment condition models. Econometrica\/ 66\/ (2), 333--357

  6. [14]

    Corbett-Davies, J

    Jung, J., S. Corbett-Davies, J. D. Gaebler, R. Shroff, and S. Goel (2018). Mitigating included-and omitted-variable bias in estimates of disparate impact. arXiv:1809.05651\/

  7. [15]

    Kaplan, D. and J. Chen (2012). A two-step B ayesian approach for propensity score analysis: Simulations and case study. Psychometrika\/ 77\/ (3), 581--609

  8. [16]

    Otsu, and K

    Kitamura, Y., T. Otsu, and K. Evdokimov (2013). Robustness, infinitesimal neighborhoods, and moment restrictions. Econometrica\/ 81\/ (3), 1185--1201

  9. [17]

    Lowe, and J

    Knox, D., W. Lowe, and J. Mummolo (2020). Administrative records mask racially biased policing. American Political Science Review\/ 114\/ (3), 619--637

  10. [18]

    Lee, M.-J. (2018). Simple least squares estimator for treatment effects using propensity score residuals. Biometrika\/ 105\/ (1), 149--164

  11. [19]

    Ding, and F

    Li, F., P. Ding, and F. Mealli (2023). Bayesian causal inference: A critical review. Philosophical Transactions of the Royal Society A\/ 381\/ (2247), 20220153

  12. [20]

    Saarela, B

    Liu, K., O. Saarela, B. M. Feldman, and E. Pullenayegum (2020). Estimation of causal effects with repeatedly measured outcomes in a B ayesian framework. Statistical Methods in Medical Research\/ 29\/ (9), 2507--2519

  13. [21]

    Luo, Y., D. J. Graham, and E. J. McCoy (2023). Semiparametric B ayesian doubly robust causal estimation. Journal of Statistical Planning and Inference\/ 225 , 171--187

  14. [22]

    Luo, Y., D. A. Stephens, D. J. Graham, and E. J. McCoy (2023). Assessing the validity of B ayesian inference using loss functions. arXiv:2103.04086\/

  15. [23]

    McCandless, L. C., I. J. Douglas, S. J. Evans, and L. Smeeth (2010). Cutting feedback in B ayesian regression adjustment for the propensity score. The International Journal of Biostatistics\/ 6\/ (2), 16

  16. [24]

    Meng, Y. (2017). Profiling minorities: Police stop-and-search practices in contemporary London . Human Geographies\/ 11\/ (1), 5--22

  17. [25]

    Millner, N. (2020). As the drone flies: Configuring a vertical politics of urban policing. Political Geography\/ 80 , 102163

  18. [26]

    Monahan, J. F. and D. D. Boos (1992). Proper likelihoods for B ayesian analysis. Biometrika\/ 79\/ (2), 271--278

  19. [27]

    Disproportionality board data pack

    MOPAC (2023). Disproportionality board data pack. Disproportionality Board Data Pack 2023\/

  20. [28]

    Newey, W. K. and R. J. Smith (2004). Higher order properties of GMM and generalized empirical likelihood estimators . Econometrica\/ 72\/ (1), 219--255

  21. [29]

    Oberwittler, D. and S. Roché (2022). How institutional contexts shape police-adolescent relations in France and Germany: Spatial and social disparities . Policing and Society\/ 32\/ (3), 378--410

  22. [30]

    Owen, A. B. (2001). Empirical likelihood . CRC press

  23. [31]

    Qin, J. and J. Lawless (1994). Empirical likelihood and general estimating equations. The Annals of Statistics\/ 22\/ (1), 300--325

  24. [32]

    Read, T. R. and N. A. Cressie (2012). Goodness-of-fit statistics for discrete multivariate data . Springer Science & Business Media

  25. [33]

    Robinson, P. M. (1988). Root- n -consistent semiparametric regression. Econometrica\/ 56\/ (4), 931--954

  26. [34]

    Rosenbaum, P. R. and D. B. Rubin (1983). The central role of the propensity score in observational studies for causal effects. Biometrika\/ 70\/ (1), 41--55

  27. [35]

    Rubin, D. B. (1981). The Bayesian bootstrap . The Annals of Statistics\/ 9\/ (1), 130--134

  28. [36]

    Schennach, S. M. (2007). Point estimation with exponentially tilted empirical likelihood. The Annals of Statistics\/ 35\/ (2), 634--672

  29. [37]

    Suss, J. and T. Oliveira (2023). Economic inequality and the spatial distribution of stop-and-search in london. British Journal of Criminology\/ 63\/ (4), 828--847

  30. [38]

    Betancourt, D

    Talts, S., M. Betancourt, D. Simpson, A. Vehtari, and A. Gelman (2018). Validating B ayesian inference algorithms with simulation-based calibration. arXiv:1804.06788\/

  31. [39]

    Quinton, and B

    Tiratelli, M., P. Quinton, and B. Bradford (2018). Does stop-and-search deter crime? Evidence from ten years of London-wide data . The British Journal of Criminology\/ 58\/ (5), 1212--1231

  32. [40]

    Yiu, A., R. J. B. Goudie, and B. D. M. Tom (2020). Inference under unequal probability sampling with the B ayesian exponentially tilted empirical likelihood. Biometrika\/ 107\/ (4), 857--873

  33. [41]

    Zhao, Q., L. J. Keele, D. S. Small, and M. M. Joffe (2022). A note on posttreatment selection in studying racial discrimination in policing. American Political Science Review\/ 116\/ (1), 337--350

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.