Pith. sign in

REVIEW 4 major objections 11 minor 55 references

Proxy estimates only become trustworthy for primary decisions when checked and corrected at four nested levels, from sample representativity to cross-domain stability.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 15:51 UTC pith:2J2PJR77

load-bearing objection Useful four-level diagnose-then-adjust playbook plus shipped package; theory is mostly standard bridges, and the post-selection CI story is thinner than the abstract promises. the 4 major comments →

arxiv 2607.24401 v1 pith:2J2PJR77 submitted 2026-07-27 stat.ML cs.LG

proxymate: Diagnosis and Adjustment of Proxy Estimates for Reliable Inference

Alexandra N. M. Darmon , Deeksha Sinha , Steve Wilkins-Reeves , Caner Gocmen This is my paper
classification stat.ML cs.LG
keywords proxy estimatessurrogate endpointsprediction-powered inferencemeta-analysiscalibrationcovariate shiftimportance weightingtransportability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Practitioners often decide on a slow, rare, or hard-to-measure primary outcome by analyzing a cheaper proxy—a short-term signal, a model score, or a surrogate endpoint. The paper’s claim is that valid inference on the proxy does not automatically transfer to the primary: unit-level accuracy can look fine while the labeled sample is skewed, individual errors can compound into biased aggregates, and a relationship that holds in one setting can fail in the next. proxymate answers this with a four-level diagnose-then-adjust workflow: first ask whether the validation sample represents the target population; then check unit measurement quality; then check whether aggregate point estimates and intervals agree; then check whether the proxy–primary map is stable across domains. Each failure mode is mapped to a specific correction—importance weighting, calibration, prediction-powered inference, or cross-domain regression calibration—or to an explicit decision to reject the proxy. In production the same structure validated proxies that unlocked thousands of experiments a year and rejected others that would have trained biased models or misestimated prevalence.

Core claim

The paper establishes that reliable primary-targeted inference from proxies requires a staged diagnostic layer before any adjustment is applied. Organizing checks into Representativity, Unit, Estimate, and Domain levels, and pairing each detected failure with the adjustment whose assumptions that failure tests, closes the gap between “the proxy looks correlated” and “the primary estimand’s confidence interval is calibrated.” Blind adjustment of an already-adequate proxy can raise mean squared error; skipping representativity can leave population bias intact even after unit or estimate corrections.

What carries the argument

The diagnostic-to-adjustment bridge: a four-level map (Table 1; Bridges B.1–B.7) that routes each failure—labeling gap, unit bias, miscalibration, aggregate bias, cross-domain shift, high heterogeneity—to a matching correction (importance weights, additive shift, Platt/isotonic/multicalibration, PPI, regression calibration, residual bias inflation) or to “skip adjustment / replace proxy,” with Theorem 1 proving that unnecessary PPI strictly inflates MSE when bias is already negligible.

Load-bearing premise

Labeled units must be missing only as a function of observed covariates, and the proxy–primary link across historical settings must behave like a stable random-effects line that can be trusted in the next setting.

What would settle it

Take a target domain where the true primary mean is known: if after the full pipeline the corrected interval still misses the primary at far from nominal rate, or if importance weighting plus PPI still leave large population bias when missingness depends on the outcome itself given covariates, the bridge’s guarantees fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Teams can gate proxy use by level: pass Unit and Estimate for direction-only decisions; require Domain stability before transporting magnitude.
  • When diagnostics show near-zero aggregate bias, adjustment should be skipped—applying PPI then only adds variance.
  • Severe representativity failures (unstable weights, low effective sample size) correctly force rejection or narrower scope rather than a false population claim.
  • The same modular checks support experimentation surrogates, prevalence classifiers, and monitoring proxies without a treatment-specific surrogate criterion.
  • An open package turns the bridge into repeatable reports so successive proxy versions can be revalidated on the same infrastructure.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Treating representativity as a hard first gate would change many “calibrated classifier” prevalence pipelines that today run PPI or recalibration on review-queue labels alone.
  • The skip-adjustment rule suggests experiment platforms should expose a bias-versus-labeled-noise threshold before auto-applying post-prediction corrections.
  • Segment-wise Unit/Estimate checks aggregated as a Domain meta-analysis offer a practical path to fairness-style audits without a separate fairness stack.
  • Multi-outcome or sequential (confidence-sequence) extensions the authors flag would let the same bridge score candidate proxy portfolios rather than one proxy at a time.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 11 minor

Summary. The paper presents proxymate, a four-level framework and open-source Python package for validating and adjusting proxy outcomes used in place of slow, rare, or expensive primary outcomes. The levels — Representativity (is the labeled validation sample representative of the target population?), Unit (does the proxy track the primary at the individual level?), Estimate (does the proxy give the same aggregate answer within a domain?), and Domain (is the proxy–primary relationship stable across domains?) — each pair diagnostics with targeted adjustments (importance weighting, additive bias correction, Platt/isotonic calibration, PPI, regression calibration, estimate-level random-effects adjustment). Appendix B formalizes the diagnostic-to-adjustment mapping as Bridges B.1–B.7, mostly restatements of standard identities (Horvitz–Thompson/IW under covariate shift, calibration-map calibration, PPI unbiasedness and split-sample variance, AUC invariance under monotone transforms), plus one simple new result, Theorem 1, showing PPI has lower MSE than the raw proxy iff the proxy bias exceeds σ_Δ/√n. The empirical contribution is a report of six production applications at Meta, with two case studies: a 60-day nonpayment proxy validated across 22 experiments for direction-based decisions, and a scam-prevalence setting where all three candidate proxies were rejected after representativity-aware reweighting.

Significance. If the workflow is as effective as reported, the practical significance is substantial: the paper documents bidirectional value (four validated proxies enabling ~7,500 experiments/year; two rejections preventing deployment of a model trained on biased outcomes), ships an open-source package with 46 modular checks and a validation planner, and provides falsifiable, checkable formal statements — Theorem 1's break-even condition |φ| > σ_Δ/√n is a genuinely useful and, to my knowledge, not-previously-stated guide for when PPI helps versus hurts, and the bridge propositions, while standard, are cleanly tied to specific diagnostics. The negative results (Case Study B, where unweighted unit-level checks look fine but IW reweighting drops sensitivity to 17–32%) are a credible and instructive demonstration of the framework's ordering argument. The conceptual contribution — a diagnostic layer that decides *whether and which* adjustment to apply — fills a real gap between the adjustment literature (PPI, calibration) and deployment practice.

major comments (4)
  1. [§3, Table 1, Theorem 1 (Bridge B.5), Appendix C.3] The guarantees in Table 1 and the bridge propositions are stated for fixed procedures applied conditionally on assumptions, but the deployed pipeline applies each adjustment conditionally on diagnostics *accepting* those assumptions — a post-selection (pre-test) procedure whose unconditional coverage is never analyzed or simulated. The cleanest instance is the paper's own Theorem 1: the break-even rule |φ| > σ_Δ/√n compares two fixed estimators, yet operationally the Estimate Level gates on φ̂ and bootstrap-coverage computed from the same labeled data PPI then consumes, so the reported estimator is a data-dependent mixture θ̂*·1{pass} + θ̂_PPI·1{fail}. Standard pre-test results (Leeb–Pötscher and successors) show coverage distortion is maximal near the indifference region |φ| ≈ σ_Δ/√n, where the gate flips with ~50% probability and the 'pass' branch retains bias of 1–2 SE while reporting
  2. [§4.3 (Case Study A) vs. §3.4.1/Table 1] The deployed verdict in Case Study A is not licensed by the framework's own bridge. The Domain Level diagnostics fail on prediction accuracy (LOO MAE = 0.84 against a median per-arm SE of 0.41, i.e., residual error roughly twice the sampling noise), and Table 1 maps LOO-error failure to 'Model misspecification → warn; report bounds.' Instead, the case study concludes the proxy is 'safe for direction-based decision making' based on 85% directional alignment (23/27 arms), I² ≈ 0, and γ₁ = 0.98. This may well be the right practical call, but the framework as stated has no formal bridge for a direction-only decision regime: there is no proposition connecting directional-alignment rates to sign-decision error rates, no treatment of the multiple-comparison character of 27 arm-level sign checks, and no threshold for when sign agreement is 'enough.' As written, the paper's flagship positive case
  3. [§3.5, Appendix A (Tables 3–6), Appendix C.2] The framework runs up to 46 checks with per-check pass/fail thresholds (SMD < 0.1, I² < 25%, ECE cutoffs, etc.), but provides no guidance on multiplicity: how many checks, at what levels, must fail to reject a proxy, and what the composite false-rejection rate of the full battery is under a good proxy. With 27 Unit-Level checks alone, some will fail by chance at any per-check level, and the paper's own Case Study A reports Unit-Level precision checks failing on a proxy that was ultimately adopted. The validation planner (Table 8) routes challenges to check subsets but does not state how subset selection interacts with error rates, and the tightening/loosening advice in §3.5 ('a team making a high-stakes policy decision should tighten thresholds') implicitly trades errors against each other without any accounting. A short subsection defining the composite decision rule (e.g., which checks
  4. [§3.4, Bridge B.6, Table 1] The Domain Level's coverage guarantees rest entirely on reference [52] (Wilkins-Reeves, Darmon, Sinha), cited as 'Submitted to KDD '26… arXiv preprint (forthcoming)'. Two of the three load-bearing Domain-Level rows in Table 1 (residual-bias estimate-level adjustment; regression-calibration coverage; the K ≲ 30 bootstrap recommendation in §3.4.3) trace to this unrefereed, currently unavailable manuscript by three of the four present authors. The regression-calibration row in Table 1 cites only [52] for its 'Guarantee,' and Bridge B.6's coverage statement ('asymptotic coverage stated in [52]') cannot be verified by a reader. The asymptotic regime stated in B.6 (K → ∞ with max_k (σ*_k)² → 0) also deserves more care: with the small K typical here (Case Study A has K = 27 arms from 22 experiments, and arms within an experiment are not independent domains), the finite-K caveat in §3.4.3 is doi
minor comments (11)
  1. [§3.1] Typo: 'this may also be generalized to where Y* also has it's own missingness indicator' — 'its', and the sentence is garbled ('One approach leverages a covariate shift assumption, involves (Y,Y*)|X=x regardless of whether… are observed' is missing a verb phrase).
  2. [§3.4.2] Notational collision: the between-domain variance of the bias φ_k is denoted γ² while the calibration slope is γ₁; readers will repeatedly misread γ² as the squared slope. Consider σ²_φ or υ².
  3. [Bridge B.6] The predictive interval is introduced as 'stated as in .' — an empty citation. The formula for I^cal_{K,1−α} should be written out explicitly.
  4. [Table 1] The 'Guarantee' column cites external papers for PPI ([2]) but not Bridge B.4, and cites nothing internal for regression calibration beyond [52]; for self-containedness, cite the bridge propositions alongside external references.
  5. [Table 6 / §3.4.1] The Durbin–Watson residual-autocorrelation check presupposes an ordering of domains (temporal?), which is never defined; domains are defined as arbitrary distinct data-generating processes in §2. Clarify when this check is applicable.
  6. [Table 4] MAPE is listed as an accuracy check with no discussion of its behavior when Y_i ≈ 0 (a common case for the revenue/prevalence outcomes used in the case studies); at minimum note the instability or recommend a symmetric variant.
  7. [§3.1.2] The 'Gap is too severe' outcome references ESS/weight-extremity cutoffs that are never given default values (unlike SMD < 0.1 and I² < 25%, which are). State the defaults used in the package, or explicitly mark this verdict as practitioner-set.
  8. [§4.2, footnote 2] The confidentiality perturbation of all reported counts and statistics limits reproducibility of the case studies; please state whether qualitative pass/fail patterns (e.g., the ESS failure in Case B, the LOO failure in Case A) are preserved exactly, since these drive the conclusions.
  9. [Appendix C.1, Table 7] Case 3 (aggregate summaries only, Domain Level executable) deserves a pointer to §3.4.2's estimate-level adjustment as the intended use; as written the table's utility is unclear without that cross-reference.
  10. [Figure 1] The workflow figure is referenced as summarizing both the four-level framework and the four data-input cases (§C.1), and is doing too much; consider splitting or adding a legend mapping colors/shapes to levels.
  11. [References] Several in-text citation anchors appear incomplete beyond [52] (e.g., [38] Sarig et al. 'arXiv preprint' with no number; [52] 'forthcoming'). Please supply stable identifiers where available.

Circularity Check

1 steps flagged

No material circularity: bridges restate standard identities; only minor same-author citation for one Domain-Level module.

specific steps
  1. self citation load bearing [§3.4.2 Adjustments; Bridge B.6; Ref. [52]]
    "Wilkins-Reeves et al. [52] show that under the random-effects meta-analytic model, the calibrated confidence interval achieves nominal coverage for the primary estimand θ_K as K→∞... proxymate integrates both regression calibration and estimate-level adjustment as Domain Level adjustments. ... [52] Steve Wilkins-Reeves, Alexandra Darmon, and Deeksha Sinha. 2026. Estimate-Level Adjustment..."

    Nominal coverage and the residual-bias random-effect form for one Domain-Level correction are justified primarily by concurrent same-author work [52], not by an external verified theorem. This is only modular support for cross-domain interval inflation/calibration—not the four-level organization, IW/PPI/Platt bridges, or Theorem 1—so it is minor self-citation, not a full self-citation chain forcing the paper’s main claim.

full rationale

The paper’s load-bearing math is standard and non-circular. Bridge B.1 is the usual IW identity under covariate shift; B.2–B.3 are linearity and calibration-map facts; B.4–B.5 give PPI unbiasedness/variance and the MSE comparison in Theorem 1 (MSE(PPI)<MSE(proxy) iff |φ|>σ_Δ/√n), which compares two fixed estimators from stated bias/variance terms rather than defining success as a fitted objective. Domain-Level regression calibration is classical random-effects meta-regression (DerSimonian–Laird / REML), with diagnostics (R², I², τ², LOO) that are separable from the target estimand. The authors explicitly disclaim novelty of the individual checks and present organization plus a diagnose-then-adjust map as the contribution. Empirical case studies report pass/fail and LOO error against held-out domains; they do not rename a fit as a first-principles prediction. The only mild circularity-adjacent item is reliance on Wilkins-Reeves et al. [52] (overlapping authors, forthcoming) for one Domain-Level residual-bias adjustment and its coverage sketch—modular and not required for Representativity/Unit/Estimate or for Theorem 1—so this does not collapse the central framework claim. Post-selection coverage of the gated pipeline is a correctness gap, not circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 7 axioms · 2 invented entities

The central operational claim rests on classical sampling and meta-analysis assumptions plus software engineering choices, not on new physical entities. Guarantees for each adjustment are conditional on diagnostics approximating those assumptions (covariate shift, additive bias, calibration function class, representative labeled set for PPI, random-effects transport). Free parameters are mostly conventional thresholds and planner defaults rather than fitted universal constants. No new particles or forces; the ‘four levels’ are an organizing taxonomy.

free parameters (5)
  • SMD representativity threshold (default ~0.1) = 0.1 (conventional default)
    Pass/fail for covariate balance follows Austin-style convention and configurable defaults (Appendix C.2); choice changes whether IW is triggered.
  • I² heterogeneity threshold (default ~25%) = 25% (conventional default)
    Higgins-style low-heterogeneity cutoff used as a Domain Level pass criterion; partly literature, partly practitioner tuning.
  • Minimum historical domains for Domain Level (flags <5; MoM stability ~10; bootstrap guidance K≲30) = K<5 unreliable; prefer bootstrap if K≲30
    Finite-K rules of thumb gate whether meta-analytic transport diagnostics/adjustments are trusted (Section 3.4.3).
  • ESS / weight-extremity cutoffs for ‘gap too severe’
    Determines fallback vs IW adjustment at Representativity Level; not uniquely determined by theory in the paper.
  • Per-check practitioner pass thresholds (ECE, rel. abs. error, LOO error, etc.)
    Configurable defaults mix conventions and Meta experience; high-stakes vs exploratory users are told to tighten/relax (Section 3.5).
axioms (7)
  • domain assumption Selection on observables / covariate shift: P(Y,Y*|X)=Q(Y,Y*|X) with positivity, so IW recovers E_P[g(O)] from labeled Q.
    Bridge B.1 / Representativity Level; fails if labels depend on Y given X (e.g., pure outcome-dependent review).
  • domain assumption Unit-level or mean additive bias E[Y*-Y]=b is the failure mode corrected by subtracting b for mean estimands.
    Bridge B.2; sufficient for means, not for arbitrary nonlinear functionals without further structure.
  • domain assumption Labeled set for PPI is representative (or IW-corrected) of the target joint law of (Y,Y*), yielding unbiased θ̂_PPI.
    Bridge B.4 Proposition 4; Case Study B shows violating this before PPI misstates prevalence.
  • domain assumption Random-effects calibration across domains: θ_k=γ0+γ1 θ*_k+ε_k, ε_k iid N(0,τ²), with inverse-variance meta-regression (not naive OLS) for consistency as K→∞.
    Section 3.4 and Bridge B.6; load-bearing for Domain Level transport intervals.
  • standard math Strictly monotone post-hoc maps cannot improve rank discrimination (AUC/Spearman/Kendall invariance).
    Bridge B.7 Proposition 6; justifies proxy replacement when discrimination fails.
  • standard math When proxy bias ϕ=0, PPI strictly increases MSE by σ²_Δ/n (sample-splitting / negligible covariance regime).
    Theorem 1; justifies the no-adjustment gate at Estimate Level.
  • ad hoc to paper Domains are distinct data-generating processes; segments are within-domain covariate slices—and historical k=1..K-1 are exchangeable enough for meta-analysis toward target K.
    Section 2 definitional split driving the four-level architecture; transport validity depends on this modeling choice.
invented entities (2)
  • Four-level proxymate taxonomy (Representativity / Unit / Estimate / Domain) no independent evidence
    purpose: Organize diagnostics and pair each failure mode with an adjustment family and software module.
    Not a physical entity; an engineering ontology. Independent value is whether the split improves decisions versus flatter checklists—supported by case narratives, not a unique theorem that four is necessary.
  • Validation planner routing table (proxy challenges → check subsets) no independent evidence
    purpose: Lower barrier for non-experts by mapping long maturation, restricted labels, low detectability, low sensitivity to recommended checks.
    Convenience layer over the same diagnostics; defaults are experiential rather than uniquely identified.

pith-pipeline@v1.2.0-grok45-kimik3 · 28508 in / 4391 out tokens · 83994 ms · 2026-07-31T15:51:52.094686+00:00 · methodology

0 comments
read the original abstract

Proxy outcomes (such as short-term behavioral signals, model predictions, or surrogate endpoints) are frequently used in place of primary outcomes that are too slow to mature, rare, or challenging to measure directly. But valid inference on a proxy does not guarantee valid inference on the primary estimate as proxy-based estimates can be systematically biased in ways that are difficult to predict, leading to improperly calibrated confidence intervals. We present proxymate, a framework and open-source Python package for proxy validation and adjustment. proxymate organizes into four levels: The Representativity Level (population validity), the Unit Level (measurement quality), the Estimate Level (decision validity), and the Domain Level (cross-domain transportability). Within each level, proxymate provides diagnostic checks, and targeted adjustment strategies that map specific failures to appropriate corrections. At Meta, proxymate has been adopted by many different use cases, spanning experimentation, prevalence estimation, and monitoring use cases, all facing different proxy challenges (limited human review time, long maturation window of outcomes, low detectability) and showcasing the modularity of the framework. Across all products, proxymate assessed and corrected millions of proxy, primary unit comparisons. It has facilitated launches across multiple work streams including enabling quick decision making on thousands of experiments.

Figures

Figures reproduced from arXiv: 2607.24401 by Alexandra N. M. Darmon, Caner Gocmen, Deeksha Sinha, Steve Wilkins-Reeves.

Figure 1
Figure 1. Figure 1: The proxymate framework overview. Diagnosis and Correction. Blind adjustment can inflate MSE (The￾orem 1 in Appendix B) or amplify bias under model misspecification; the diagnostic layer selects an adjustment only when its assump￾tions hold [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Case Study A: Scatter plot capturing the [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 6 linked inside Pith

  1. [1]

    Angelopoulos and Stephen Bates

    Anastasios N. Angelopoulos and Stephen Bates. 2023. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.Founda- tions and Trends in Machine Learning(2023)

    2023

  2. [2]

    Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I

    Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zrnic. 2023. Prediction-Powered Inference.Science382, 6671 (2023), 669–674

    2023

  3. [3]

    Angelopoulos, John C

    Anastasios N. Angelopoulos, John C. Duchi, and Tijana Zrnic. 2023. PPI++: Efficient Prediction-Powered Inference.arXiv preprint arXiv:2311.01453(2023)

    Pith/arXiv arXiv 2023

  4. [4]

    Imbens, and Hyunseung Kang

    Susan Athey, Raj Chetty, Guido W. Imbens, and Hyunseung Kang. 2019.The Surrogate Index: Combining Short-Term Proxies to Estimate Long-Term Treatment Effects More Rapidly and Precisely. Working Paper. NBER

    2019

  5. [5]

    Peter C. Austin. 2011. An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies.Multivariate Behavioral Research46, 3 (2011), 399–424

    2011

  6. [6]

    Aurélien Bibaut, Winston Chou, Simon Ejdemyr, and Nathan Kallus. 2024. Learn- ing the Covariance of Treatment Effects Across Many Weak Experiments. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 153–162. proxymate: Diagnosis and Adjustment of Proxy Estimates for Reliable Inference Conference’17, July 2017, Washing...

    2024

  7. [7]

    Martin Bland and Douglas G

    J. Martin Bland and Douglas G. Altman. 1986. Statistical Methods for Assessing Agreement Between Two Methods of Clinical Measurement.The Lancet327, 8476 (1986), 307–310

    1986

  8. [8]

    Glenn W. Brier. 1950. Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review78, 1 (1950), 1–3

    1950

  9. [9]

    Marc Buyse and Geert Molenberghs. 1998. Criteria for the Validation of Surrogate Endpoints in Randomized Experiments.Biometrics54, 4 (1998), 1014–1029

    1998

  10. [10]

    Marc Buyse, Geert Molenberghs, Tomasz Burzykowski, Didier Renard, and He- lena Geys. 2000. The Validation of Surrogate Endpoints in Meta-Analyses of Randomized Experiments.Biostatistics1, 1 (2000), 49–67

    2000

  11. [11]

    Carroll, David Ruppert, Leonard A

    Raymond J. Carroll, David Ruppert, Leonard A. Stefanski, and Ciprian M. Crainiceanu. 2006.Measurement Error in Nonlinear Models: A Modern Perspective (2nd ed.). Chapman and Hall/CRC

    2006

  12. [12]

    Xingran Chen, Tyler McCormick, Bhramar Mukherjee, and Zhenke Wu. 2025. A Unified Framework for Inference with General Missingness Patterns and Machine Learning Imputation.arXiv preprint arXiv:2508.15162(2025)

    arXiv 2025

  13. [13]

    William G. Cochran. 1954. The Combination of Estimates from Different Experi- ments.Biometrics10, 1 (1954), 101–129

    1954

  14. [14]

    Rebecca DerSimonian and Nan Laird. 1986. Meta-Analysis in Clinical Trials. Controlled Clinical Trials7, 3 (1986), 177–188

    1986

  15. [15]

    George Forman. 2008. Quantifying Counts and Costs via Classification.Data Mining and Knowledge Discovery17, 2 (2008), 164–206

    2008

  16. [16]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. InProceedings of the 34th International Conference on Machine Learning (ICML). 1321–1330

    2017

  17. [17]

    Somit Gupta, Liudmila Ulanova, Siddharth Bhardwaj, Pavel Dmitriev, Peter Raff, and Aleksander Fabijan. 2019. Top Challenges from the First Practical Online Controlled Experiments Summit.ACM SIGKDD Explorations Newsletter21, 1 (2019), 20–35

    2019

  18. [18]

    Hanley and Barbara J

    James A. Hanley and Barbara J. McNeil. 1982. The Meaning and Use of the Area Under a Receiver Operating Characteristic (ROC) Curve.Radiology143, 1 (1982), 29–36

    1982

  19. [19]

    Julian P. T. Higgins, Simon G. Thompson, Jonathan J. Deeks, and Douglas G. Altman. 2003. Measuring Inconsistency in Meta-Analyses.BMJ327, 7414 (2003), 557–560

    2003

  20. [20]

    D. G. Horvitz and D. J. Thompson. 1952. A Generalization of Sampling Without Replacement from a Finite Universe.J. Amer. Statist. Assoc.47, 260 (1952), 663– 685

    1952

  21. [21]

    Wenlong Ji, Lihua Lei, and Tijana Zrnic. 2025. Predictions as Surrogates: Re- visiting Surrogate Outcomes in the Age of AI.arXiv preprint arXiv:2501.09731 (2025)

    Pith/arXiv arXiv 2025

  22. [22]

    Maurice G. Kendall. 1938. A New Measure of Rank Correlation.Biometrika30, 1/2 (1938), 81–93

    1938

  23. [23]

    1965.Survey Sampling

    Leslie Kish. 1965.Survey Sampling. John Wiley & Sons

    1965

  24. [24]

    Kluger, Kerri Lu, Tijana Zrnic, Sherrie Wang, and Stephen Bates

    Dan M. Kluger, Kerri Lu, Tijana Zrnic, Sherrie Wang, and Stephen Bates. 2025. Prediction-Powered Inference with Imputed Covariates and Nonuniform Sam- pling.arXiv preprint arXiv:2501.18577(2025)

    arXiv 2025

  25. [25]

    A. N. Kolmogorov. 1933. Sulla Determinazione Empirica di una Legge di Dis- tribuzione.Giornale dell’Istituto Italiano degli Attuari4 (1933), 83–91

    1933

  26. [26]

    Lawrence I-Kuei Lin. 1989. A Concordance Correlation Coefficient to Evaluate Reproducibility.Biometrics45, 1 (1989), 255–268

    1989

  27. [27]

    Reagan Mozer. 2026. PPI is the Difference Estimator: Recognizing the Survey Sampling Roots of Prediction-Powered Inference.arXiv preprint arXiv:2603.19160 (2026)

    arXiv 2026

  28. [28]

    Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining Well Calibrated Probabilities Using Bayesian Binning. InProceedings of the AAAI Conference on Artificial Intelligence. 2901–2907

    2015

  29. [29]

    Lin, Steve Metz, Neil Chandra, Jane Jing, and Dimitris Kalimeris

    Viet-An Nguyen, Peibei Shi, Jagdish Ramakrishnan, Udi Weinsberg, Henry C. Lin, Steve Metz, Neil Chandra, Jane Jing, and Dimitris Kalimeris. 2020. CLARA: Confi- dence of Labels and Raters. InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2542–2552

    2020

  30. [30]

    Normand, Mary Beth Landrum, Edward Guadagnoli, John Z

    Sharon-Lise T. Normand, Mary Beth Landrum, Edward Guadagnoli, John Z. Ayanian, Thomas J. Ryan, Paul D. Cleary, and Barbara J. McNeil. 2001. Validat- ing Recommendations for Coronary Angiography Following Acute Myocardial Infarction in the Elderly.Journal of Clinical Epidemiology54, 4 (2001), 387–398

    2001

  31. [31]

    Judea Pearl and Elias Bareinboim. 2016. Causal Inference and the Data-Fusion Problem.Proceedings of the National Academy of Sciences113, 27 (2016), 7345– 7352

    2016

  32. [32]

    Karl Pearson. 1895. Notes on Regression and Inheritance in the Case of Two Parents.Proceedings of the Royal Society of London58 (1895), 240–242

  33. [33]

    John C. Platt. 1999. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. InAdvances in Large Margin Classifiers. Vol. 10. 61–74

    1999

  34. [34]

    Prentice

    Ross L. Prentice. 1989. Surrogate Endpoints in Clinical Trials: Definition and Operational Criteria.Statistics in Medicine8, 4 (1989), 431–440

    1989

  35. [35]

    Rogan and Beth Gladen

    Walter J. Rogan and Beth Gladen. 1978. Estimating Prevalence from the Results of a Screening Test.American Journal of Epidemiology107, 1 (1978), 71–76

    1978

  36. [36]

    Rosenbaum

    Paul R. Rosenbaum. 2002.Observational Studies(2nd ed.). Springer. Table 3:Representativity Levelchecks (5 checks). Check Question Statistic Coverage What fraction is covered?|V 𝑘|/|𝑃𝑘| Sample Representativity Is V𝑘 representative of 𝑃𝑘? Max SMD, KS Effective Sample Size Are weights stable? Kish’s ESS Weight Extremity Any dominant weights? Max/min, CV Weig...

    2002

  37. [37]

    Rosenbaum and Donald B

    Paul R. Rosenbaum and Donald B. Rubin. 1983. The Central Role of the Propensity Score in Observational Studies for Causal Effects.Biometrika70, 1 (1983), 41–55

    1983

  38. [38]

    Tal Sarig, Aurélien Bibaut, and Simon Ejdemyr. 2025. Mind the Gap: Utilizing Delayed Labels to Improve Proxy Metric Validation.arXiv preprint(2025)

    2025

  39. [39]

    Amit Sharma and Emre Kiciman. 2020. DoWhy: An End-to-End Library for Causal Inference.arXiv preprint arXiv:2011.04216(2020)

    Pith/arXiv arXiv 2020

  40. [40]

    Hidetoshi Shimodaira. 2000. Improving Predictive Inference Under Covariate Shift by Weighting the Log-Likelihood Function.Journal of Statistical Planning and Inference90, 2 (2000), 227–244

    2000

  41. [41]

    Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Buenau, and Mo- toaki Kawanabe. 2007. Direct Importance Estimation with Model Selection and Its Application to Covariate Shift Adaptation. InAdvances in Neural Information Processing Systems, Vol. 20

    2007

  42. [42]

    2012.Density Ratio Estimation in Machine Learning

    Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. 2012.Density Ratio Estimation in Machine Learning. Cambridge University Press

    2012

  43. [43]

    Niek Tax, Lorenzo Perini, Fridolin Linder, Daniel Haimovich, Dima Karamshuk, Nastaran Okati, Milan Vojnovic, and Pavlos Athanasios Apostolopoulos. 2026. MCGrad: Multicalibration at Web Scale. InProceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1. 2470–2481

    2026

  44. [44]

    Nilesh Tripuraneni, Lee Richardson, Alexander D’Amour, Jacopo Soriano, and Steve Yadlowsky. 2024. Choosing a Proxy Metric from Past Experiments.arXiv preprint arXiv:2309.07893(2024)

    Pith/arXiv arXiv 2024

  45. [45]

    van der Laan

    Lars van der Laan and Mark J. van der Laan. 2026. Calibeating Prediction-Powered Inference.arXiv preprint arXiv:2604.21260(2026)

    Pith/arXiv arXiv 2026

  46. [46]

    VanderWeele

    Tyler J. VanderWeele. 2013. Surrogate Measures and Consistent Surrogates. Biometrics69, 3 (2013), 561–569

    2013

  47. [47]

    Areti Angeliki Veroniki, Dan Jackson, Wolfgang Viechtbauer, Ralf Bender, Jack Bowden, Guido Knapp, Oliver Kuss, Julian P. T. Higgins, Dean Langan, and Georgia Salanti. 2016. Methods to Estimate the Between-Study Variance and Its Uncertainty in Meta-Analysis.Research Synthesis Methods7, 1 (2016), 55–79

    2016

  48. [48]

    2009.Optimal Transport: Old and New

    Cedric Villani. 2009.Optimal Transport: Old and New. Springer

    2009

  49. [49]

    2005.Algorithmic Learning in a Random World

    Vladimir Vovk, Alex Gammerman, and Glenn Shafer. 2005.Algorithmic Learning in a Random World. Springer

    2005

  50. [50]

    McCormick, and Jeffrey T

    Siruo Wang, Tyler H. McCormick, and Jeffrey T. Leek. 2020. Methods for Correct- ing Inference Based on Outcomes Predicted by Machine Learning.Proceedings of the National Academy of Sciences117, 48 (2020), 30266–30275

    2020

  51. [51]

    Student’s

    Bernard L. Welch. 1947. The Generalization of “Student’s” Problem When Several Different Population Variances Are Involved.Biometrika34, 1/2 (1947), 28–35

    1947

  52. [52]

    Steve Wilkins-Reeves, Alexandra Darmon, and Deeksha Sinha. 2026. Estimate- Level Adjustment for Inference with Proxies Under Random Distribution Shifts. Submitted to KDD ’26 Research Track. arXiv preprint (forthcoming)(2026)

    2026

  53. [53]

    Siqi Wu and Paul Resnick. 2024. Calibrate-Extrapolate: Rethinking Prevalence Estimation with Black Box Classifiers. InProceedings of the International AAAI Conference on Web and Social Media, Vol. 18. 1634–1647

    2024

  54. [54]

    Bianca Zadrozny and Charles Elkan. 2002. Transforming Classifier Scores into Accurate Multiclass Probability Estimates. InProceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 694–699

    2002

  55. [55]

    diagnostic-to-adjustment bridge

    Alessandro Zito, Dylan Greaves, Jacopo Soriano, and Lee Richardson. 2025. Pareto Optimal Proxy Metrics.arXiv preprint arXiv:2307.01000(2025). A Complete Check Inventory Tables 3–6 list all 46 checks implemented inproxymate. Conference’17, July 2017, Washington, DC, USA Darmon et al. Table 4:Unit Levelchecks (27 checks). Check Question Statistic Accuracy T...

    Pith/arXiv arXiv 2025