REVIEW 3 major objections 5 minor 17 references
Conformal C2ST: Turning weak classifiers into strong two-sample tests
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Conformal calibration of classifier scores turns any weak classifier into a valid two-sample test.
desk verdict The conformal C2ST paper is basically right; the stress-test worry about moment bounds not implying KS power is actually wrong, and the real issues are presentation gaps in the robustness theorem and the multiple-test null theory. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the conformal p-value $U$, the rank of a test point's score among $m$ calibration scores, with uniform random tie-breaking. The method depends only on the ordering of scores, so the oracle density ratio $r(x)=p(x)/q(x)$, and any strictly increasing transform of it, is the optimal score. The proofs connect $U$ to AUC and total variation through $\mathbb{E}[U]=1-\mathrm{AUC}(r)\le \frac12 - \frac12 TV(p,q)$, then control the error of a plug-in ratio estimate by comparing mis-ranking events $Z=r(\tilde X)-r(\tilde X')$ with their estimated counterparts, using a bounded-density low-noise assumption on $Z$ near zero and an optimized threshold to get the $O(\epsilon^{2/3})$ bound.
What would settle it
Simulate $p$ and $q$ with a density ratio that is locally flat or atomic over a region of positive $q$-mass, so that Assumption 3.4 fails, insert a classifier with measured mean-square error $\epsilon$, and run both conformal tests; if the rejection rate at $p \neq q$ remains at the nominal level for small but nonzero $\epsilon$, then the claimed power guarantee is not doing the work.
Extended reading notes
Core claim
The paper's central claim is a robustness guarantee for rank-based, conformal two-sample tests. Using any scoring function $s$ that approximates the oracle density ratio $r = p/q$ with mean-square error $\epsilon^2$ under $q$, the expected conformal p-value under the alternative stays within $O(\epsilon^{2/3})$ of the oracle p-value, provided a low-noise condition on $r$ holds; the multiple test's statistic enjoys the same relative gap. Under the null, the conformal p-values are exactly uniform for any $s$ and any finite calibration size, so type-I error is decoupled from classifier quality. Under the alternative, $\mathbb{E}[U] = 1-\mathrm{AUC}(r) \le \frac12 - \frac12 TV(p,q)$, so weak ranking signal translates directly into power. The conclusion is that a near-Bayes-optimal classifier is not required: a weak but informative ranking suffices.
Load-bearing premise
The load-bearing premise is that the true density ratio is not locally constant, in the sense that pairwise differences of $r$ under $q$ have a bounded density near zero, and that the learned score approximates the oracle ratio in mean square with error $\epsilon^2$; if either fails, most dramatically when the classifier is random, the $O(\epsilon^{2/3})$ power guarantee becomes vacuous and the test can sit at nominal power even though $p$ and $q$ differ.
Editorial extensions
If this is right
- Posterior validation can be run with off-the-shelf or poorly calibrated classifiers, as long as their score ordering carries any signal.
- The uniform test controls type-I error exactly at any finite calibration size, so it is a valid diagnostic even when training data are scarce.
- The multiple test matches the standard C2ST's sample budget while keeping a relative power gap of $O(\epsilon^{2/3})$, making it a drop-in replacement for C2ST.
- The tests detect discrepancies that affect the joint posterior without changing one-dimensional marginals, a regime where rank-based simulation-based calibration is insensitive.
- On Gaussian perturbations, image corruptions, and a gravitational-lensing posterior benchmark, the conformal tests detect smaller misspecifications and retain more power under classifier degradation than the compared baselines.
Reading between the lines
- Because the test depends only on ranks, any monotone transform of the score yields the same p-values; this suggests one can freeze a feature extractor and use a lightweight head, and the method will still be valid.
- The $O(\epsilon^{2/3})$ bound is worst-case; a sharper local expansion might show the effective cost of misestimation is often closer to $\epsilon^2$ when the score is near the oracle.
- The same conformal calibration should extend to conditional two-sample tests with unequal marginals, provided a weak but informative ratio estimate is available.
- A testable corollary is that power of the uniform test should saturate once $m$ is large relative to the reciprocal of the AUC deviation; the paper's ablations suggest $m=50$ is near that point.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes two conformal variants of the classifier two-sample test (C2ST): a uniform test that draws a fresh calibration set for every test point and a multiple test that shares one calibration set and matches the C2ST simulation budget. The authors show that under the null, conformal p-values are exactly uniform for any deterministic scoring function, and under the alternative the expected conformal p-value is 1 - AUC(r), which is at most 1/2 - TV(p,q)/2. They further prove an O(epsilon^{2/3}) bound on the gap between the expected p-values obtained from an oracle density ratio and from an estimated score with L2 error epsilon^2, under a low-noise condition on the density ratio. Experiments on perturbed Gaussian posteriors, CIFAR-10 corruptions, and a gravitational-lensing posterior benchmark demonstrate higher power and greater robustness to classifier degradation than standard C2ST and several other baselines.
Significance. If the theoretical claims are fully established, the paper makes a valuable practical point: exact finite-sample Type-I error control can be decoupled from classifier accuracy, and power degrades gently with the quality of the score function rather than collapsing when the classifier is far from Bayes optimal. This is directly relevant to simulation-based inference, where neural posterior estimators must be validated against the true posterior. The paper ships code, and the experiments cover a useful range of low-dimensional, high-dimensional, and real-data problems. The finite-sample validity lemma and the AUC-Total-Variation identity are clean and correct. The main caveat is that the central power claim for the uniform test, as stated, requires an additional elementary step connecting the first-moment bound to the KS sup-norm statistic; this step is missing from the manuscript. Theorems 3.5 and 4.2 are also stated more strongly than their proofs currently support. These issues are local and fixable rather than fundamental.
major comments (3)
- [§3.1, Lemma 3.3 and the KS statistic] The power argument for the uniform test jumps from the first-moment bound E[U] <= 1/2 - TV(p,q)/2 to the claim that the KS statistic sup_u |G(u)-u| has non-trivial power. This implication is true but is not stated or proved in the manuscript. If F(u)=P(U<=u), then integration by parts gives E[U]=1-∫_0^1 F(u)du, so |E[U]-1/2| <= sup_u |F(u)-u|. Therefore a mean gap of at least TV(p,q)/2 forces a population KS deviation of at least TV(p,q)/2, and the empirical KS statistic converges to this deviation by Glivenko-Cantelli. Please add this as an explicit lemma and use it to state the uniform test's power guarantee. As written, the central 'weak classifier suffices' claim for the uniform test is not certified by the displayed results.
- [Theorem 3.5 and Appendix A.2] Theorem 3.5 is stated as a finite-sample statement ('for all m > M'), but the proof only establishes the bound after passing to the limit m→∞ via the strong law of large numbers and dominated convergence. No argument is given for uniformity of the convergence in the error epsilon, and Lemma A.3's O(1/m) variance bound is not used to control the finite-m gap. Please restate Theorem 3.5 as an asymptotic result (m→∞, followed by n_q→∞), or prove a finite-m version with an explicit rate in m. This matters because Theorem 4.2 invokes Theorem 3.5 as a black box.
- [Theorem 4.2 and Appendix A.4.3] The proof of the relative-gap claim is too compressed in two places. First, the statement that the fluctuation term is 'O_p(n_q^{-1/2})' is only true after dividing the O_p(1) fluctuation gap by the O_p(√n_q Δ_r) denominator; the intermediate step should be shown explicitly. Second, the bound |AUC(hat r)-AUC(r)| <= C epsilon^{2/3} used for the drift gap follows from Theorem 3.5 only in the m→∞ limit and under tie-free scores; these conditions should be recorded in the statement of Theorem 4.2 or in its proof.
minor comments (5)
- [§5.1, Evaluation setup] The text says rejection rates are computed across 200 independent trials and later says 'All experiments use three random seeds'; please clarify whether the 200 trials are per seed or in total.
- [§5.1, Table 1 and Figure 2] Conformal Uniform is compared at a larger sampling budget from p (mN versus N for C2ST). The paper acknowledges this trade-off, but the main-text power comparison should state explicitly which curves use which value of m.
- [§3.1, paragraph after Lemma 3.3] The sentence 'conformal p-values become stochastically smaller than uniform' is stronger than what is proved; the proved statement is that their expectation is less than 1/2. Please rephrase to avoid claiming a stochastic ordering that does not follow from the displayed results.
- [Appendix A.2] The notation E_ξ[U_hat] is used before it is defined; please define it as expectation over the tie-breaking variable ξ conditional on the calibration and test data.
- [Abstract and Section 3.2] Minor wording issues include 'despite simplicity of the C2ST' in the abstract and the sentence 'We defer the proof to Appendix A.2', which appears mid-paragraph immediately after Theorem 3.5; both should be rephrased.
Circularity Check
No circularity: the conformal C2ST derivation is self-contained; validity follows from conformal exchangeability and power bounds are analytic identities, not fitted or citation-dependent conclusions.
full rationale
I walked the paper's derivation chain and found no step in which a claimed prediction reduces by construction to an input, a fitted parameter, or a load-bearing self-citation. Lemma 3.1 (exact finite-sample uniformity under H0) is a standard conformal property quoted from Vovk et al. (2005) and Lei & G'Sell (2018); it holds for any deterministic scoring function and any finite calibration size, and it does not depend on the paper's own experiments or fitted values. Lemma 3.2 and Corollary A.1 are proved by an explicit antisymmetry argument showing AUC(s) <= AUC(r), with equality iff s is a strictly increasing transform of the oracle density ratio; this is a mathematical statement about the definitions of AUC and r, not a fitted relation. Lemma 3.3 computes E[U] = 1 - AUC(r) by taking the m->infinity limit of the rank-based conformal p-value; this is an algebraic identity between two quantities defined from the same ranking, and it is not used to define either quantity. Theorem 3.5 is an analytic robustness bound: the proof decomposes E[U_hat] - E[U] into a term involving Z = r(X~)-r(X~'), uses the bounded-density condition (Assumption 3.4) and the assumed L2 error epsilon^2, and optimizes a threshold to obtain O(epsilon^{2/3}). No constant is tuned to data, and the bound is not derived from the conclusion it purports to support. Theorem 4.2 similarly follows from the U-statistic CLT, the tie-noise assumption, and the previous bound; even if its power implication is debatable (a moment bound may not control the KS statistic), that is a correctness or strength concern, not circularity. The only self-citations are the NPTBench benchmark suite (Chen et al. 2024) and the authors' own code link, used for experimental perturbations and reproducibility; neither is load-bearing for the theoretical claims. There is no uniqueness theorem imported from the authors' prior work, no ansatz smuggled in via self-citation, and no renaming of a known result as a new one; the paper explicitly credits Hu and Lei (2024) for the conformal framework and positions its contribution as robustness theory for weak plug-in classifiers. Under the stated review rules, the honest finding is 0: no significant circularity.
Assumptions & free parameters
assumptions (5)
- domain assumption Assumption 3.4: Z = r(X~)-r(X~') has a density bounded in a neighborhood of zero under q.
- domain assumption Assumption 4.1: ties in the score r-hat occur at rate O_p(1).
- standard math Hu and Lei (2024, Theorem 2): asymptotic null normality and divergence under the alternative for the shared-calibration U-statistic.
- domain assumption Shared marginal assumption: q(theta,y) = pi(y) q(theta|y) with the same y-marginal as p.
- domain assumption Absolute continuity of p(theta|y) and q(theta|y) with respect to a common base measure.
Cite this review
Pith. "Pith review of Conformal C2ST: Turning weak classifiers into strong two-sample tests." pith.science (2026). https://pith.science/paper/BR6WF6OL
@misc{pith2026250717026,
author = {Pith},
title = {Pith review of: Conformal C2ST: Turning weak classifiers into strong two-sample tests},
year = {2026},
howpublished = {\url{https://pith.science/paper/BR6WF6OL}},
note = {Machine review of arXiv:2507.17026}
}
abstract
The two-sample testing problem, a fundamental task in statistics and machine learning, seeks to determine whether two sets of samples, drawn from underlying distributions $p$ and $q$, are in fact identically distributed (i.e. whether $p=q$). A popular and intuitive approach is the classifier two-sample test (C2ST), where a classifier is trained to distinguish between samples from $p$ and $q$. Yet despite simplicity of the C2ST, its reliability hinges on access to a near-Bayes-optimal classifier, a requirement that is rarely met and difficult to verify. This raises a major open question: can a weak classifier still be useful for two-sample testing? We show that the answer is a definitive yes. Building on the work of Hu and Lei (2024), we analyze two conformal variants of the C2ST that convert the scores from any trained classifier -- even if weak, biased, or overfit -- into exact, finite-sample p-values. We establish two key theoretical properties of the conformal C2ST: (i) finite-sample Type-I error control, and (ii) non-trivial power that degrades gently in tandem with the error of the trained classifier. The upshot is that even poorly performing classifiers can yield powerful and reliable two-sample tests. This general framework finds a powerful application in Bayesian inference, particularly for validating Neural Posterior Estimation (NPE) models, where the task of comparing a learned posterior approximation $q(\theta \mid y)$ to the true posterior $p(\theta \mid y)$ can be framed as a two-sample test. Empirically, the Conformal C2ST outperforms classical discriminative tests across a wide range of benchmarks for this task. Our results establish the conformal C2ST as a practical, theoretically grounded diagnostic tool.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
A two-sample conditional distribution test using conformal prediction and weighted rank sum
Xiaoyu Hu and Jing Lei. A two-sample conditional distribution test using conformal prediction and weighted rank sum. Journal of the American Statistical Association, 119 0 (546): 0 1136--1154, 2024. doi:10.1080/01621459.2023.2177165. URL https://doi.org/10.1080/01621459.2023.2177165
arXiv 2024
-
[2]
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 0 6840--6851, 2020
2020
-
[3]
Compositional score modeling for simulation-based inference
Tomas Geffner, George Papamakarios, and Andriy Mnih. Compositional score modeling for simulation-based inference. In International Conference on Machine Learning, pages 11098--11116. PMLR, 2023
work page 2023
-
[4]
Conditional diffusions for amortized neural posterior estimation
Tianyu Chen, Vansh Bansal, and James G Scott. Conditional diffusions for amortized neural posterior estimation. In International Conference on Artificial Intelligence and Statistics, pages 2377--2385. PMLR, 2025
work page 2025
-
[5]
All-in-one simulation-based inference
Manuel Gloeckler, Michael Deistler, Christian Weilbach, Frank Wood, and Jakob H Macke. All-in-one simulation-based inference. arXiv preprint arXiv:2404.09636, 2024
arXiv 2024
-
[6]
Normalizing flows for probabilistic modeling and inference
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22 0 (57): 0 1--64, 2021
2021
-
[7]
Flow matching for scalable simulation-based inference
Jonas Wildberger, Maximilian Dax, Simon Buchholz, Stephen Green, Jakob H Macke, and Bernhard Sch \"o lkopf. Flow matching for scalable simulation-based inference. Advances in Neural Information Processing Systems, 36: 0 16837--16864, 2023
work page 2023
-
[8]
Auto-encoding variational bayes, 2013
Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013
2013
Show all 17 references
-
[9]
Validating bayesian inference algorithms with simulation-based calibration
Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. Validating bayesian inference algorithms with simulation-based calibration. arXiv preprint arXiv:1804.06788, 2018
2018 arXiv
-
[10]
Sampling-based accuracy testing of posterior estimators for general inference
Pablo Lemos, Adam Coogan, Yashar Hezaveh, and Laurence Perreault-Levasseur. Sampling-based accuracy testing of posterior estimators for general inference. In International Conference on Machine Learning, pages 19256--19273. PMLR, 2023
2023
-
[11]
Revisiting classifier two-sample tests
David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. arXiv preprint arXiv:1610.06545, 2016
2016 arXiv
-
[12]
Algorithmic Learning in a Random World
Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005
2005
-
[13]
Distribution-free predictive inference for regression
Jing Lei and Max G’Sell. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113 0 (523): 0 1094--1111, 2018
2018
-
[14]
E. L. Lehmann and Joseph P. Romano. Testing Statistical Hypotheses. Springer Texts in Statistics. Springer, New York, 3rd edition, 2005. ISBN 0-387-98864-5
2005
-
[15]
An introduction to roc analysis
Tom Fawcett. An introduction to roc analysis. Pattern Recognition Letters, 27 0 (8): 0 861--874, 2006
2006
-
[16]
NPTBench : A benchmark suite for neural posterior testing
Tianyu Chen, Vansh Bansal, and James Scott. NPTBench : A benchmark suite for neural posterior testing. Technical report, 2024. URL https://github.com/TianyuCodings/NPTBench
2024
-
[17]
Density estimation using real nvp
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016
2016 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.