REVIEW 2 major objections 6 minor 41 references
Flexible Selective Inference with Flow-based Transport Maps
T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A normalizing flow that learns a transport map between pre- and post-selection distributions enables valid selective inference even when the selection event has no closed-form description.
desk verdict A genuinely useful transport-map method for selective inference with intractable selection events, but the confidence-set validity claim runs ahead of what the training procedure actually certifies. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the transport map $\tau^*_{\theta_0}$, a diffeomorphism on $\mathbb{R}^d$ that pushes the pre-selection distribution $P_{\theta_0}$ of the test statistic onto the conditional distribution $P^*_{\theta_0}$ given the observed selection event. Its inverse is the debiasing transformation: applying $\tau^{*-1}_{\theta_0}$ to a statistic drawn from the conditional distribution reproduces the pre-selection distribution, so a standard test can be run on the pullback. Because the selection event has no analytic form, the paper replaces the exact map with a normalizing flow in the RealNVP architecture, trained to minimize the KL divergence between the pushforward and the target conditional distribution; the flow's triangular Jacobian makes the training objective tractable. A conditional version, parameterized with the candidate parameter value as an additional input, supplies transport maps across a range of $\theta$ values, enabling confidence-set inversion. The map also yields an approximate selective density $q^*_\theta(t) = p_\theta(\hat{\tau}^{-1}_\theta(t)) \,|\nabla \hat{\tau}^{-1}_\theta(t)|$, which drives selective MLE and quantile inference and is multiplied by a known selection probability when merging with existing fixed-$\lambda$ lasso corrections.
What would settle it
Simulate many global-null datasets through the same selection rule, compute the flow-based p-values, and test their empirical distribution for uniformity; a systematic departure from Uniform[0,1] would show that the KL error bound is not met in practice.
Extended reading notes
Core claim
The central claim is that post-selection validity reduces to learning a transport map. For a statistic $T$ with pre-selection distribution $P_{\theta_0}$, the ideal map $\tau^*_{\theta_0}$ pushes $P_{\theta_0}$ forward to the conditional distribution $P^*_{\theta_0}$ given the observed selection event; by Lemma 1, applying the inverse map to a statistic drawn from $P^*_{\theta_0}$ yields a statistic with distribution $P_{\theta_0}$. Theorem 3.2 shows that if an approximate map $\hat{\tau}_{\theta_0}$ satisfies $\mathrm{KL}(P^*_{\theta_0} \| \hat{\tau}_{\theta_0}\#P_{\theta_0}) \le \varepsilon$, then composing any level-$\alpha$ test with $\hat{\tau}_{\theta_0}^{-1}$ gives selective Type I error at most $\alpha + \sqrt{\varepsilon/2}$, and Theorem 3.3 gives the analogous near-nominal coverage guarantee for inverted tests. The map is learned by minimizing this KL divergence over a normalizing flow, using rejection-sampled draws from the conditional distribution; a conditional flow indexed by the parameter $\theta$ is used when constructing confidence sets. The same learned map also provides an approximate selective density, supporting selective maximum likelihood, quantile-based p-values, and a compositional correction that multiplies the learned density for an intractable selection step by a known selection probability for a tractable step such as fixed-$\lambda$ lasso selection.
Load-bearing premise
The flow must actually get close to the true conditional distribution for every parameter value used in inference, but the training objective only checks an average over training samples and a user-chosen prior, and rejection sampling must be frequent enough to represent the observed selection event.
Editorial extensions
If this is right
- Selective p-values and confidence sets become available for selection events that are purely algorithmic, such as cross-validation choices of knot counts, polynomial degrees, and principal component numbers.
- The theorems separate approximation quality from inference validity: any flow achieving small KL divergence automatically inherits near-nominal error and coverage, so training the flow is the remaining statistical task.
- Users gain a closed-form approximate selective density, making likelihood-based and quantile-based inference available alongside conventional tests.
- The method composes with existing fixed-$\lambda$ lasso corrections, so multi-stage pipelines containing both tractable and intractable selection steps can be handled without re-deriving the full conditional distribution.
- Because data-carving reuses the full data, the paper's simulations show confidence intervals roughly 1.5 to 2 times shorter than splitting-based alternatives at comparable coverage.
Reading between the lines
- Editorial inference: the only interface the method needs is the ability to replay the selection algorithm on synthetic data, so the same recipe should transfer to stochastic selection rules whose randomness is not Gaussian, such as bagging or early stopping, provided rejection sampling is affordable.
- Editorial inference: the paper trains one conditional flow over a user-specified prior $\pi$, but the coverage guarantee in Theorem 3.3 is per-$\theta$; a user who cares about a specific region of the parameter space should check the flow's validation loss there, because an average certificate can conceal a poorly learned region.
- Editorial inference: a calibration test is immediately available: under the global null, flow-based p-values should be uniform across simulation replicates, and a systematic departure from uniformity would empirically falsify the KL assumption without knowing the true conditional distribution.
- Editorial inference: the learned selective density opens a route to Bayesian selective inference by appending a prior to $q^*_\theta$ and sampling the posterior; the paper itself flags this direction as future work.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a data-carving method for selective inference when the selection event has no analytic description. Given a pre-selection distribution P_{θ_0} of a statistic T and the conditional distribution P*_{θ_0} given the observed selection event, the method trains a normalizing flow τ̂ so that τ̂#P_{θ_0} approximates P*_{θ_0}; the inverse map is then used as a debiasing transformation. The paper proves (Theorems 3.1–3.3) that if KL(P*_{θ_0}∥τ̂#P_{θ_0}) ≤ ε for the relevant parameter values, then the selective Type I error is at most α + √(ε/2) and confidence sets have coverage at least 1 − α − √(ε/2). Conditional flows are trained by minimizing an empirical KL objective, and experiments cover polynomial degree selection via ANOVA, spline knot selection via CV, lasso with CV-selected regularization, PCR with CV-selected components, and a single-cell logistic PCA application.
Significance. Should the KL premise be satisfiable and certifiable, the method is genuinely broad: it replaces analytic descriptions of selection events by simulation and flow-based density estimation, and it offers a practical way to handle adaptive tuning steps that existing data-carving tools cannot accommodate. The theoretical results are elementary but correct, the implementation is described in enough detail to be reproducible, and the authors provide code. The main deliverable—valid selective p-values and confidence sets for intractable-selection problems—is of clear interest to the selective-inference and simulation-based-inference communities. The central caveat is that the coverage guarantee requires a uniform KL bound that the training procedure never certifies; I regard this as a substantive but fixable gap rather than a fatal flaw.
major comments (2)
- [Theorem 3.3 and Section 4.3 (Eq. (12))] The coverage guarantee is not supported by the training objective actually solved. Theorem 3.3 requires KL(P*_{θ_0}∥τ̂_{θ_0}#P_{θ_0}) ≤ ε for every θ_0 ∈ Θ, but Eq. (12) minimizes the empirical π-average of this KL over θ^{(b)} ∼ π, and the validation procedure in Section 6 and Appendix B.2 monitors the same π-average. An average can be small while individual θ in the tails of π have very large KL; since π is centered at the observed unconditional MLE (Appendix B.3), the θ values in the tails of the eventual confidence interval are precisely the least constrained by training. Moreover, no per-θ KL certificate is reported, so the paper never establishes the uniform premise of Theorem 3.3 for its implemented confidence sets. This is a logical gap in the main validity claim, not a cosmetic issue; the authors should either certify a uniform KL bound (for example, by reporting per-θ KL over a grid and a tail bound), choose a π that provably covers Θ, or restate the guarantee as coverage averaged over θ ∼ π rather than the conditional selective coverage of Eq. (2).
- [Appendix B.3 and Section 4.3] The rejection-sampling scheme with a 100-retry cap biases the effective training distribution away from the stated π. For conditional flows, Appendix B.3 says that if no accepted sample is obtained after 100 tries, the algorithm skips to the next θ^{(b+1)}. Parameter values with low selection probability P(cM(D,W)=Mo | θ) are therefore under-represented relative to π in the training set for Eq. (12), and for extreme θ they may be absent altogether. Since Theorem 3.3 needs control for every θ0 ∈ Θ, the retry cap actively worsens the average-versus-uniform gap identified above. The paper should report the acceptance rate per θ region, or replace the cap with a mechanism that still yields an unbiased estimate of the π-averaged objective, such as importance weights.
minor comments (6)
- [Section 3.2] The proposition following Theorem 3.2 is numbered Proposition 3.2 although Theorem 3.3 follows it; renumber to avoid the apparent numbering conflict.
- [Eq. (13) in Section 5.2] The first factor of the conditional density is denoted p_{θ_{Mo}}(t | bλ = λ, A = Ao), but it should condition on the observed value bλ = λo; the same subscript appears in the surrounding text.
- [Figure 6 caption] The x-axis label 'SR-1 SR-1 SR-3' appears to be a typo and should read 'SR-1 SR-2 SR-3'.
- [Section 6.1 and Section 6.3] When only about 100 selected simulations (SR-1) and about 270 nonempty selections (Section 6.3) underlie the reported coverage proportions, the Monte Carlo standard error is around two to three percentage points, so the plots cannot sharply distinguish 0.95 from, say, 0.91 or 0.99; please report selected-sample counts and standard errors alongside the coverage plots.
- [Section 6, training details] The text says 2,000 training samples and 500 validation samples are generated per simulation; it should clarify whether a fresh conditional flow is trained for every simulation repetition or once per setting, since this materially affects the computational cost claim.
- [Proof of Theorem 3.2] The equality KL(τ̂^{-1}#P*∥P) = KL(P*∥τ̂#P) requires a change of variables and hence that τ̂ be a diffeomorphism, whereas the theorem states only that τ̂ is invertible and differentiable; the inverse differentiability should be stated explicitly.
Circularity Check
No circular derivation: transport-map guarantees are conditional on an explicit KL premise, and the flow is trained on independent synthetic draws from the target conditional distribution.
full rationale
The central theorems are conditional statements rather than fitted predictions: Theorem 3.2 proves, via Pinsker's inequality, that if KL(P*_theta0 || tau_hat_theta0#P_theta0) <= epsilon then the selective error is at most alpha + sqrt(epsilon/2); Theorem 3.3 uses the same premise uniformly over the parameter space. The flow is learned by minimizing an empirical KL objective (equations (10)-(12)) over synthetic draws T^(b) from P*_theta0, or from P*_theta(b) with theta(b) ~ pi; the observed statistic enters only through the conditioning event Mo and the center of pi, never as a fitted target that is later called a prediction. Figure 1's pullback to N(0, Sigma) is a sanity check of the training objective, not an independent forecast. The self-citations (Panigrahi and Taylor 2023; Liu 2023; Panigrahi et al. 2024) provide external components or comparisons for tractable selection steps and selective-MLE intervals, and do not carry the paper's core transport-map reduction. The main legitimate concerns are non-circular validity gaps: the pi-averaged training objective in (12) does not certify the per-theta KL bound required by Theorem 3.3, and Appendix B.3 caps rejection sampling at 100 tries, so rare selection events may be under-sampled. These are correctness risks, not cases where an output reduces by definition or by self-citation to its input; thus the paper is not circular.
Assumptions & free parameters
free parameters (4)
- training sample size B and validation size =
B = 2000, n_val = 500
- RealNVP architecture (L, hidden units, spline bins) =
L = 12, 1 hidden layer of 8 neurons; 20 bins for 1D splines
- π distribution for conditional flow =
Normal centered at observed unconditional MLE with MLE covariance
- Rejection sampling retry cap =
100 attempts
assumptions (7)
- domain assumption An ideal transport map τ* exists and is a diffeomorphism for each θ
- domain assumption The selection algorithm can be rerun on synthetic data and the observed event Mo has positive probability under simulation
- ad hoc to paper KL(P*θ || τ̂θ#Pθ) ≤ ε for all θ relevant to inference
- ad hoc to paper For confidence sets, a uniform KL bound over Θ is achieved by minimizing expected KL under π
- domain assumption Pre-selection density pθ is known and computable
- domain assumption Lasso selection is conditionally independent of λ-selection given T and A
- standard math Pinsker's inequality and KL invariance under invertible differentiable maps
Cite this review
Pith. "Pith review of Flexible Selective Inference with Flow-based Transport Maps." pith.science (2026). https://pith.science/paper/KAQQDDNS
@misc{pith2026250601150,
author = {Pith},
title = {Pith review of: Flexible Selective Inference with Flow-based Transport Maps},
year = {2026},
howpublished = {\url{https://pith.science/paper/KAQQDDNS}},
note = {Machine review of arXiv:2506.01150}
}
read the original abstract
Data-carving methods perform selective inference by conditioning the distribution of data on the observed selection event. However, existing data-carving approaches typically require an analytically tractable characterization of the selection event. This paper introduces a new method that leverages tools from flow-based generative modeling to approximate a potentially complex conditional distribution, even when the underlying selection event lacks an analytical description -- take, for example, the data-adaptive tuning of model parameters. The key idea is to learn a transport map that pushes forward a simple reference distribution to the conditional distribution given selection. This map is efficiently learned via a normalizing flow, without imposing any further restrictions on the nature of the selection event. Through extensive numerical experiments on both simulated and real data, we demonstrate that this method enables flexible selective inference by providing: (i) valid p-values and confidence sets for adaptively selected hypotheses and parameters, (ii) a closed-form expression for the conditional density function, enabling likelihood-based and quantile-based inference, and (iii) adjustments for intractable selection steps that can be easily integrated with existing methods designed to account for the tractable steps in a selection procedure involving multiple steps.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Alquicira-Hernandez, J., Sathe, A., Ji, H. P., Nguyen, Q., and Powell, J. E. (2019). scPred : accurate supervised method for cell-type classification from single-cell RNA -seq data. Genome biology , 20:1--17
work page 2019
-
[2]
Awan, J. and Wang, Z. (2024). Simulation-based, finite-sample inference for privatized data. Journal of the American Statistical Association , pages 1--14
work page 2024
-
[3]
Bakshi, S., Huang, Y., Panigrahi, S., and Dempsey, W. (2024). Inference with randomized regression trees. arXiv preprint arXiv:2412.20535
arXiv 2024
-
[4]
Berk, R., Brown, L., Buja, A., Zhang, K., and Zhao, L. (2013). Valid post-selection inference. The Annals of Statistics , 41(2):802--837
2013
-
[5]
Cox, D. R. (1975). A note on data-splitting for the evaluation of significance levels. Biometrika , pages 441--444
work page 1975
-
[6]
Cranmer, K., Pavez, J., and Louppe, G. (2015). Approximating likelihood ratios with calibrated discriminative classifiers. arXiv preprint arXiv:1506.02169
arXiv 2015
-
[7]
Dharamshi, A., Neufeld, A., Motwani, K., Gao, L. L., Witten, D., and Bien, J. (2025). Generalized data thinning using sufficient statistics. Journal of the American Statistical Association , 120(549):511--523
work page 2025
-
[8]
Dinh, L., Sohl-Dickstein, J., and Bengio, S. (2017). Density estimation using Real NVP . In International Conference on Learning Representations
work page 2017
Show all 41 references
-
[9]
Durkan, C., Bekasov, A., Murray, I., and Papamakarios, G. (2019). Neural spline flows. Advances in neural information processing systems , 32
2019
-
[10]
Fithian, W., Sun, D., and Taylor, J. (2014). Optimal inference after model selection. arXiv preprint arXiv:1410.2597
2014 arXiv
-
[11]
L., Bien, J., and Witten, D
Gao, L. L., Bien, J., and Witten, D. (2024). Selective inference for hierarchical clustering. Journal of the American Statistical Association , 119(545):332--342
2024
-
[12]
Guglielmini, S., Claeskens, G., and Panigrahi, S. (2025). Selective inference in graphical models via maximum likelihood. arXiv preprint arXiv:2503.24311
2025 arXiv
-
[13]
Hao, Y., Hao, S., Andersen-Nissen, E., III, W. M. M., Zheng, S., Butler, A., Lee, M. J., Wilk, A. J., Darby, C., Zagar, M., Hoffman, P., Stoeckius, M., Papalexi, E., Mimitou, E. P., Jain, J., Srivastava, A., Stuart, T., Fleming, L. B., Yeung, B., Rogers, A. J., McElrath, J. M....
2021
-
[14]
Huang, Y., Pirenne, S., Panigrahi, S., and Claeskens, G. (2023). Selective inference using randomized group lasso estimators for general models. arXiv preprint arXiv:2306.13829
2023 arXiv
-
[15]
James, G., Witten, D., Hastie, T., Tibshirani, R., et al. (2013). An introduction to statistical learning , volume 112. Springer
2013
-
[16]
and Leeb, H
Kivaranovic, D. and Leeb, H. (2021). On the length of post-model-selection confidence intervals conditional on polyhedral constraints. Journal of the American Statistical Association , 116(534):845--857
2021
-
[17]
Le Duy, V. N. and Takeuchi, I. (2022). More powerful conditional selective inference for generalized lasso by parametric programming. Journal of Machine Learning Research , 23(300):1--37
2022
-
[18]
L., Sun, Y., and Taylor, J
Lee, J., Sun, D. L., Sun, Y., and Taylor, J. (2016). Exact post-selection inference, with application to the lasso. The Annals of Statistics , 44(3):907--927
2016
-
[19]
Leiner, J., Duan, B., Wasserman, L., and Ramdas, A. (2025). Data fission: splitting a single data point. Journal of the American Statistical Association , 120(549):135--146
2025
-
[20]
Liu, Q., Xu, J., Jiang, R., and Wong, W. H. (2021). Density estimation using deep generative neural networks. Proceedings of the National Academy of Sciences , 118(15):e2101344118
2021
-
[21]
Liu, S. (2023). An exact sampler for inference after polyhedral model selection. arXiv preprint arXiv:2308.10346
2023 arXiv
-
[22]
Liu, S., Markovic, J., and Taylor, J. (2022). Black-box selective inference via bootstrapping. arXiv preprint arXiv:2203.14504
2022 arXiv
-
[23]
Liu, S., Panigrahi, S., and Soloff, J. A. (2024). Cross-validation with antithetic gaussian randomization. arXiv preprint arXiv:2412.14423
2024
-
[24]
P., and Ryder, R
Marin, J.-M., Pudlo, P., Robert, C. P., and Ryder, R. J. (2012). Approximate B ayesian computational methods. Statistics and computing , 22(6):1167--1180
2012
-
[25]
L., and Witten, D
Neufeld, A., Dharamshi, A., Gao, L. L., and Witten, D. (2024). Data thinning for convolution-closed distributions. Journal of Machine Learning Research , 25(57):1--35
2024
-
[26]
Panigrahi, S., Fry, K., and Taylor, J. (2024). Exact selective inference with randomization. Biometrika , 111(4):1109--1127
2024
-
[27]
W., and Kessler, D
Panigrahi, S., MacDonald, P. W., and Kessler, D. (2023). Approximate post-selective inference for regression with the group lasso. Journal of machine learning research , 24(79):1--49
2023
-
[28]
and Taylor, J
Panigrahi, S. and Taylor, J. (2023). Approximate selective inference via maximum likelihood. Journal of the American Statistical Association , 118(544):2810--2820
2023
-
[29]
and Murray, I
Papamakarios, G. and Murray, I. (2016). Fast -free inference of simulation models with B ayesian conditional density estimation. Advances in neural information processing systems , 29
2016
-
[30]
Papamakarios, G., Sterratt, D., and Murray, I. (2019). Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows. In The 22nd international conference on artificial intelligence and statistics , pages 837--848. PMLR
2019
-
[31]
Perry, R., Panigrahi, S., Bien, J., and Witten, D. (2024). Inference on the proportion of variance explained in principal component analysis. arXiv preprint arXiv:2402.16725
2024 arXiv
-
[32]
and Claeskens, G
Pirenne, S. and Claeskens, G. (2024). Parametric programming-based approximate selective inference for adaptive lasso, adaptive elastic net and group lasso. Journal of Statistical Computation and Simulation , pages 1--24
2024
-
[33]
F., Drovandi, C
Price, L. F., Drovandi, C. C., Lee, A., and Nott, D. J. (2018). Bayesian synthetic likelihood. Journal of Computational and Graphical Statistics , 27(1):1--11
2018
-
[34]
Rasines, D. G. and Young, G. A. (2023). Splitting strategies for post-selection inference. Biometrika , 110(3):597--614
2023
-
[35]
A., Lander, E
Regev, A., Teichmann, S. A., Lander, E. S., Amit, I., Benoist, C., Birney, E., Bodenmiller, B., Campbell, P., Carninci, P., Clatworthy, M., et al. (2017). The human cell atlas. elife , 6:e27041
2017
-
[36]
A., Gennert, D., Schier, A
Satija, R., Farrell, J. A., Gennert, D., Schier, A. F., and Regev, A. (2015). Spatial reconstruction of single-cell gene expression data. Nature Biotechnology , 33:495--502
2015
-
[37]
Stuart, T., Butler, A., Hoffman, P., Hafemeister, C., Papalexi, E., III, W. M. M., Hao, Y., Stoeckius, M., Smibert, P., and Satija, R. (2019). Comprehensive integration of single-cell data. Cell , 177:1888--1902
2019
-
[38]
Thomas, O., Dutta, R., Corander, J., Kaski, S., and Gutmann, M. U. (2022). Likelihood-free inference by ratio estimation. Bayesian Analysis , 17(1):1--31
2022
-
[39]
and Taylor, J
Tian, X. and Taylor, J. (2018). Selective inference with a randomized response. The Annals of Statistics , 46(2):679--710
2018
-
[40]
and Wang, P
Xie, M.-g. and Wang, P. (2022). Repro samples method for finite-and large-sample inferences. arXiv preprint arXiv:2206.06421
2022 arXiv
-
[41]
and Fithian, W
Zrnic, T. and Fithian, W. (2024). Locally simultaneous inference. The Annals of Statistics , 52(3):1227--1253
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.