Pith. sign in

REVIEW 3 major objections 4 minor 109 references

Direct Prediction Set Minimization via Bilevel Conformal Classifier Training

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read The paper claims that conformal prediction sets can be trained to be smaller by solving a bilevel problem that learns the quantile threshold, with an $O(1/\sqrt{n})$ learning bound and a 20.46% average reduction in set size.

desk verdict A genuinely novel conformal-training formulation whose O(1/√n) theory only applies to the exact lower-level solution, not to the stochastic algorithm that actually runs. read the letter →

arxiv 2506.06599 v1 pith:4B6Y22HZ submitted 2025-06-07 cs.LG stat.ML

classification cs.LGstat.ML
keywords conformalpredictionsetsizebileveloptimizationquantileregressiontraininguncertaintyquantificationlearningboundmini-batch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to make conformal prediction sets smaller by training the underlying classifier for that goal, not only calibrating it afterwards. It reformulates conformal training as a bilevel optimization problem in which the upper level minimizes the average prediction-set size and the lower level learns the needed quantile of non-conformity scores by quantile regression. The paper's central theoretical claim is that this achieves a learning bound of $O(1/\sqrt{n})$ in the number of training samples, while prior stochastic-approximation methods have a bound of at least $\Omega(1/s)$ for batch size $s$. If the claim holds, conformal classifiers can be trained end-to-end with the same sample-complexity rate as ordinary supervised learning, and the reported experiments put the average prediction-set-size reduction at 20.46% over the best baseline.

What carries the argument

The machine carrying the argument is a bilevel objective that couples classifier $f$ and a scalar quantile $q$. The lower level minimizes the empirical pinball loss $\hat L_{QR}(f,q)=\frac1n\sum_i \rho_\alpha(q,S_f(X_i,Y_i))$, whose minimizer $q^*_f$ is a $(1-\alpha)$-quantile of conformity scores; the upper level minimizes the classification loss plus the smoothed prediction-set-size loss $\hat L_c^{DM}(f,q)$ conditioned on that $q$. Explicitly parameterizing $q$ decouples the threshold from batch randomness. The proof chain uses concentration of the empirical CDF to show $|Q_f-q^*_f|\le \tilde O(1/\sqrt{n})$, Lipschitzness of the smoothed set-size loss in $q$, and a H\"olderian error bound for the QR loss to connect the penalized problem's solution to the original bilevel problem.

What would settle it

Run DPSM's Algorithm 1 at fixed batch size $s$ with increasing training sizes $n$ and measure the gap between training and test average set size (or between learned $q_t$ and the dataset-level quantile); if that gap decays like $\tilde O(1/\sqrt{s})$ rather than $\tilde O(1/\sqrt{n})$, or if $q_t$ fails to approach the dataset-level quantile, the bound does not describe the implemented method.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that conformal training does not have to estimate the quantile threshold from whatever mini-batch happens to be drawn. By making the quantile an explicit scalar $q$ and training it with the pinball loss in a lower-level subproblem while the upper level minimizes a smoothed average prediction-set size, the alignment loss becomes a function of a learned threshold. Under the paper's bi-Lipschitz assumption on conformity scores, Theorem 4.1 bounds the gap between this implicit conformal loss and the population conformal loss by $\tilde O(1/\sqrt{n})$ with probability at least $1-\delta$, independent of batch size. Experiments on CIFAR-100, Caltech-101, and iNaturalist report a 20.46% average reduction in prediction set size over the best prior conformal training baseline at 90% coverage.

Load-bearing premise

The load-bearing premise is that the stochastic gradient updates in Algorithm 1 drive the scalar quantile to the exact lower-level minimizer $q^*_f$; the paper proves its $O(1/\sqrt{n})$ bound only for that exact minimizer and leaves convergence of the implemented update as an open challenge.

Editorial extensions

If this is right

  • Conformal training can match the $O(1/\sqrt{n})$ generalization rate of standard empirical risk minimization rather than the $\Omega(1/s)$ rate of batch-quantile methods.
  • The learned-quantile formulation gives smaller prediction sets at the same marginal coverage: 20.46% average reduction across CIFAR-100, Caltech-101, and iNaturalist with ResNet and DenseNet at $\alpha=0.1$.
  • The quantile estimation error of DPSM shrinks toward zero during training, giving a tighter empirical learning bound than ConfTr.
  • The method applies across HPS, APS, and RAPS scoring functions, with the largest gains on Caltech-101 and the HPS score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: the theorem's exact-minimizer assumption leaves open whether the SGD-updated $q_t$ enjoys the same rate; a convergence proof for Algorithm 1 would close the gap and is the natural next step.
  • Extension: the same bilevel template could be applied to other conformal objectives (conditional coverage, conformal risk control) by swapping the smoothed set-size loss in the upper level while retaining the learned quantile.
  • Extension: if the quantile update does converge at a batch-dependent rate, DPSM's practical gain may vary with batch size in ways the current bound does not capture; measuring the training-test gap at fixed $n$ and varying $s$ would test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes DPSM, a bilevel optimization method for training deep classifiers to minimize the expected size of conformal prediction sets. The upper-level objective is a differentiable prediction-set-size loss evaluated at a quantile q of the non-conformity scores, and the lower-level problem learns q by minimizing the pinball loss on the training scores. The authors prove a learning bound of O(1/√n) for the exact lower-level minimizer q*_f, contrasting with an Ω(1/s) lower bound they derive for prior stochastic-approximation (SA) conformal training methods. They present a stochastic first-order algorithm (Algorithm 1) that alternately updates the classifier and q, and they report experiments on CIFAR-100, Caltech-101, and iNaturalist showing an average 20.46% reduction in prediction set size over the best baseline.

Significance. If the O(1/√n) bound were matched to the implemented algorithm, the paper would be a solid contribution to conformal training, since prior analyses degrade with batch size as O(1/√s). The bilevel formulation with quantile regression is a natural way to decouple quantile estimation from mini-batch noise, and the paper is transparent about the open optimization question, which is methodologically honest. However, the current disconnect between Theorem 4.1 and Algorithm 1 prevents the theoretical claim from supporting the method that produced the empirical results, and the empirical headline is not uniform across settings. With these issues addressed, the paper could become a valuable reference for prediction-set-efficient classifier training.

major comments (3)
  1. [§4.2, Algorithm 1 and Theorem 4.1] The learning bound in Theorem 4.1 applies to \bar{L}_c^DM(f) = \hat{L}_c^DM(f, q*_f), where q*_f is an exact minimizer of the full-training-set pinball loss. Algorithm 1 (lines 9–10) updates q by a single stochastic gradient step on a mini-batch pinball loss, and the paper explicitly states in Section 4.2 that proving convergence of such an algorithm is an open challenge left for future work. Therefore the O(1/√n) bound is not established for the implemented method; the quantity that matters for the trained f_T and last iterate q_T is |\hat{L}_c^DM(f_T, q_T) - L_c(f_T)|, which includes the optimization error in q. In addition, the penalty-based reformulation (10) and Corollary 4.6 are not connected to Algorithm 1, since Algorithm 1 does not optimize the penalized objective. The authors should either provide a convergence guarantee for the q-updates (e.g., a bound on |q_T - q*_f|), revise the theorem to bound the actual quantity including the optimization error, or modify the algorithm to provably solve the lower-level problem.
  2. [Appendix B.2, proof of Theorem 3.5] The lower-bound proof of Theorem 3.5 expands (\hat{L}_c^SA(f) - L_c(f))^2 and lower-bounds the cross term 2(E[\hat{ℓ}(f,\hat{q}_f)] - \hat{ℓ}(f,E[\hat{q}_f]))(\hat{ℓ}(f,E[\hat{q}_f]) - ℓ(f,Q_f)) by a positive quantity using Lemma B.3. However, since \hat{ℓ}(f,·) is μ-strongly concave (Assumption 3.2), Jensen's inequality gives \hat{ℓ}(f,E[\hat{q}_f]) - E[\hat{ℓ}(f,\hat{q}_f)] ≥ 0, so E[\hat{ℓ}(f,\hat{q}_f)] - \hat{ℓ}(f,E[\hat{q}_f]) ≤ 0. The sign of this factor is opposite to what the proof assumes, making the product with the nonnegative factor from Lemma B.4 nonpositive. As stated, the proof does not establish the claimed Ω(1/s) lower bound; it needs to control the sign of the total deviation or use a different decomposition.
  3. [Table 1 and Appendix F.1] The headline reduction of 20.46% is an average over all settings and is driven by the Caltech-101 results (e.g., 59.17% reduction for DenseNet with APS). On several settings DPSM is worse than the best baseline: CIFAR-100 DenseNet with APS is 2.64 vs 2.41 for CUT (8.71% increase), iNaturalist ResNet with APS is 79.43 vs 79.00 (0.54% increase), and additional increases appear for RAPS in Tables 5–6 (e.g., CIFAR-100 DenseNet 8.55% increase, iNaturalist ResNet 0.08% increase). The abstract's claim that DPSM 'significantly outperforms' the best baseline is therefore not supported uniformly. The authors should report per-setting improvements with statistical significance and either temper the abstract or restrict the claim to the settings where the improvement is consistent.
minor comments (4)
  1. [Table 5] The entry for iNaturalist ResNet reads '↑-0.54%', which is self-contradictory; it should be either '↑0.54%' or '-0.54%' depending on the intended meaning.
  2. [§4.2] There is a typo in 'Lipscthiz Hessians'; it should be 'Lipschitz Hessians'.
  3. [Figure 3 and Appendix F.1] The notation bQn f is used without a formal definition; it is introduced in the text of Figure 3 but should be defined at first use in Section 3 or the caption.
  4. [§5.2, Figure 2] The convergence justification is based only on CIFAR-100 with DenseNet (and ResNet in the appendix); a statement that this is a representative configuration rather than evidence for all datasets would be helpful.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: DPSM's learning bound is a self-contained concentration result, but it is not yet matched to the implemented stochastic algorithm.

full rationale

The paper's central theoretical claim, Theorem 4.1, bounds |\bar L^{DM}_c(f) - L_c(f)| where \bar L^{DM}_c(f) is the empirical soft set size evaluated at the exact pinball-loss minimizer q*_f and L_c(f) is the population conformal loss at the true quantile Q_f. The proof in Appendix C is self-contained: Lemma C.2 shows |q*_f - Q_f| ≤ O~(1/√n) via Chernoff bounds on the empirical CDF and the quantile property of the QR minimizer, Lemma C.3 gives Lipschitz continuity of the soft set size in q, and Lemma C.4 is a standard Hoeffding generalization bound. This is a genuine concentration result, not a restatement of fitted values and not a consequence of the authors' own prior results. The empirical 20.46% reduction is an experimental comparison, not a fitted quantity renamed as a prediction. The self-citations (e.g., Ghosh et al. 2023a; Shi et al. 2024; Shahrokhi et al. 2025; Yuan et al. 2019) appear only as background or related-work references and are not load-bearing for the derivation. The paper itself flags a real limitation: Theorem 4.1 applies to the exact lower-level minimizer q*_f, while Algorithm 1 updates q by stochastic gradient steps and Section 4.2 states that proving convergence 'is non-trivial and an open challenge.' This mismatch is a correctness or completeness concern, not circularity, because the theorem does not presuppose the algorithm's convergence. Similarly, Figure 3(c) is explicitly described as an approximation of the learning bound via a train-test APSS gap, which does not constitute circular reasoning. Overall, the derivation chain is independent of its conclusions, so no circular step is exhibited.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim depends on the exact-solution assumption for the lower level, which is not proven for Algorithm 1, on bi-Lipschitz score order statistics, and on standard i.i.d. and concentration assumptions. No new particles, forces, dimensions, or mediators are introduced; q is a scalar quantile parameter, not an invented entity.

free parameters (4)
  • lambda_regularization = varies by dataset/model, e.g., 0.01, 0.05, 0.1, 1.0 (Table 3)
    Weights the conformal alignment loss against the classification loss in objective (8); selected by hyperparameter search for best predictive efficiency.
  • gamma_lower_level_lr = 0.001 to 0.1 (Table 3)
    Learning rate for the stochastic quantile updates in Algorithm 1; no theoretical convergence guarantee is provided.
  • tau_Sigmoid = not reported in main text
    Temperature of the smoothed indicator that approximates prediction set size; changes the loss surface and the Lipschitz constant K/(4*tau).
  • RAPS_regularization = lambda_RAPS=0.01, k_reg=5
    Used only for RAPS-scored evaluation in Table 6, not for DPSM training; affects the reported set sizes for that scoring function.
assumptions (6)
  • domain assumption Assumption 3.1: conformity scores S_(j) are bi-Lipschitz in normalized rank j/n
    Used in Theorem 3.5, Lemma C.2, and Theorem 4.1; verified empirically only on CIFAR-100 with DenseNet and HPS scores (Figures 4a and 10a).
  • domain assumption Assumption 3.2: smoothed conformal loss b_l(f,q) is locally mu-strongly concave in q around E[bq_f]
    Needed for the Omega(1/s) lower bound on SA methods in Theorem 3.5; verified empirically only near coverage 0.9 (Figures 4b and 10b).
  • domain assumption No ties in non-conformity scores
    Ensures unique empirical quantiles and supports the HEB proof in Lemma 4.5; APS and RAPS break ties by uniform randomization.
  • domain assumption IID or exchangeable samples for training and calibration
    Standard conformal prediction assumption; required by Lemma C.4 and by the coverage guarantee in equation (1).
  • standard math Bounded conformity scores and standard concentration inequalities (Hoeffding, Chernoff, Chebyshev)
    Used to obtain the O(1/sqrt n) and O(1/sqrt s) rates in Lemmas C.2, C.4, and B.5.
  • ad hoc to paper The stochastic q-updates in Algorithm 1 converge to the exact lower-level minimizer q*_f
    This is the unproven link between Theorem 4.1 and the implemented algorithm; the paper explicitly leaves this challenge for future work in Section 4.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Direct Prediction Set Minimization via Bilevel Conformal Classifier Training." pith.science (2026). https://pith.science/paper/4B6Y22HZ

@misc{pith2026250606599,
  author       = {Pith},
  title        = {Pith review of: Direct Prediction Set Minimization via Bilevel Conformal Classifier Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4B6Y22HZ}},
  note         = {Machine review of arXiv:2506.06599}
}
abstract

Conformal prediction (CP) is a promising uncertainty quantification framework which works as a wrapper around a black-box classifier to construct prediction sets (i.e., subset of candidate classes) with provable guarantees. However, standard calibration methods for CP tend to produce large prediction sets which makes them less useful in practice. This paper considers the problem of integrating conformal principles into the training process of deep classifiers to directly minimize the size of prediction sets. We formulate conformal training as a bilevel optimization problem and propose the {\em Direct Prediction Set Minimization (DPSM)} algorithm to solve it. The key insight behind DPSM is to minimize a measure of the prediction set size (upper level) that is conditioned on the learned quantile of conformity scores (lower level). We analyze that DPSM has a learning bound of $O(1/\sqrt{n})$ (with $n$ training samples), while prior conformal training methods based on stochastic approximation for the quantile has a bound of $\Omega(1/s)$ (with batch size $s$ and typically $s \ll \sqrt{n}$). Experiments on various benchmark datasets and deep models show that DPSM significantly outperforms the best prior conformal training baseline with $20.46\%\downarrow$ in the prediction set size and validates our theory.

Figures

Figures reproduced from arXiv: 2506.06599 by the authors.

Figure 1
Figure 1. Box plots of coverage (Top row) and APSS (Bottom row) of all methods using DenseNet and HPS score. DPSM achieves significantly smaller prediction set size while maintaining the valid coverage. (a) Upper loss (b) Lower loss (c) Conformal loss optimization gap (d) QR loss optimization gap [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Justification experiments for the convergence of DPSM on CIFAR-100 using DenseNet and HPS score. (a) Upper level loss (i.e., a combination of classification loss and conformal alignment loss); (b) Lower level loss (i.e., QR loss); (c) Opti￾mization gap of conformal loss, defined as the difference between conformal losses using learned batch-level quantiles and dataset￾level quantiles on the training set; (d) Optimiz… view at source ↗
Figure 3
Figure 3. Justification experiments for the learning bound of DPSM on CIFAR-100 using DenseNet and HPS score. (a) Estimation error between the Qbn f (i.e., the dataset-level quantiles on training data) and qbf (i.e., either the batch-level quantiles evaluated in ConfTr or the learned ones in DPSM); (b) Average soft set size (approximated using Sigmoid function) of DPSM and ConfTr; (c) Approximated learning error comparison be… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Assumption verification on CIFAR-100 using DenseNet and HPS score on the calibration dataset. (a) HPS scores over corresponding normalized index produced by ConfTr. The x-axis is the normalized order, the y-axis is the corresponding conformity score; (b) The soft set s…
Figure 5
Figure 5. Figure 5: Box plots of coverage (Top row) and APSS (Bottom row) of all methods using ResNet and HPS score. DPSM achieves significantly smaller prediction set size while maintaining the valid coverage. (a) Upper loss (b) Lower loss (c) Conformal loss optimization gap (d) QR loss …
Figure 6
Figure 6. Figure 6: Justification experiments for the convergence of DPSM on CIFAR-100 using ResNet and HPS score. (a) Upper level loss (i.e., a combination of classification loss and conformal alignment loss); (b) Lower level loss (i.e., QR loss); (c) Optimization gap of conformal loss, …
Figure 7
Figure 7. Figure 7: Justification experiments for effectiveness of DPSM trained with 40 epochs on CIFAR-100 using DenseNet and HPS score. (a) Upper level loss (i.e., a combination of classification loss and conformal alignment loss) in DPSM; (b) Lower level loss (i.e., QR loss) in DPSM; (…
Figure 8
Figure 8. Figure 8: Justification experiments for effectiveness of DPSM trained with 40 epochs on CIFAR-100 using ResNet and HPS score. (a) Upper level loss (i.e., a combination of classification loss and conformal alignment loss) in DPSM; (b) Lower level loss (i.e., QR loss) in DPSM; (c)…
Figure 9
Figure 9. Figure 9: Justification experiments for the learning bound of DPSM on CIFAR-100 using ResNet and HPS score. (a) Estimation error between the Qbn f (dataset-level quantiles on training data) and qbf (batch-level quantiles evaluated in ConfTr or learned in DPSM); (b) Average soft …
Figure 10
Figure 10. Figure 10: Verification studies on CIFAR-100 with ResNet model using HPS scoring function on calibration dataset. (a): HPS scores over corresponding normalized order produced by ConfTr. The x-axis is the normalized order, the y-axis is the corresponding conformity score; (b): Th…
Figure 11
Figure 11. Figure 11: Class conditional coverage and class-wise prediction set size of all methods on CIFAR-100 with DenseNet and HPS. To better compare the class conditional coverage, we compare the class-conditional coverage between DPSM and 3 baselines in (a), (b) and (c) separately. DP…
Figure 12
Figure 12. Figure 12: Class conditional coverage and class-wise prediction set size of all methods on CIFAR-100 with ResNet and HPS. To better compare the class conditional coverage, we compare the class-conditional coverage between DPSM and 3 baselines in (a), (b) and (c) separately. DPSM…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

109 extracted references · 58 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    R., et al

    Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76: 0 243--297, 2021

  3. [3]

    Angelopoulos, A. N. and Bates, S. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021

  4. [4]

    N., Bates, S., Jordan, M., and Malik, J

    Angelopoulos, A. N., Bates, S., Jordan, M., and Malik, J. Uncertainty sets for image classifiers using conformal prediction. In International Conference on Learning Representations, 2021

  5. [5]

    N., Kohli, A

    Angelopoulos, A. N., Kohli, A. P., Bates, S., Jordan, M., Malik, J., Alshaabi, T., Upadhyayula, S., and Romano, Y. Image-to-image regression with distribution-free uncertainty quantification and applications in imaging. In International Conference on Machine Learning, pp.\ 717--730. PMLR, 2022

  6. [6]

    N., Candes, E., and Tibshirani, R

    Angelopoulos, A. N., Candes, E., and Tibshirani, R. Conformal pid control for time series prediction. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

  7. [7]

    N., Bates, S., Fisch, A., Lei, L., and Schuster, T

    Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L., and Schuster, T. Conformal risk control. In The Twelfth International Conference on Learning Representations, 2024

  8. [8]

    On the utility of prediction sets in human-ai teams

    Babbar, V., Bhatt, U., and Weller, A. On the utility of prediction sets in human-ai teams. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI-22), 2022

Show all 109 references
  1. [9]

    F., Candes, E

    Barber, R. F., Candes, E. J., Ramdas, A., and Tibshirani, R. J. Predictive inference with the jackknife+. The Annals of Statistics, 49 0 (1): 0 486--507, 2021

  2. [10]

    Distribution-free, risk-controlling prediction sets

    Bates, S., Angelopoulos, A., Lei, L., Malik, J., and Jordan, M. Distribution-free, risk-controlling prediction sets. Journal of the ACM (JACM), 68 0 (6): 0 1--34, 2021

  3. [11]

    A survey on bilevel optimization under uncertainty

    Beck, Y., Ljubi \'c , I., and Schmidt, M. A survey on bilevel optimization under uncertainty. European Journal of Operational Research, 311 0 (2): 0 401--426, 2023

  4. [12]

    The need for uncertainty quantification in machine-assisted medical decision making

    Begoli, E., Bhattacharya, T., and Kusnezov, D. The need for uncertainty quantification in machine-assisted medical decision making. Nature Machine Intelligence, 1 0 (1): 0 20--23, 2019

  5. [13]

    Optimized conformal classification using gradient descent approximation

    Bellotti, A. Optimized conformal classification using gradient descent approximation. arXiv preprint arXiv:2105.11255, 2021

  6. [14]

    Improved online conformal prediction via strongly adaptive online learning

    Bhatnagar, A., Wang, H., Xiong, C., and Bai, Y. Improved online conformal prediction via strongly adaptive online learning. In International Conference on Machine Learning, pp.\ 2337--2363. PMLR, 2023

  7. [15]

    P., Peypouquet, J., and Suter, B

    Bolte, J., Nguyen, T. P., Peypouquet, J., and Suter, B. W. From error bounds to the complexity of first-order descent methods for convex functions. Mathematical Programming, 165: 0 471--507, 2017

  8. [16]

    Nonsmooth implicit differentiation for machine-learning and optimization

    Bolte, J., Le, T., Pauwels, E., and Silveti-Falls, T. Nonsmooth implicit differentiation for machine-learning and optimization. Advances in neural information processing systems, 34: 0 13537--13549, 2021

  9. [17]

    Automatic differentiation of nonsmooth iterative algorithms

    Bolte, J., Pauwels, E., and Vaiter, S. Automatic differentiation of nonsmooth iterative algorithms. Advances in Neural Information Processing Systems, 35: 0 26404--26417, 2022

  10. [18]

    Coresets via bilevel optimization for continual learning and streaming

    Borsos, Z., Mutny, M., and Krause, A. Coresets via bilevel optimization for continual learning and streaming. Advances in neural information processing systems, 33: 0 14879--14890, 2020

  11. [19]

    Data summarization via bilevel optimization

    Borsos, Z., Mutn \`y , M., Tagliasacchi, M., and Krause, A. Data summarization via bilevel optimization. Journal of Machine Learning Research, 25 0 (73): 0 1--53, 2024

  12. [20]

    Cauchois, M., Gupta, S., and Duchi, J. C. Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction. Journal of machine learning research, 22 0 (81): 0 1--42, 2021

  13. [21]

    On finding small hyper-gradients in bilevel optimization: Hardness results and improved analysis

    Chen, L., Xu, J., and Zhang, J. On finding small hyper-gradients in bilevel optimization: Hardness results and improved analysis. In The Thirty Seventh Annual Conference on Learning Theory, pp.\ 947--980. PMLR, 2024 a

  14. [22]

    Penalty-based methods for simple bilevel optimization under h\"olderian error bounds

    Chen, P., Shi, X., Jiang, R., and Wang, J. Penalty-based methods for simple bilevel optimization under h\"olderian error bounds. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 b

  15. [23]

    Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems

    Chen, T., Sun, Y., and Yin, W. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021

  16. [24]

    Optimal algorithms for stochastic bilevel optimization under relaxed smoothness conditions

    Chen, X., Xiao, T., and Balasubramanian, K. Optimal algorithms for stochastic bilevel optimization under relaxed smoothness conditions. Journal of Machine Learning Research, 25 0 (151): 0 1--51, 2024 c

  17. [25]

    Adversarially trained actor critic for offline reinforcement learning

    Cheng, C.-A., Xie, T., Jiang, N., and Agarwal, A. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pp.\ 3852--3878. PMLR, 2022

  18. [26]

    C., Sui, Y., Kumar, B., and Vouitsis, N

    Cresswell, J. C., Sui, Y., Kumar, B., and Vouitsis, N. Conformal prediction sets improve human decision making. In Forty-first International Conference on Machine Learning, 2024

  19. [27]

    A framework for bilevel optimization that enables stochastic and global variance reduction algorithms

    Dagr \'e ou, M., Ablin, P., Vaiter, S., and Moreau, T. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. Advances in Neural Information Processing Systems, 35: 0 26698--26710, 2022

  20. [28]

    Online calibrated and conformal prediction improves bayesian optimization

    Deshpande, S., Marx, C., and Kuleshov, V. Online calibrated and conformal prediction improves bayesian optimization. In International Conference on Artificial Intelligence and Statistics, pp.\ 1450--1458. PMLR, 2024

  21. [29]

    Ding, T., Angelopoulos, A., Bates, S., Jordan, M., and Tibshirani, R. J. Class-conditional conformal prediction with many classes. Advances in Neural Information Processing Systems, 36, 2024

  22. [30]

    Generic methods for optimization-based modeling

    Domke, J. Generic methods for optimization-based modeling. In Artificial Intelligence and Statistics, pp.\ 318--326. PMLR, 2012

  23. [31]

    Training uncertainty-aware classifiers with conformalized deep learning

    Einbinder, B.-S., Romano, Y., Sesia, M., and Zhou, Y. Training uncertainty-aware classifiers with conformalized deep learning. Advances in Neural Information Processing Systems, 35: 0 22380--22395, 2022

  24. [32]

    Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories

    Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Pattern Recognition Workshop, 2004

  25. [33]

    Efficient conformal prediction via cascaded inference with expanded admission

    Fisch, A., Schuster, T., Jaakkola, T., and Barzilay, R. Efficient conformal prediction via cascaded inference with expanded admission. In Proceedings of The Tenth International Conference on Learning Representations, 2021 a

  26. [34]

    Few-shot conformal prediction with auxiliary tasks

    Fisch, A., Schuster, T., Jaakkola, T., and Barzilay, R. Few-shot conformal prediction with auxiliary tasks. In International Conference on Machine Learning, pp.\ 3329--3339. PMLR, 2021 b

  27. [35]

    and Wang, M

    Ghadimi, S. and Wang, M. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246, 2018

  28. [36]

    Ghosh, S., Belkhouja, T., Yan, Y., and Doppa, J. R. Improving uncertainty quantification of deep classifiers via neighborhood conformal prediction: Novel algorithm and theoretical analysis. In AAAI , pp.\ 7722--7730, 2023 a

  29. [37]

    Probabilistically robust conformal prediction

    Ghosh, S., Shi, Y., Belkhouja, T., Yan, Y., Doppa, J., and Jones, B. Probabilistically robust conformal prediction. In Uncertainty in Artificial Intelligence, pp.\ 681--690. PMLR, 2023 b

  30. [38]

    and Candes, E

    Gibbs, I. and Candes, E. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34: 0 1660--1672, 2021

  31. [39]

    and Cand \`e s, E

    Gibbs, I. and Cand \`e s, E. J. Conformal inference for online prediction with arbitrary distribution shifts. Journal of Machine Learning Research, 25 0 (162): 0 1--36, 2024

  32. [40]

    J., and Cand \`e s, E

    Gibbs, I., Cherian, J. J., and Cand \`e s, E. J. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305.12616, 2023

  33. [41]

    An accelerated algorithm for stochastic bilevel optimization under unbounded smoothness

    Gong, X., Hao, J., and Liu, M. An accelerated algorithm for stochastic bilevel optimization under unbounded smoothness. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 a

  34. [42]

    A nearly optimal single loop algorithm for stochastic bilevel optimization under unbounded smoothness

    Gong, X., Hao, J., and Liu, M. A nearly optimal single loop algorithm for stochastic bilevel optimization under unbounded smoothness. In ICML, 2024 b

  35. [43]

    Localized conformal prediction: A generalized inference framework for conformal prediction

    Guan, L. Localized conformal prediction: A generalized inference framework for conformal prediction. Biometrika, 110 0 (1): 0 33--50, 2023

  36. [44]

    Bilevel coreset selection in continual learning: A new formulation and algorithm

    Hao, J., Ji, K., and Liu, M. Bilevel coreset selection in continual learning: A new formulation and algorithm. Advances in Neural Information Processing Systems, 36, 2024

  37. [45]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

  38. [46]

    Mask r-cnn

    He, K., Gkioxari, G., Doll \'a r, P., and Girshick, R. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp.\ 2961--2969, 2017

  39. [47]

    A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic

    Hong, M., Wai, H.-T., Wang, Z., and Yang, Z. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic. SIAM Journal on Optimization, 33 0 (1): 0 147--180, 2023. doi:10.1137/20M1387341

  40. [48]

    Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 4700--4708, 2017

  41. [49]

    Conformal prediction for deep classifier via label ranking

    Huang, J., Xi, H., Zhang, L., Yao, H., Qiu, Y., and Wei, H. Conformal prediction for deep classifier via label ranking. arXiv preprint arXiv:2310.06430, 2023 a

  42. [50]

    Achieving linear speedup in non-iid federated bilevel learning

    Huang, M., Zhang, D., and Ji, K. Achieving linear speedup in non-iid federated bilevel learning. In International Conference on Machine Learning, pp.\ 14039--14059. PMLR, 2023 b

  43. [51]

    Bilevel optimization: Convergence analysis and enhanced design

    Ji, K., Yang, J., and Liang, Y. Bilevel optimization: Convergence analysis and enhanced design. In International conference on machine learning, pp.\ 4882--4892. PMLR, 2021

  44. [52]

    and Lu, H

    Kawaguchi, K. and Lu, H. Ordered sgd: A new stochastic optimization framework for empirical risk minimization. In International Conference on Artificial Intelligence and Statistics, pp.\ 669--679. PMLR, 2020

  45. [53]

    T., Wang, Z., and Yang, Z

    Khanduri, P., Zeng, S., Hong, M., Wai, H. T., Wang, Z., and Yang, Z. A near-optimal algorithm for stochastic bilevel optimization via double-momentum. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021

  46. [54]

    J., and Hassani, H

    Kiyani, S., Pappas, G. J., and Hassani, H. Conformal prediction with learned features. In Forty-first International Conference on Machine Learning, 2024 a

  47. [55]

    J., and Hassani, H

    Kiyani, S., Pappas, G. J., and Hassani, H. Length optimization in conformal prediction. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 b

  48. [56]

    Quantile regression, volume 38

    Koenker, R. Quantile regression, volume 38. Cambridge university press, 2005

  49. [57]

    and Bassett Jr, G

    Koenker, R. and Bassett Jr, G. Regression quantiles. Econometrica: journal of the Econometric Society, pp.\ 33--50, 1978

  50. [58]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009

  51. [59]

    Ask me anything: Dynamic memory networks for natural language processing

    Kumar, A., Irsoy, O., Ondruska, P., Iyyer, M., Bradbury, J., Gulrajani, I., Zhong, V., Paulus, R., and Socher, R. Ask me anything: Dynamic memory networks for natural language processing. In International conference on machine learning, pp.\ 1378--1387. PMLR, 2016

  52. [60]

    Kwon, J., Kwon, D., Wright, S., and Nowak, R. D. A fully first-order method for stochastic bilevel optimization. In International Conference on Machine Learning, pp.\ 18083--18113. PMLR, 2023

  53. [61]

    Kwon, J., Kwon, D., Wright, S., and Nowak, R. D. On penalty methods for nonconvex bilevel optimization and first-order stochastic approximation. In The Twelfth International Conference on Learning Representations, 2024

  54. [62]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.\ 1207--1216, Stanford, CA, 2000. Morgan Kaufmann

  55. [63]

    Distribution-free prediction sets

    Lei, J., Robins, J., and Wasserman, L. Distribution-free prediction sets. Journal of the American Statistical Association, 108 0 (501): 0 278--287, 2013

  56. [64]

    BOME ! bilevel optimization made easy: A simple first-order approach

    Liu, B., Ye, M., Wright, S., Stone, P., and qiang liu. BOME ! bilevel optimization made easy: A simple first-order approach. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022

  57. [65]

    A generic first-order algorithmic framework for bi-level programming beyond lower-level singleton

    Liu, R., Mu, P., Yuan, X., Zeng, S., and Zhang, J. A generic first-order algorithmic framework for bi-level programming beyond lower-level singleton. In International conference on machine learning, pp.\ 6305--6315. PMLR, 2020

  58. [66]

    Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond

    Liu, R., Gao, J., Zhang, J., Meng, D., and Lin, Z. Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 0 (12): 0 10045--10067, 2021

  59. [67]

    Liu, Z., Cui, Y., Yan, Y., Xu, Y., Ji, X., Liu, X., and Chan, A. B. The pitfalls and promise of conformal inference under adversarial attacks. In International Conference on Machine Learning, pp.\ 30908--30928. PMLR, 2024

  60. [68]

    N., and Pomerantz, S

    Lu, C., Angelopoulos, A. N., and Pomerantz, S. Improving trustworthiness of ai disease severity rating in medical imaging with ordinal conformal prediction sets. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp.\ 545--554. Springer, 2022 a

  61. [69]

    Fair conformal predictors for applications in medical imaging

    Lu, C., Lemay, A., Chang, K., H \"o bel, K., and Kalpathy-Cramer, J. Fair conformal predictors for applications in medical imaging. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 12008--12016, 2022 b

  62. [70]

    and Mei, S

    Lu, Z. and Mei, S. First-order penalty methods for bilevel optimization. SIAM Journal on Optimization, 34 0 (2): 0 1937--1969, 2024

  63. [71]

    and Luschi, C

    Masters, D. and Luschi, C. Revisiting small batch training for deep neural networks. arXiv preprint arXiv:1804.07612, 2018

  64. [72]

    Foundations of machine learning

    Mohri, M., Rostamizadeh, A., and Talwalkar, A. Foundations of machine learning. MIT press, 2018

  65. [73]

    L., Canini, K

    Narayan, T., Wang, S. L., Canini, K. R., and Gupta, M. Expected pinball loss for quantile regression and inverse cdf estimation. Transactions on Machine Learning Research, 2024

  66. [74]

    I., Orenstein, P., Ramos, T., and Romano, J

    Oliveira, R. I., Orenstein, P., Ramos, T., and Romano, J. V. Split conformal prediction and non-exchangeable data. Journal of Machine Learning Research, 25 0 (225): 0 1--38, 2024

  67. [75]

    Error bounds in mathematical programming

    Pang, J.-S. Error bounds in mathematical programming. Mathematical Programming, 79 0 (1): 0 299--332, 1997

  68. [76]

    Hyperparameter optimization with approximate gradient

    Pedregosa, F. Hyperparameter optimization with approximate gradient. In International conference on machine learning, pp.\ 737--746. PMLR, 2016

  69. [77]

    Adaptive conformal inference by betting

    Podkopaev, A., Xu, D., and chih Lee, K. Adaptive conformal inference by betting. In Forty-first International Conference on Machine Learning, 2024

  70. [78]

    Conformalized quantile regression

    Romano, Y., Patterson, E., and Candes, E. Conformalized quantile regression. Advances in neural information processing systems, 32, 2019

  71. [79]

    Classification with valid and adaptive coverage

    Romano, Y., Sesia, M., and Candes, E. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems, 33: 0 3581--3591, 2020

  72. [80]

    and d'Aspremont, A

    Roulet, V. and d'Aspremont, A. Sharpness, restart and acceleration. Advances in Neural Information Processing Systems, 30, 2017

  73. [81]

    and Shtern, S

    Sabach, S. and Shtern, S. A first order method for solving convex bilevel optimization problems. SIAM Journal on Optimization, 27 0 (2): 0 640--660, 2017. doi:10.1137/16M105592X

  74. [82]

    Least ambiguous set-valued classifiers with bounded error levels

    Sadinle, M., Lei, J., and Wasserman, L. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, 114 0 (525): 0 223--234, 2019

  75. [83]

    and Romano, Y

    Sesia, M. and Romano, Y. Conformal prediction using conditional histograms. Advances in Neural Information Processing Systems, 34: 0 6304--6315, 2021

  76. [84]

    and Vovk, V

    Shafer, G. and Vovk, V. A tutorial on conformal prediction. Journal of Machine Learning Research, 9 0 (3), 2008

  77. [85]

    R., Yan, Y., Arnaoudova, V., and Doppa, J

    Shahrokhi, H., Roy, D. R., Yan, Y., Arnaoudova, V., and Doppa, J. R. Conformal prediction sets for deep generative models via reduction to conformal regression. CoRR, abs/2503.10512, 2025. doi:10.48550/ARXIV.2503.10512. URL https://doi.org/10.48550/arXiv.2503.10512

  78. [86]

    and Ben-David, S

    Shalev-Shwartz, S. and Ben-David, S. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014

  79. [87]

    and Chen, T

    Shen, H. and Chen, T. On penalty-based bilevel gradient descent method. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp.\ 30992--31015, 2023

  80. [88]

    and Chen, T

    Shen, H. and Chen, T. On penalty-based bilevel gradient descent method. Mathematical Programming (MAPR), to appear, 2025

  81. [89]

    Principled penalty-based methods for bilevel reinforcement learning and rlhf

    Shen, H., Yang, Z., and Chen, T. Principled penalty-based methods for bilevel reinforcement learning and rlhf. In Forty-first International Conference on Machine Learning

  82. [90]

    Conformal prediction for class-wise coverage via augmented label rank calibration

    Shi, Y., GHOSH, S., Belkhouja, T., Doppa, J., and Yan, Y. Conformal prediction for class-wise coverage via augmented label rank calibration. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

  83. [91]

    Straitouri, E., Wang, L., Okati, N., and Rodriguez, M. G. Improving expert predictions with conformal prediction. In International Conference on Machine Learning (ICML), 2023

  84. [92]

    T., Doucet, A., et al

    Stutz, D., Cemgil, A. T., Doucet, A., et al. Learning optimal conformal classifiers. arXiv preprint arXiv:2110.09192, 2021

  85. [93]

    The inaturalist species classification and detection dataset

    Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 8769--8778, 2018

  86. [94]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

  87. [95]

    Cross-conformal predictors

    Vovk, V. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence, 74 0 (1): 0 9--28, 2015

  88. [96]

    Machine-learning applications of algorithmic randomness

    Vovk, V., Gammerman, A., and Saunders, C. Machine-learning applications of algorithmic randomness. 1999

  89. [97]

    Algorithmic learning in a random world

    Vovk, V., Gammerman, A., and Shafer, G. Algorithmic learning in a random world. Springer Science & Business Media, 2005

  90. [98]

    Criteria of efficiency for conformal prediction

    Vovk, V., Fedorova, V., Nouretdinov, I., and Gammerman, A. Criteria of efficiency for conformal prediction. In Conformal and Probabilistic Prediction with Applications: 5th International Symposium, COPA 2016, Madrid, Spain, April 20-22, 2016, Proceedings 5, pp.\ 23--39. Springer, 2016

  91. [99]

    Provably robust conformal prediction with improved efficiency

    Yan, G., Romano, Y., and Weng, T.-W. Provably robust conformal prediction with improved efficiency. In The Twelfth International Conference on Learning Representations, 2024

  92. [100]

    Provably faster algorithms for bilevel optimization

    Yang, J., Ji, K., and Liang, Y. Provably faster algorithms for bilevel optimization. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021 a

  93. [101]

    and Fevens, T

    Yang, S. and Fevens, T. Uncertainty quantification and estimation in medical image classification. In Artificial Neural Networks and Machine Learning--ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14--17, 2021, Proceed...

  94. [102]

    and Lin, Q

    Yang, T. and Lin, Q. Rsg: Beating subgradient method without smoothness and strong convexity. Journal of Machine Learning Research, 19 0 (6): 0 1--33, 2018

  95. [103]

    Sim FBO : Towards simple, flexible and communication-efficient federated bilevel learning

    Yang, Y., Xiao, P., and Ji, K. Sim FBO : Towards simple, flexible and communication-efficient federated bilevel learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

  96. [104]

    Exact gap between generalization error and uniform convergence in random feature models

    Yang, Z., Bai, Y., and Mei, S. Exact gap between generalization error and uniform convergence in random feature models. In International Conference on Machine Learning, pp.\ 11704--11715. PMLR, 2021 b

  97. [105]

    Stagewise training accelerates convergence of testing error over sgd

    Yuan, Z., Yan, Y., Jin, R., and Yang, T. Stagewise training accelerates convergence of testing error over sgd. Advances in Neural Information Processing Systems, 32, 2019

  98. [106]

    An introduction to bilevel optimization: Foundations and applications in signal processing and machine learning

    Zhang, Y., Khanduri, P., Tsaknakis, I., Yao, Y., Hong, M., and Liu, S. An introduction to bilevel optimization: Foundations and applications in signal processing and machine learning. IEEE Signal Processing Magazine, 41 0 (1): 0 38--59, 2024

  99. [107]

    Bi-level offline policy optimization with limited exploration

    Zhou, W. Bi-level offline policy optimization with limited exploration. Advances in Neural Information Processing Systems, 36, 2024

  100. [108]

    Probabilistic bilevel coreset selection

    Zhou, X., Pi, R., Zhang, W., Lin, Y., Chen, Z., and Zhang, T. Probabilistic bilevel coreset selection. In International Conference on Machine Learning, pp.\ 27287--27302. PMLR, 2022

  101. [109]

    Predicate-conditional conformalized answer sets for knowledge graph embeddings

    Zhu, Y., Hern \'a ndez, D., He, Y., Ding, Z., Xiong, B., Kharlamov, E., and Staab, S. Predicate-conditional conformalized answer sets for knowledge graph embeddings. arXiv preprint arXiv:2505.16877, 2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.