Pith. sign in

REVIEW 3 major objections 4 minor 38 references

CLEAR: Calibrated Learning for Epistemic and Aleatoric Risk

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CLEAR calibrates aleatoric and epistemic uncertainty with two scaling parameters, cutting 95% interval width by 28.3% on average over single-component baselines while holding coverage.

desk verdict Solid incremental calibration paper with a big reproducible benchmark; trust the direction, not the NCIW headline numbers. read the letter →

arxiv 2507.08150 v4 pith:EJNZARR5 submitted 2025-07-10 stat.ML cs.LGstat.ME

Ilia Azizi , Juraj Bodik , Jakob Heiss , Bin Yu This is my paper · ORCID
classification stat.MLcs.LGstat.ME MSC 62G0862G0562F40
keywords uncertaintyquantificationpredictionintervalsconformalcalibrationaleatoricepistemicquantileregressionPCSframework
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes CLEAR, a calibration scheme for regression prediction intervals that treats aleatoric noise and epistemic model uncertainty as separate components, each with its own scale parameter, and learns the ratio between them from data. Its central claim is that two-parameter calibration produces intervals that are markedly narrower than individually calibrated aleatoric-only or epistemic-only baselines while keeping nominal coverage. Across 17 real datasets, CLEAR reports average normalized interval-width reductions of 28.3% and 17.5% over those two baselines, with the largest gains where the uncertainty sources are unbalanced. The practical payoff is that a practitioner does not have to fix the epistemic-aleatoric ratio by hand: the validation data decides which component dominates, and the interval widens in data-sparse regions where a purely aleatoric interval would undercover.

What carries the argument

The load-bearing object is the two-parameter interval $C(x) = [\hat f(x) - \gamma_1 \hat q_{\alpha/2}^{ale}(x) - \gamma_2 \hat q_{\alpha/2}^{epi}(x), \hat f(x) + \gamma_1 \hat q_{1-\alpha/2}^{ale}(x) + \gamma_2 \hat q_{1-\alpha/2}^{epi}(x)]$ with the constraint $\gamma_2 = \lambda \gamma_1$. The aleatoric pieces $\hat q^{ale}$ are fit by quantile regression on the residuals $Y_i - \hat f(X_i)$, bagged over bootstrap samples; the epistemic pieces $\hat q^{epi}$ are the distances from the ensemble median to the ensemble quantiles. Conformal calibration of $\gamma_1$ enforces marginal coverage, while $\lambda$ rebalances the two components so that scale and ratio are adjusted separately, which is what lets the method compensate when one uncertainty estimator is miscalibrated.

What would settle it

Run a synthetic experiment with known heteroskedastic noise, deliberately use a median estimator whose bias grows with distance from the training mass, and record CLEAR's conditional coverage as a function of $x$ on a dense grid; if coverage falls below the nominal level in the high-bias regions, the residual-based aleatoric assumption is the source of the failure.

Watch

Extended reading notes

Core claim

CLEAR's central discovery is that the two sources of predictive uncertainty should be calibrated jointly, with two degrees of freedom, instead of separately or with a fixed 1:1 ratio. The interval takes the form $C(x) = [\hat f(x) - \gamma_1 \hat q_{\alpha/2}^{ale}(x) - \gamma_2 \hat q_{\alpha/2}^{epi}(x), \hat f(x) + \gamma_1 \hat q_{1-\alpha/2}^{ale}(x) + \gamma_2 \hat q_{1-\alpha/2}^{epi}(x)]$, with $\gamma_2 = \lambda \gamma_1$; $\gamma_1$ is fixed by the standard conformal coverage calibration on a calibration set, and $\lambda$ is selected from a grid by minimizing quantile loss on validation data. Aleatoric quantiles are obtained by fitting quantile regression to the residuals $Y_i - \hat f(X_i)$ and taking medians over bootstrap members, while epistemic quantiles are the ensemble perturbation quantiles. The paper argues and demonstrates empirically that this adaptive reweighting repairs conditional coverage in data-sparse regions without sacrificing sharpness: average NCIW gains of 28.3% over the aleatoric baseline and 17.5% over the epistemic baseline, and similar gains when the components are deep ensembles and simultaneous quantile regression.

Load-bearing premise

The weakest load-bearing assumption is that the fitted quantiles of the residuals $Y_i - \hat f(X_i)$ equal the quantiles of the true aleatoric noise; if the ensemble median is biased or the residual spread does not match the true noise spread, the two scaling parameters can resize the interval but not reshape it, and conditional coverage in extrapolation regions can fail.

Editorial extensions

If this is right

  • CLEAR can wrap any pair of aleatoric and epistemic estimators, so the same calibration layer can sit on top of tree-based, deep ensemble, or quantile regression models.
  • The learned ratio $\lambda$ gives a data-driven statement about which uncertainty source dominates, which can guide active learning and data collection decisions.
  • Joint calibration is asymptotically no worse than calibrating either component alone, so the second parameter does not sacrifice long-run coverage guarantees.
  • With a separate calibration split, CLEAR inherits the finite-sample marginal coverage guarantee of split conformal prediction.
  • The reported benchmark gains (top performer on 15 of 17 datasets in the main configuration) mean the improvement is not confined to one model family.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension, already flagged in the paper, is to carry the same dual-parameter idea into classification, calibrating a confidence score and an ensemble-dispersion score separately.
  • Since $\lambda$ is estimated on finite validation data, small calibration samples will overfit the ratio; regularizing $\lambda$ toward 1 is a testable modification likely to help in low-data regimes.
  • The residual-based aleatoric step implies CLEAR is only as good as the median's shape; deliberately misspecifying the median so its bias varies with $x$ is a direct way to probe the limits of the additive interval form.
  • Under covariate shift, the ratio learned on the validation distribution may need re-estimation; the paper's sphere-of-increasing-radius simulation protocol is a ready-made template for testing how far $\lambda$ carries.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CLEAR, a calibration method that combines an aleatoric uncertainty estimator (quantile regression on residuals) and an epistemic uncertainty estimator (PCS ensembles) by jointly calibrating two parameters, γ1 and γ2 = λγ1, so as to maintain nominal coverage while reducing interval width. Across 17 real-world regression datasets, the abstract reports average NCIW improvements of 28.3% and 17.5% over the two individually calibrated baselines, with similar gains for Deep Ensembles and Simultaneous Quantile Regression. The paper also provides asymptotic conditional coverage theory, a conformalized variant with finite-sample marginal coverage guarantees, extensive appendix tables, and a case study on the Ames Housing data.

Significance. If the reported gains hold as stated, the work is a useful contribution to regression UQ: it demonstrates a simple, estimator-agnostic mechanism for reweighting aleatoric and epistemic components, and it ships a large empirical study (17 datasets, 10 seeds, three model variants, separate conformalized results) with code. The paper is unusually transparent about its main limitation—reusing Dval as Dcal in the default procedure—and it explicitly separates the conformalized variant. The residual-based aleatoric estimator is a practically motivated idea, and the empirical evidence that the joint calibration improves over both single-component baselines is directionally consistent across raw NIW, quantile loss, and AISL tables. The main weakness is that the headline width improvements are computed with a test-time calibration constant (NCIW), so the abstract's 'interval width' claim is not exactly what was measured; this is fixable by reporting raw-width aggregates in the main text and tempering the wording.

major comments (3)
  1. [Abstract and §3.3] The headline numbers 28.3% and 17.5% are NCIW averages, not raw interval-width improvements. NCIW applies a calibration constant ctest-cal chosen on the test set to enforce PICP ≥ 1 − α. This makes the comparison a post-hoc rescaled width comparison; a method with a validation-to-test calibration gap can be differentially rescaled, and the statement that ctest-cal ≈ 1 because methods are already calibrated does not remove the test-label dependence. The raw NIW tables in Appendix F.1 support the general direction, but the advertised magnitudes are not recomputed without ctest-cal. Please report aggregate raw NIW, quantile loss, and AISL for the PCS/ALEATORIC/ALEATORIC-R comparison in the main text and qualify the abstract accordingly.
  2. [Algorithm 1 and Appendix B.1] In the default configuration the algorithm sets Dcal = Dval and optimizes λ on the same dataset, so the finite-sample conformal coverage guarantee of Lemma B.2 does not apply. The paper discloses this in Appendix B.1 and provides a conformalized version in Appendix G, which is commendable. However, the abstract's 'while maintaining nominal coverage' is stated for the standard configuration, where coverage is only empirical. Please either make the conformalized variant the primary comparison or explicitly state in the abstract and Section 4.2 that the standard variant achieves nominal coverage empirically rather than by guarantee.
  3. [Section 2.3 and Lemma 2.1] The residual-based aleatoric estimator is one of the paper's stated contributions, but no theoretical result connects the quantiles of Yi − f̂(Xi) to the conditional aleatoric quantiles when f̂ is not exactly the conditional median. Lemma 2.1 assumes consistency of quantile regression estimators for the conditional quantiles of Y, not for residuals of a possibly biased point predictor. If f̂ is biased or the residual distribution differs from the aleatoric noise distribution, the parameters γ1 and λ can rescale the interval but cannot repair its conditional coverage shape in extrapolation regions. Please add explicit conditions under which residual quantiles equal noise quantiles, or empirically validate the decomposition on synthetic data with known model bias and known noise.
minor comments (4)
  1. [Section 4.2] The sentence 'CLEAR (a) was, in fact, the top-performing method on 15 of the 17 datasets' should specify which metric (NCIW, quantile loss, or both) this ranking refers to, since the ranking can differ across metrics.
  2. [Algorithm 1] Step 3 defines γ1 as the smallest value satisfying a coverage constraint but does not specify the behavior when no finite γ1 achieves the constraint; Algorithm 2 in Appendix B.5 provides the necessary detail and should be referenced directly from Step 3.
  3. [Table 1] The notation +∞1/10 v for infinitely wide UACQR-P intervals should be explained in the table caption rather than only in the surrounding text.
  4. [Appendix B.6] Lemma B.9 is essentially a no-worse-than-baselines statement because CLEAR's grid contains the baseline parameter choices; this is fine, but the surrounding text could state more explicitly that the result does not imply a strict improvement over the baselines at the population level.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CLEAR fits its two calibration parameters on validation data and evaluates on held-out test data; the self-cited PCS components are inputs, not conclusions, and the near-tautological Lemma B.9 plus the NCIW metric do not turn the derivation into a re-statement of its inputs.

full rationale

The paper's derivation chain is self-contained at the level where its central claim lives. CLEAR takes pre-trained epistemic and aleatoric estimators as inputs, defines the interval family C(x) with parameters (gamma1, lambda), fixes gamma1 by split-conformal coverage on the calibration set, selects lambda by quantile loss on the validation set, and then measures PICP, NIW, NCIW, and quantile loss on a separate test set. No parameter that appears in the claimed result is fit to the test-set quantity that is being reported, so the headline improvements are not predictions forced by a fitted input. The PCS ensemble is cited from Yu & Kumbier (2020) and Yu & Barter (2024), which overlap with the present authors, but the epistemic estimator is only an input to the calibration layer; the paper also validates CLEAR with Deep Ensembles and Simultaneous Quantile Regression, so the central calibration contribution does not reduce to the self-cited PCS machinery. Lemma B.9 is indeed near-tautological in that CLEAR minimizes quantile loss over a grid that is assumed to contain the baseline ratios, so the inequality CLEAR <= baseline holds by construction; however, this is an optimality statement about the search space, not a derivation that predicts the empirical results, and the paper does not rely on it to establish the 28.3%/17.5% improvements. The NCIW metric applies a test-set calibration constant ctest-cal to enforce coverage before comparing widths; this is a disclosed evaluation choice that could affect the magnitude of the reported gains, and the raw NIW tables in Appendix F.1 independently support the direction of the results. That is a correctness or reporting concern about metric choice, not a circular reduction of the derivation to its own outputs. The residual-based aleatoric estimator is an assumption about how to model noise, and its possible bias under model misspecification is a limitation, not a circular step. Overall, the calibration parameters are genuinely selected on validation data and evaluated out of sample, so no step in the derivation chain reduces to its own input by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on two fitted calibration parameters, a hand-chosen lambda grid, and a test-time constant used only in the NCIW evaluation metric. The method also assumes that PCS residual quantiles represent aleatoric uncertainty and that the additive linear combination captures the total predictive uncertainty. The asymptotic theory assumes consistency of the underlying estimators, which is standard for such guarantees but is not verified empirically.

free parameters (4)
  • gamma1 = per dataset, selected by conformal calibration on Dcal
    Global scaling factor for the combined interval; the central width and coverage claims depend on it.
  • lambda = per dataset, grid search over 0 to 100 with about 4000 values
    Ratio gamma2/gamma1 selected by quantile loss on Dval; determines the epistemic/aleatoric balance.
  • lambda grid = linear 0 to 0.09 plus logarithmic 0.1 to 100
    Hand-chosen discretization that bounds the achievable balance; the result could differ on a coarser grid.
  • ctest-cal = per dataset, chosen on the test set for NCIW
    Test-time calibration constant used in the headline NCIW metric; makes the width comparison partially post-hoc.
assumptions (4)
  • domain assumption Data are i.i.d. from PX times PY|X (Section 2.1)
    Underlies the conformal reasoning and Lemma 2.1; the paper acknowledges this is often unrealistic in real deployments.
  • domain assumption At least k PCS base models are consistent for f, and the quantile regression estimators are consistent for the conditional quantiles of the noise (Lemma 2.1)
    Needed for the asymptotic conditional coverage proof; it is a strong consistency assumption on top of the method rather than a consequence of CLEAR.
  • ad hoc to paper Quantile regression on residuals Yi - fhat(Xi) estimates aleatoric uncertainty (Section 2.3)
    Core modeling choice of the paper; if residual quantiles are distorted by model bias, gamma1 scaling cannot fix the interval shape.
  • ad hoc to paper The additive interval form with gamma1 and lambda in Equation (3) is sufficient for good conditional coverage
    Structural choice; non-additive or input-dependent scaling might be needed for stronger conditional guarantees.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLEAR: Calibrated Learning for Epistemic and Aleatoric Risk." pith.science (2026). https://pith.science/paper/EJNZARR5

@misc{pith2026250708150,
  author       = {Pith},
  title        = {Pith review of: CLEAR: Calibrated Learning for Epistemic and Aleatoric Risk},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EJNZARR5}},
  note         = {Machine review of arXiv:2507.08150}
}
abstract

Accurate uncertainty quantification is critical for reliable predictive modeling. Existing methods typically address either aleatoric uncertainty due to measurement noise or epistemic uncertainty resulting from limited data, but not both in a balanced manner. We propose CLEAR, a calibration method with two distinct parameters, $\gamma_1$ and $\gamma_2$, to combine the two uncertainty components and improve the conditional coverage of predictive intervals for regression tasks. CLEAR is compatible with any pair of aleatoric and epistemic estimators; we show how it can be used with (i) quantile regression for aleatoric uncertainty and (ii) ensembles drawn from the Predictability-Computability-Stability (PCS) framework for epistemic uncertainty. Across 17 diverse real-world datasets, CLEAR achieves an average improvement of 28.3\% and 17.5\% in the interval width compared to the two individually calibrated baselines while maintaining nominal coverage. Similar improvements are observed when applying CLEAR to Deep Ensembles (epistemic) and Simultaneous Quantile Regression (aleatoric). The benefits are especially evident in scenarios dominated by high aleatoric or epistemic uncertainty. Project page: https://unco3892.github.io/clear/

Figures

Figures reproduced from arXiv: 2507.08150 by the authors.

Figure 1
Figure 1. Left: Blue represents aleatoric uncertainty, which reflects randomness inherent in the data such as measurement noise. Red represents epistemic uncertainty, which arises from limited sample size. Right: Estimated prediction sets using the CLEAR method, which combines both sources of uncertainty in a data-driven manner. Lopez-Paz, 2019; Laves et al., 2021). For instance, in active learning, epistemic uncertainty help… view at source ↗
Figure 2
Figure 2. Results for univariate homoskedastic case averaged over 100 simulations: On the left, [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Results for real-world data: Quantile loss and NCIW performance of different methods [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Univariate conditional coverage and average width of the prediction intervals for a het [PITH_FULL_IMAGE:figures/full_fig_p039_4.png]
Figure 5
Figure 5. Figure 5: Univariate conditional coverage and average width of the prediction intervals for a het [PITH_FULL_IMAGE:figures/full_fig_p040_5.png]
Figure 6
Figure 6. Figure 6: Multivariate conditional coverage and average width (distance from the origin is shown) of [PITH_FULL_IMAGE:figures/full_fig_p040_6.png]
Figure 7
Figure 7. Figure 7: Quantile loss and NCIW performance of different methods (CLEAR, PCS, ALEATORIC, [PITH_FULL_IMAGE:figures/full_fig_p052_7.png]
Figure 8
Figure 8. Figure 8: Quantile loss and NCIW performance of different methods (CLEAR, PCS, ALEATORIC, [PITH_FULL_IMAGE:figures/full_fig_p055_8.png]
Figure 9
Figure 9. Figure 9: Quantile loss and NCIW performance of different variants of CLEAR (a, b and c) over 10 [PITH_FULL_IMAGE:figures/full_fig_p058_9.png]
Figure 10
Figure 10. Figure 10: Quantile loss and NCIW performance of UACQR-P and UACQR-S over 10 seeds [PITH_FULL_IMAGE:figures/full_fig_p070_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 29 canonical work pages

  1. [1]

    We reuse the validation dataset Dval as the calibration dataset, i.e., Dcal = Dval

    work page

  2. [2]

    However, our empirical results in Tables 14, 19 and 24 show that for reasonably sized datasets, this theoretical discrepancy does not visibly impact our practical marginal coverage

    We optimize two parameters γ1 and λ on the calibration dataset Dcal. However, our empirical results in Tables 14, 19 and 24 show that for reasonably sized datasets, this theoretical discrepancy does not visibly impact our practical marginal coverage. From a theoretical perspective, CLEAR still achieves asymptotic marginal coverage (as a consequence of asy...

    work page

  3. [3]

    If ⌈(1 − α)(|Dcal| + 1)⌉ > |Dcal|, set γ∗ 1 = ∞

    Calibrate the Prediction Interval: Set the calibration parameter γ∗ 1 to be the ⌈(1 − α)(|Dcal| + 1)⌉-th smallest value among the conformity scores Sλ∗ i from Dcal. If ⌈(1 − α)(|Dcal| + 1)⌉ > |Dcal|, set γ∗ 1 = ∞

    work page

  4. [4]

    Lemma B.2

    Form the Final Prediction Interval: For a new test point xnew, the final (1 − α)-prediction interval is given by: C(xnew) = h ˆf (xnew) − γ∗ 1 ˆqale α/2 + λ∗ ˆqepi α/2 , ˆf (xnew) + γ∗ 1 ˆqale 1−α/2 + λ∗ ˆqepi 1−α/2 i This modified version of CLEAR, which we call Conformalized CLEAR, satisfies the standard finite-sample marginal coverage guarantee of conf...

    work page 2024

  5. [6]

    URL https://openreview.net/forum?id=MhLnSoWp3p. N. Hollmann, S. M¨uller, L. Purucker, A. Krishnakumar, M. K¨orfer, S. B. Hoo, R. T. Schirrmeister, and F. Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 01

    work page

  6. [7]

    URL https://www.nature.com/articles/ s41586-024-08328-6

    doi: 10.1038/s41586-024-08328-6. URL https://www.nature.com/articles/ s41586-024-08328-6 . S. B. Hoo, S. M¨uller, D. Salinas, and F. Hutter. The tabular foundation model tabpfn outperforms specialized time series forecasting models based on simple features, 2025. URL https:// arxiv.org/abs/2501.02945. E. H ¨ullermeier and W. Waegeman. Aleatoric and episte...

    arXiv

  7. [9]

    https://iclr-blogposts.github.io/2025/blog/reexamining-the-aleatoric-and-epistemic-uncertainty- dichotomy/

    URL https://iclr-blogposts.github.io/2025/blog/ reexamining-the-aleatoric-and-epistemic-uncertainty-dichotomy/ . https://iclr-blogposts.github.io/2025/blog/reexamining-the-aleatoric-and-epistemic-uncertainty- dichotomy/. D. Kivaranovic, R. Ristl, M. Posch, and H. Leeb. Conformal prediction intervals for the individual treatment effect. arXiv preprint, arX...

    arXiv 2025

  8. [10]

    doi: 10.59275/j.melba.2021-a6fd

    ISSN 2766-905X. doi: 10.59275/j.melba.2021-a6fd. J. Lei and L. Wasserman. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 76, 2014. J. Lei, M. G’Sell, A. Rinaldo, R. J Tibshirani, and L. Wasserman. Distribution-free predictive inference for regression. Journal ...

    arXiv 2021

Show all 38 references
  1. [12]

    doi: 10.1093/jrsssb/qkae108

    ISSN 1369-7412. doi: 10.1093/jrsssb/qkae108. URL https://doi.org/10.1093/ jrsssb/qkae108. Y . Shen, N. Daheim, B. Cong, P. Nickl, G. Marconi, B. Raoul, R. Yokota, I. Gurevych, D. Cremers, M. Khan, and T. M ¨ollenhoff. Variational learning is effective for large deep networks. ...

    arXiv

  2. [13]

    doi: 10.1109/CVPR52733.2024.01051. Z. Wang, T. Ren, J. Zhu, and B. Zhang. Function space particle optimization for bayesian neural networks. In International Conference on Learning Representations , 2019. URL https:// openreview.net/forum?id=BkgtDsCcKQ. J. Weissteiner, J. Heis...

    2024

  3. [15]

    B EYOND PINBALL LOSS : QUANTILE METHODS FOR CALIBRATED UNCERTAINTY QUANTIFICATION

    and empirical (Wenzel et al., 2020a) studies suggest that some of these approximations can actually provide superior estimates compared to their exact counterparts, due to poor choices of priors, such as i.i.d. Gaussian priors, in certain settings. A.3.3 EPICSCORE The recent w...

    2025

  4. [18]

    The optimal value λ∗ is found by optimizing the QuantileLoss on Dval, without using any data from Dcal (Step 4 and 5 of Algorithm 1)

    Train and Optimize λ by running Algorithm 1 on Dtrain and Dval: The model ˆf and the uncertainty estimators are trained on Dtrain (Step 2 and 3 of Algorithm 1). The optimal value λ∗ is found by optimizing the QuantileLoss on Dval, without using any data from Dcal (Step 4 and 5...

  5. [19]

    Compute Conformity Scores: For each data point (Xi, Yi) ∈ Dcal, compute the conformity score Sλ∗ i : Sλ∗ i = max ( ˆf (Xi) − Yi ˆf (Xi) − ˜lλ∗ (Xi) , Yi − ˆf (Xi) ˜uλ∗ (Xi) − ˆf (Xi) ) where ˜lλ∗ := ˆf − ˆqale α/2 − λ∗ ˆqepi α/2 and ˜uλ∗ := ˆf + ˆqale 1−α/2 + λ∗ ˆqepi 1−α/2

  6. [22]

    If the calibration residuals are small by chance, conformal intervals may be too narrow, especially with tiny calibration datasets

    The conformal marginal coverage guarantee P[Ynew ∈ C(Xnew)] = EDtrain,Dcal[P[Ynew ∈ C(Xnew)|Dtrain, Dcal]] ≥ 1 − α does not imply that P[Ynew ∈ C(Xnew)|Dtrain, Dcal] ≥ 1 − α for a fixed realization of the calibration set Dcal, as already discussed before. If the calibration re...

    2020

  7. [23]

    This is crucial in human-in-the-loop settings, where interventions are prioritized based on an accurate ranking of predictive uncertainty across data points (see Appendix I)

    Beyond marginal coverage, CLEAR is designed to improve conditional calibration : P[Ynew ∈ C(Xnew)|Xnew] ≈ 1 − α. This is crucial in human-in-the-loop settings, where interventions are prioritized based on an accurate ranking of predictive uncertainty across data points (see Ap...

    2026

  8. [24]

    Similarly, classical Random Forests are known to be consistent under standard assumptions (Scornet et al., 2015)

    Quantile Random Forests have been shown to consistently estimate quantiles under rea- sonable conditions, such as by regularizing the minimum number of samples per leaf (Meinshausen, 2006). Similarly, classical Random Forests are known to be consistent under standard assumptio...

    2006

  9. [25]

    This suggests that XGBoost and its quantile version, QXGB, can be consistent for suitable choices of hyperparameters

    Boosting methods trained on general loss functions can be made consistent under certain conditions, especially when regularized through early stopping (Zhang & Yu, 2005). This suggests that XGBoost and its quantile version, QXGB, can be consistent for suitable choices of hyper...

    2005

  10. [26]

    pinball loss

    More generally, regularized minimization of QuantileLoss over a sufficiently expressive function class on Dtrain yields consistent quantile estimators under broad assumptions (Steinwart & Christmann, 2011). In our experiments, we set k = 1, so Lemma 2.1 requires only one of th...

    2011

  11. [27]

    Some QR-methods in Step 2 of Algorithm 1 use the QuantileLoss to train their models

  12. [28]

    In other words, Step 3 and 4 of Algorithm 1 together (approximately) solve (γ⋆ 1 , λ⋆) = arg min (γ1,λ)∈(0,∞)×Λ s.t

    In Step 4 of Algorithm 1 we minimize the QuantileLoss to determine λ∗. In other words, Step 3 and 4 of Algorithm 1 together (approximately) solve (γ⋆ 1 , λ⋆) = arg min (γ1,λ)∈(0,∞)×Λ s.t. |{(x,y)∈Dcal:y∈Cγ1 ,λ(x)}|≥⌈(1−α)(|Dcal|+1)⌉ QuantileLoss(Dval, Cγ1,λ), (4) where Cγ1,λ =...

  13. [29]

    P[Y | X]

    We use the QuantileLoss as an evaluation metric on the test dataset Dtest. In the following, we will discuss multiple favorable properties of the QuantileLoss (which equivalently also hold for the Interval Score Loss, see Appendix B.3.4). B.3.1 I NTUITION BEHIND THE QUANTILE L...

    2025

  14. [30]

    CLEAR (CCLEAR): Selects λ∗ = arg minλ∈Λ L(λ) and outputs Cγ1(λ∗),λ∗γ1(λ∗)

  15. [31]

    E.g., λ0 = 0 for methods that do not explicitly model epistemic uncertainty (see Ap- pendix A.6), or λ0 = 1 for methods that model both but do not rebalance them (see Appendix A.2)

    Fixed-λ Baseline (Cbase-λ): Fixes a hyperparameter λ0 ∈ Λ and outputs Cγ1(λ0),λ0γ1(λ0). E.g., λ0 = 0 for methods that do not explicitly model epistemic uncertainty (see Ap- pendix A.6), or λ0 = 1 for methods that model both but do not rebalance them (see Appendix A.2)

  16. [32]

    It outputs Cγ1,0,γ2,base

    Fixed-γ1 Baseline ( Cbase-γ1): Fixes a hyperparameter γ1,0 > 0 (e.g., γ1,0 = 1 as in Appendix A.1) and calibrates γ2 to achieve exact coverage, finding γ2,base = inf {γ >0 : Cov(γ1,0, γ) ≥ 1 − α}. It outputs Cγ1,0,γ2,base. We assume its implicit ratio λ′ = γ2,base γ1,0 is cont...

    2025

  17. [33]

    On the calibration set, absolute residuals ai = |yi − ˆf (xi)| are computed

    Conformalized PCS median (Naive): For the PCS ensemble, we compute the median prediction ˆf (x) = ˆfPCS(x) for each x. On the calibration set, absolute residuals ai = |yi − ˆf (xi)| are computed. The (1 − α)-th quantile γnaive of these absolute residuals is then used to define...

  18. [34]

    This implies γ1 = γ2

    CLEAR with fixed λ = 1: This is a variant of the main CLEAR methodology where the ratio λ = γ2/γ1 is fixed to 1. This implies γ1 = γ2. The prediction interval, based on Equation (3), becomes: Cλ=1(x) = h ˆf (x) − γ1 ˆqale α/2(x) + ˆqepi α/2(x) , ˆf (x) + γ1 ˆqale 1−α/2(x) + ˆq...

  19. [35]

    With γ1 = 1, then γ2 = λ and the prediction interval from Equation (3) reads as: Cγ1=1(x) = h ˆf (x) − ˆqale α/2(x) − λˆqepi α/2(x), ˆf (x) + ˆqale 1−α/2(x) + λˆqepi 1−α/2(x) i

    CLEAR with fixed γ1 = 1: Another variant of CLEAR where γ1 is fixed to 1. With γ1 = 1, then γ2 = λ and the prediction interval from Equation (3) reads as: Cγ1=1(x) = h ˆf (x) − ˆqale α/2(x) − λˆqepi α/2(x), ˆf (x) + ˆqale 1−α/2(x) + λˆqepi 1−α/2(x) i . Here, λ (or equivalently...

  20. [36]

    relative vs. absolute uncertainty

    UACQR-S and UACQR-PFor variant (c), instead of computing Cγ1=1(x) and Cλ=1(x) with CLEAR, we directly use the implementation from Rossellini et al. (2024) to assess our performance against this alternative. Since this is only relevant for variant (c), we use the exact same con...

    2024

  21. [37]

    Estimating relative uncertainty accurately—capturing how uncertainty varies across inputs

  22. [38]

    Even when conditional coverage is not required, improving relative uncertainty tends to reduce average interval width and improve marginal calibration under distribution shifts

    Calibrating the absolute scale to ensure the desired coverage level holds at each input. Even when conditional coverage is not required, improving relative uncertainty tends to reduce average interval width and improve marginal calibration under distribution shifts. Thus, adap...

  23. [1968]

    ISSN 1369-7412. R. Dwivedi, Y . Tan, B. Park, M. Wei, K. Horgan, D. Madigan, and B. Yu. Stable discovery of interpretable subgroups via calibration in causal studies (stadisc). International Statistical Review, 2020. R. C. Edward. Gaussian processes in machine learning. In Sum...

    2020

  24. [2006]

    URL http://jmlr.org/papers/v7/meinshausen06a.html. V . Moroshan, J. Siems, A. Zela, T. Carstensen, and F. Hutter. Tempopfn: Synthetic pre-training of linear rnns for zero-shot time series forecasting, 2026. URL https://arxiv.org/abs/ 2510.25502. S. M¨uller, N. Hollmann, S. P. ...

    2026

  25. [2017]

    kin8nm dataset

    URL https://arxiv.org/abs/1703.04977. kin8nm dataset. kin8nm dataset. https://www.openml.org/d/189, 2014. Accessed: 2025- 04-15. M. Kirchhof, G. Kasneci, and E. Kasneci. Reexamining the aleatoric and epistemic uncertainty dichotomy. In ICLR Blogposts 2025 ,

    2014 arXiv

  26. [2019]

    ambiguity in targets y for a given x

    ALEATORIC Bootstrapped CQR CQR variant, where we compute CQR on boot- strapped data and take the median as final estimate. Follows from PCS framework (Yu & Barter, 2024) ALEATORIC-R Residual-based Bootstrapped CQR ALEATORIC applied on residuals Yi − ˆf (Xi), where ˆf is obtain...

    2021

  27. [2020]

    Accessed: 2025-04-15. A. Altieri, M. Romanelli, G. Pichler, F. Alberge, and P. Piantanida. Beyond the norms: detecting prediction errors in regression models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024. 11 Published as a co...

    2025

  28. [2021]

    URL https://proceedings.neurips.cc/paper_files/paper/2021/ file/a7c9585703d275249f30a088cebba0ad-Paper.pdf. G. De Ath, R. M. Everson, A. A. M. Rahat, and J. E. Fieldsend. Greed is good: Exploration and exploitation trade-offs in bayesian optimisation. ACM Trans. Evol. Learn. O...

    2021 doi

  29. [2024]

    URL https://arxiv.org/abs/2411.11824. I. Azizi, M. Boldi, and V . Chavez-Demoulin. Semf: Supervised expectation-maximization framework for predicting intervals. In Proceedings of the Fourteenth Symposium on Conformal and Proba- bilistic Prediction with Applications, volume 266...

    2025 arXiv

  30. [2025]

    URL https://arxiv.org/abs/2510.00087. A. Agarwal, M. Xiao, R. Barter, O. Ronen, B. Fan, and B. Yu. Pcs-uq: Uncertainty quantification via the predictability-computability-stability framework. arXiv preprint arXiv:2505.08784, 2025. S. Allen, G. Gavrilopoulos, A. Henzi, G. Klege...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.