Pith. sign in

REVIEW 3 major objections 5 minor 100 references

A Planning Framework for Adaptive Labeling

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Even one-step planning sharply beats adaptive-labeling heuristics.

desk verdict A useful MDP framing and a clever smoothed gradient trick, but the weighted GP posterior update in Section 5.3 is wrong as written and the Autodiff pipeline currently rests on it. read the letter →

arxiv 2502.06076 v1 pith:57GFM3YT submitted 2025-02-10 cs.LG

Daksh Mittal , Yuanzhe Ma , Shalmali Joshi , Hongseok Namkoong This is my paper · ORCID
classification cs.LG MSC 68T0562L0590C4062F15
keywords AdaptivelabelingMarkovdecisionprocessPolicygradientSmoothed-AutodiffSoftK-subsetsamplingUncertaintyquantificationOne-steplookaheadSelectionbias
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that adaptive labeling, the sequential choice of which expensive data points to label in batches, is best formulated as a planning problem rather than a greedy heuristic. By modeling posterior beliefs as the state of a Markov decision process and optimizing a one-step lookahead policy with policy gradients, the authors show substantially lower final uncertainty about the estimand (model performance or average treatment effect) than uncertainty sampling or random sampling on synthetic and real medical data. To make planning tractable, the paper constructs a fully differentiable surrogate of the non-differentiable labeling pipeline and introduces Smoothed-Autodiff, a biased but low-variance pathwise gradient estimator; they prove and empirically demonstrate that it dominates the unbiased REINFORCE estimator in sample efficiency and downstream task performance. The paper also documents that current deep uncertainty quantification methods are often unreliable under distribution shift and in sequential updating, which limits but does not invalidate the planning framework.

What carries the argument

The differentiable planning pipeline is the central mechanism: a soft K-subset sampling procedure, defined as a continuous relaxation of weighted reservoir sampling, produces a vector $a(\theta) \in [0,1]^n$ that approximates a discrete batch of $K$ labels; the paper reinterprets $a(\theta)$ as fractional sample weights in the posterior update, yielding a weighted Gaussian process update or weighted gradient steps for deep ensembles. This makes the full roll-out, consisting of pseudo-label generation, soft subset sampling, posterior update, and objective evaluation, end-to-end differentiable, allowing direct backpropagation. The Smoothed-Autodiff estimator then computes pathwise gradients through this smoothed simulator, trading unbiasedness for drastically lower variance, with the temperature $\tau$ controlling the bias-variance trade-off.

What would settle it

On a small discrete-action problem, compute the exact objective for every possible batch, evaluate the true policy gradient by finite differences, and compare it to the Smoothed-Autodiff gradient at the operating temperature $\tau=0.1$; if the cosine similarity is low or if a policy optimized by Smoothed-Autodiff underperforms simple uncertainty sampling on held-out data, the soft-update surrogate fails.

Watch

Extended reading notes

Core claim

The central claim is that even a single lookahead step, optimized through a smooth differentiable surrogate of the non-differentiable MDP, yields near-optimal adaptive labeling policies, and that the Smoothed-Autodiff gradient estimator, biased but low-variance, is statistically more efficient than the unbiased REINFORCE estimator in exactly the small-sample regime where planning is practical. The paper establishes this with theorems for binary and multi-action settings where REINFORCE's mean squared error grows unboundedly as the policy becomes extreme while the smoothed estimator's MSE stays bounded, and with experiments on synthetic Gaussian process data and real eICU patient data. The key is the bias-variance trade-off: adding a carefully controlled smoothing temperature produces gradients whose direction is more reliable on a single trajectory than score-function estimates. The planning benefits appear even when the optimized policy is evaluated by its effect on the variance of the posterior over the estimand, confirming that one-step lookahead captures the structure of batched information gain.

Load-bearing premise

The weighted soft posterior update, which reinterprets fractional sample weights as if they were actual labeled data, faithfully approximates the true discrete posterior update at the operating temperature, so that the smoothed gradients point in a direction that actually reduces uncertainty.

Editorial extensions

If this is right

  • Practitioners can use one-step lookahead planning with standard uncertainty quantification tools to allocate labels efficiently when datasets have severe selection bias, achieving better model evaluation and causal effect estimation with fewer labels.
  • Smoothed-Autodiff makes planning feasible with a single simulated trajectory, cutting gradient estimation cost by orders of magnitude relative to REINFORCE and enabling policy optimization in compute-constrained settings.
  • The differentiable-simulator recipe extends to other combinatorial sequential decision problems with known dynamics, including adaptive average treatment effect estimation and model-based policy gradients.
  • The framework is agnostic to the uncertainty quantification module, working with Gaussian processes, deep ensembles, Ensemble+, and newer Bayesian models, so improvements in UQ directly translate into better planning.
  • The bias-variance analysis implies that in few-sample regimes, biased pathwise gradients can dominate unbiased score-function gradients, informing future gradient estimation for non-differentiable simulation objectives.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to calibrate the temperature $\tau$ against the true posterior update on a small validation set before deployment; if the soft weighted update is inaccurate at that temperature, the smoothed gradients may mislead the policy, and adaptive adjustment of $\tau$ could become a robustness knob.
  • The observed success of one-step lookaheads suggests that multi-step lookaheads, currently bottlenecked by second-order automatic differentiation, may yield further sampling gains once efficient Hessian estimation matures.
  • The paper's UQ consistency study indicates that planning guarantees only hold when the posterior sharpens reliably on unobserved regions; this motivates a new benchmark for evaluating uncertainty quantification in dynamic settings, not just static one-shot calibration.
  • The differentiable-simulation approach could be transferred to experimental design, pure-exploration bandits, and black-box optimization where the dynamics are known and differentiable, going beyond labeling to any costly sequential measurement problem.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a planning framework for adaptive labeling, formulated as a Markov decision process over posterior beliefs, with continuous policy parameterization via weighted reservoir sampling. The main methodological contribution is Smoothed-Autodiff, a pathwise policy-gradient estimator that backpropagates through a smoothed version of the MDP: soft K-subset sampling replaces discrete batch selection, and a weighted (soft) posterior update is used for Gaussian processes and deep ensembles. The authors report that a one-step lookahead policy optimized with this estimator outperforms uncertainty-sampling and random-sampling heuristics on a synthetic GP experiment and on eICU data, and they provide theoretical and empirical evidence that the biased Smoothed-Autodiff gradient has lower MSE than REINFORCE when the number of Monte Carlo samples is small. Section 8 documents practical limitations of current UQ methods in dynamic settings.

Significance. If the differentiable surrogate were a faithful relaxation of the original discrete lookahead objective, the framework would be a useful contribution: it gives a principled way to apply planning to adaptive labeling with combinatorial action spaces, and the demonstration that a biased low-variance gradient estimator can beat REINFORCE in this setting is valuable. The paper also ships a code repository and provides a reproducible experimental setup, which is a strength. However, the load-bearing component is the soft posterior update of Section 5.3, which is introduced as a heuristic, is not derived from a probabilistic model, and is not shown to approximate the exact discrete posterior update with any quantified bias. The manuscript's own Section 8.1 independently shows that posterior updates of current UQ methods are unreliable under distribution shift, which makes the unvalidated surrogate a serious correctness risk for the central claim that the planning gains reflect genuine lookahead value rather than the artifacts of the relaxation.

major comments (3)
  1. [Section 5.3] The claim that the weighted GP posterior update "can be easily verified" to recover the standard GP update when a is a K-hot vector is incorrect as written. With the standard meanings of 1 (all-ones matrix) and I (identity), the matrix Ka = K((1-I)aa^T + I) is not symmetric in general, and for a K-hot vector with |S| > 1 it does not reduce to K_{S,S} + sigma^2 I. The expression involves a matrix product rather than an elementwise (Hadamard) weighting, so the pseudo-covariance inserted into the Cholesky factorization in Algorithm 3 does not correspond to conditioning the GP on the subset S. Please provide a correct derivation of the soft weighted GP update, or at minimum an explicit elementwise construction with a proof that it recovers exact conditioning, and add an experiment measuring the bias of the soft update against the exact discrete posterior update for K-hot vectors.
  2. [Section 5.3 / Eq. (10)] The weighted Ensemble+ update is introduced as a heuristic: the one-gradient-step rule eta1(a(θ)) = eta0 - sum_i a_i ∇ℓ(X_i, Y_i, η)|_{η=η0} is not derived as an approximate posterior update for the selected batch, and no bound is given on the difference between optimizing the smoothed objective and optimizing the original discrete objective in Eq. (9). Because the algorithm differentiates through a finite number of SGD steps to approximate the argmin, the gradient ∇θ G(η+(a(θ))) may not approximate the gradient of the original expectation over discrete subsets. The paper should either provide a formal bias analysis for this surrogate or include an ablation that compares the smoothed pipeline's decisions with the exact (unsmoothed) posterior update on small problems where the latter is tractable.
  3. [Section 7.2] The cosine-similarity comparison in Figure 11 uses a 10^6-sample REINFORCE estimate as the "true gradient". This estimate is for the discrete objective with S ~ πθ, while Smoothed-Autodiff computes the gradient of the smoothed surrogate objective G(μ_{a(θ)}). The comparison therefore measures alignment with the discrete-objective gradient only if the surrogate is accurate, which is exactly the unvalidated premise of Section 5.3. Moreover, Theorems 1 and 2 are proved for generic scalar actions with a smooth G, not for the GP/Ensemble+ posterior-update pipeline; they establish that a biased estimator can have lower MSE than REINFORCE in those simplified settings, but they do not show that the specific soft posterior update used in Section 6 has favorable bias relative to the true discrete lookahead. Please clarify this gap and, if possible, compute the true gradient of the one-step lookahead in a small synthetic setting to validate the cosine-similarity results.
minor comments (5)
  1. [Section 5.3] The symbols "1" and "I" in the GP update formula are not defined; if "1" denotes the all-ones matrix and I the identity, the expression is ambiguous and should be rewritten with explicit notation (e.g., \mathbb{1}_n and I_n).
  2. [Figures 7-10] The text states that results are averaged over 10 seeds, but the figures do not show error bars or confidence bands; please add them or report the per-seed variance, particularly for the eICU results where the gap between methods is smaller.
  3. [Table 1] REINFORCE and Autodiff 1-lookahead report identical values (1697.1 and 45.27) with zero variance; please state whether both methods converged to the same deterministic batch and clarify whether the displayed variance is across seeds or across optimization restarts.
  4. [Section 7.2] The ground-truth gradient is itself a stochastic estimate from 10^6 REINFORCE samples; please report the standard error of this estimate or use a setting where the gradient can be computed analytically, so the cosine similarities are not measured against a noisy baseline.
  5. [Algorithm 3 / step 2] Even if the intended operation were elementwise, the comment "Ka = K((1-I)aa^T + I)" does not make this explicit; please clarify whether the multiplication is matrix multiplication or Hadamard product, and adjust the equation in the main text accordingly.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the smoothed pipeline is an optimization heuristic, while the reported planning gains are evaluated on exact posterior updates after hard top-K selection; the only same-author citation is related work, not load-bearing.

full rationale

The paper's central claim (one-step lookahead beats active-learning heuristics) is not forced by construction. In Algorithm 1, the deployed action is the hard top-K batch (line 5), and the state transition after real labels is the exact posterior update (line 7), so the evaluation objective is the original discrete MDP objective, not the relaxed soft objective used only to compute gradients. The soft weighted posterior update in Section 5.3 is explicitly presented as a reinterpretation ('we reinterpret a(theta) as weights ... and adapt the posterior updates ... accordingly'), i.e., an ansatz for differentiability. The statement that a K-hot vector recovers the standard GP update is a consistency check, not a derivation of planning performance; even if the printed matrix formula fails to reduce to K_{S,S}+sigma^2 I as written, that is a faithfulness or correctness risk, not a circular reduction. The bias-variance theorems in Section 7 are standalone analyses of simplified estimators, and the empirical gradient comparison uses a high-sample REINFORCE estimate as ground truth; no parameter is fitted to the reported outcome. The only same-author citation is [16] (Che, Dong, Namkoong), used in related work to distinguish the smoothing approach, and [69] (Namkoong et al.) is a practical feasibility aside; neither is load-bearing. Section 8.1 candidly reports that current UQ posterior updates are unreliable under shift, which is a limitation of the UQ module rather than evidence that the planning derivation reduces to its inputs. Overall, the derivation is self-contained with respect to the evaluation protocol, and the soft-proxy concern is a validity issue outside the circularity definition.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework's claims rest on standard relaxation results (soft K-subset), on the availability of a reliable UQ module, and, critically, on an ad hoc weighted posterior update that makes the pipeline differentiable. The latter is not justified theoretically and is the main source of bias in Smoothed-Autodiff.

free parameters (3)
  • softmax temperature tau = 0.1
    Controls the degree of smoothing in soft K-subset sampling (Algorithm 2) and the recall surrogate; the theory only shows existence of some tau for favorable bias-variance, but the experiments fix tau=0.1 across all runs without sensitivity analysis.
  • Monte Carlo samples for G = 100
    Number of posterior function samples used to estimate Var_{f~mu}(g(f)) in the roll-out objective (Section B.1); larger values reduce noise at computational cost.
  • Ensemble+ prior scale alpha = 100
    Set to reflect high prior uncertainty in the Ensemble+ toy experiment (Section B.3); it is part of the UQ module's prior, not tuned by the planning algorithm, but it strongly influences the scale of the objective.
assumptions (4)
  • standard math Weighted reservoir sampling distribution p_wrs and soft K-subset sampling from Xie and Ermon (2019) are valid continuous relaxations of discrete batch selection.
    Used in Section 4.2 and Algorithm 2. This is cited prior work and is standard.
  • domain assumption The data generating process is Y = f*(X) + epsilon with known likelihood p_eps(y-f(x)), and the UQ module provides a posterior over f that is approximately correct.
    Stated at the start of Section 3. If the posterior is misspecified (as Section 8 shows for many deep UQ methods), the pseudo-label roll-outs and the variance objective are no longer reliable.
  • ad hoc to paper The weighted posterior update for GPs and Ensemble+ (Section 5.3) is a differentiable surrogate whose gradient points in a useful direction for the original discrete objective.
    The weighted GP formulas and the eta_1(a) update are introduced to enable backpropagation; they are not derived from a Bayesian likelihood and their bias is not quantified. This is the central methodological assumption behind Smoothed-Autodiff.
  • domain assumption The one-step lookahead objective using pseudo-labels drawn from the current posterior, with the posterior updated accordingly, approximates the true expected posterior variance.
    Used in the roll-out in Section 4.3. This is a standard simulation-based approximation, but its accuracy depends on the quality of the posterior.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Planning Framework for Adaptive Labeling." pith.science (2026). https://pith.science/paper/57GFM3YT

@misc{pith2026250206076,
  author       = {Pith},
  title        = {Pith review of: A Planning Framework for Adaptive Labeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/57GFM3YT}},
  note         = {Machine review of arXiv:2502.06076}
}
read the original abstract

Ground truth labels/outcomes are critical for advancing scientific and engineering applications, e.g., evaluating the treatment effect of an intervention or performance of a predictive model. Since randomly sampling inputs for labeling can be prohibitively expensive, we introduce an adaptive labeling framework where measurement effort can be reallocated in batches. We formulate this problem as a Markov decision process where posterior beliefs evolve over time as batches of labels are collected (state transition), and batches (actions) are chosen to minimize uncertainty at the end of data collection. We design a computational framework that is agnostic to different uncertainty quantification approaches including those based on deep learning, and allows a diverse array of policy gradient approaches by relying on continuous policy parameterizations. On real and synthetic datasets, we demonstrate even a one-step lookahead policy can substantially outperform common adaptive labeling heuristics, highlighting the virtue of planning. On the methodological side, we note that standard REINFORCE-style policy gradient estimators can suffer high variance since they rely only on zeroth order information. We propose a direct backpropagation-based approach, Smoothed-Autodiff, based on a carefully smoothed version of the original non-differentiable MDP. Our method enjoys low variance at the price of introducing bias, and we theoretically and empirically show that this trade-off can be favorable.

Figures

Figures reproduced from arXiv: 2502.06076 by the authors.

Figure 1
Figure 1. Adaptive labeling to reduce epistemic uncertainty over model performance. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our adaptive sampling framework. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. MDP framework for adaptive labeling to efficiently estimate the average treatment effect (ATE). 4 Continuous Reformulation The dynamic programs (2) and (4) involve continuous states and action spaces that are combina￾torially large in the number of data points in Xpool. In order to bring to bear the power of planning through policy gradients, we introduce continuous reformulations of the original planning problems. … view at source ↗
Figures from the paper (27 more)
Figure 4
Figure 4. Figure 4: One-step lookahead roll-out for policy gradient estimation the batch X t+1, we observe the corresponding labels Y t+1 and update posterior belief over the data-generating function f to µt+1. Algorithm 1 summarizes each of these steps of the overall procedure. The polic…
Figure 5
Figure 5. Figure 5: Differentiable one-step lookahead pipeline for efficient adaptive sampling 5.4 Putting everything together Note that when using GPs as our UQ methodology, the gradient takes the form ∇θG (µ+ (a(θ))). Assuming G(·) is differentiable, pathwise gradients can be computed s…
Figure 6
Figure 6. Figure 6: Description of Autodiff 1-lookahead algorithm (Algorithm 1), and show that even basic planning can hold great promise. We illustrate our framework using two uncertainty quantification modules—GPs and Ensembles/ Ensemble+. Throughout this section, we focus on evaluating…
Figure 7
Figure 7. Figure 7: (Synthetic data) Variance of mean squared loss evaluated through the posterior belief µt at each horizon t. This is the ob￾jective that policy gradient methods like RE￾INFORCE and Autodiff 1-lookahead opti￾mizes. 1-step lookaheads are surprisingly ef￾fective even in lo…
Figure 9
Figure 9. Figure 9: (Real-world eICU data) Variance of mean squared loss evaluated through the posterior belief µt at each horizon t. Even 1- step lookaheads are extremely effective plan￾ners, and auto-differentiation-based path￾wise policy gradients provide a reliable op￾timization algor…
Figure 11
Figure 11. Figure 11: Comparison of Smoothed-Autodiff and REINFORCE estimators under different tempera￾ture (τ ) hyperparameter for the Smoothed-Autodiff gradient - metric used is cosine similarity As discussed, we denote the true gradient by ∇θH(θ). For any gradient estimator ∇ˆ N (either…
Figure 13
Figure 13. Figure 13: Comparing the impact of Smoothed-Autodiff and REINFORCE gradient estimators (with different values of N) on the performance of Autodiff 1-lookahead. (Synthetic data) Error between MSE calcu￾lated based on collected data D0:T vs. popu￾lation oracle MSE over Deval ∼ PX.…
Figure 14
Figure 14. Figure 14: Joint log-loss on OOD data with increasing selection bias. Though some agents such as ensemble and dropout perform well under no selection bias, their performance deteriorates signif￾icantly with a large selection bias (larger k). the posteriors are sharp on observed …
Figure 15
Figure 15. Figure 15: Trade-off between in-distribution and out-of-distribution joint log-loss with prior scale as the hyperparameter (eICU, Clustering bias). Performance under dynamic settings We observe that even static OOD performance does not comprehensively capture the reliability of …
Figure 16
Figure 16. Figure 16: Performance of different UQ modules in a dynamic setting. Experimental Setup: Population is distributed across 2 clusters. At T = 0, agent observes data D0 sampled from Cluster 1. Figure (a) and Figure (b) show how different UQ models perform on test data from Cluster…
Figure 17
Figure 17. Figure 17: Joint log-loss on OOD data with increasing selection bias with a GP oracle. 1e-4 1e-3 1e-2 0.1 1 10 100 L2 weight decay 10 1 10 0 10 1 joint log-loss in-distribution out-of-distribution Ensemble+ 1e-4 1e-3 0.01 0.1 1 10 L2 weight decay 10 2 10 1 10 0 10 1 10 2 in-dist…
Figure 18
Figure 18. Figure 18: Trade-off between in-distribution (ID) and out-of-distribution (OOD) performance with weight decay as the hyperparameter (eICU, Clustering bias). D.7.3 Dynamic settings for other datasets To further examine our findings in [PITH_FULL_IMAGE:figures/full_fig_p044_18.png]
Figure 19
Figure 19. Figure 19: Trade-off between in-distribution (ID) performance and out-of-distribution (OOD) per￾formance with stopping time as the hyperparameter (eICU, clustering). ensemble+ epinet hypermodel 10 3 10 2 10 1 10 0 Joint log-loss (a) ID performance (T = 0) ensemble+ epinet hyperm…
Figure 20
Figure 20. Figure 20: Performance of different UQ modules in a dynamic setting for IEEE-CIS Fraud Detection. Although Hypermodels perform the best in the ID setting (a), they perform the worst in the OOD setting at T = 0. However, we see that Hypermodels are good at adapting to new data fo…
Figure 21
Figure 21. Figure 21: Performance of different UQ modules in a dynamic setting for Credit Card fraud. In this case hypermodels perform the best for both the ID setting (a) and the OOD setting (b) at T = 0. However, we can see that the OOD performance improvement (d) is the best for the epi…
Figure 22
Figure 22. Figure 22: Performance of different agents in a dynamic setting for Fraud Ecommerce. Surprisingly, we observe that all three models have a larger OOD joint log-loss after seeing new data. The reason for such a behaviour might be that the models start from a prior belief with low…
Figure 23
Figure 23. Figure 23: Performance of different agents in a dynamic setting for Vehicle Loan Default Prediction. Although hypermodels perform the best in the ID setting (a), they perform the worst in the OOD setting at T = 0. However, we see that hypermodels are good at adapting to new data…
Figure 24
Figure 24. Figure 24: Performance of different agents in a dynamic setting for ACS Employment. In this case hypermodels perform the best in both the ID setting (a) but in the OOD setting [(b),(c), and (d)], all the models perform similarly at T = 0 and T = 1. ensemble+ epinet hypermodel 10…
Figure 25
Figure 25. Figure 25: Performance of different agents in a dynamic setting for ACS Income. Although hyper￾models perform the best in the ID setting (a), they perform the worst in the OOD setting at T = 0. However, we see that hypermodels are good at adapting to new data for this dataset an…
Figure 26
Figure 26. Figure 26: Performance of ensemble + in a dynamic setting with varying weight decays [eICU data with clustering bias]. We can see the weight decay with values from 1e − 4 to 1e − 2 is the best for the in-distribution data performance [plot (a)], while weight decay of 10 is the b…
Figure 27
Figure 27. Figure 27: Performance of epinets in a dynamic setting with varying weight decays [eICU data with clustering bias]. We can see the weight decay of 1e − 4 is the best for the in-distribution data performance [plot (a)], while weight decay of 0.1 is the best for the OOD data perfo…
Figure 28
Figure 28. Figure 28: Performance of hypermodels in a dynamic setting with varying weight decay [eICU data with clustering bias]. We can see the weight decay with values from 1e−4 to 1e−2 is the best for the in-distribution data performance [plot (a)], while weight decay of 10 to 100 is be…
Figure 29
Figure 29. Figure 29: Performance of ensemble + in a dynamic setting with varying prior scale [eICU data with clustering bias]. We can see that the prior scale 0.1 to 1 is the best for the in-distribution data performance [plot (a)]. However, prior scale of 10 is the best for the OOD data …
Figure 30
Figure 30. Figure 30: Performance of epinets in a dynamic setting with varying prior scale [eICU data with clustering bias]. We can see the prior scale of 10 to 30 works best for both the in-distribution data performance [plot (a)] the OOD data performance [plot (b)] at T = 0. However, in …
Figure 31
Figure 31. Figure 31: Performance of Hypermodels in a dynamic setting with varying prior scale [eICU data with clustering bias]. We can see the prior scale 100 is best for the in-distribution data performance [plot (a)] while, prior scale of 10 is the best for the OOD data performance [plo…
Figure 32
Figure 32. Figure 32: Performance of UQ methodologies deteriorates with early stopping - Joint log-loss for in￾distribution (ID) data - (eICU data with clustering bias) 20 50 100 200 Training iterations 0.00 0.05 0.10 0.15 0.20 Standard Deviation [PITH_FULL_IMAGE:figures/full_fig_p051_32.png]
Figure 35
Figure 35. Figure 35: Sensitivity of UQ methodologies to random infer￾ence seeds - standard deviation of Joint log-loss for out-of-distribution (OOD) data - (eICU data with clus￾tering bias) 51 [PITH_FULL_IMAGE:figures/full_fig_p051_35.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

100 extracted references · 66 canonical work pages

  1. [1]

    C. C. Aggarwal, X. Kong, Q. Gu, J. Han, and S. Y. Philip. Active learning: A survey. In Data classification, pages 599–634. Chapman and Hall/CRC, 2014

    2014

  2. [2]

    Alesiani

    F. Alesiani. Implicit bilevel optimization: differentiating through bilevel optimization pro- gramming. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence , 2023

    2023

  3. [3]

    M. Alvo, D. Russo, and Y. Kanoria. Neural inventory control in networks via hindsight differentiable policy optimization. arXiv:2306.11246 [cs.LG] , 2024

    arXiv 2024

  4. [4]

    Amaran, N

    S. Amaran, N. V. Sahinidis, B. Sharda, and S. J. Bury. Simulation optimization: a review of algorithms and applications. Annals of Operations Research, 2016

    2016

  5. [5]

    B. Amos. Tutorial on amortized optimization. Foundations and Trends® in Machine Learn- ing, 2022

    2022

  6. [6]

    Astudillo, D

    R. Astudillo, D. Jiang, M. Balandat, E. Bakshy, and P. Frazier. Multi-step budgeted bayesian optimization with unknown evaluation costs. Advances in Neural Information Processing Systems 21 , 2021. 25

    2021

  7. [7]

    Avadhanula, O

    V. Avadhanula, O. A. Baki, H. Bastani, O. Bastani, C. Gocmen, D. Haimovich, D. Hwang, D. Karamshuk, T. Leeper, J. Ma, et al. Bandits for online calibration: An application to content moderation on social media platforms. arXiv preprint arXiv:2211.06516 , 2022

    arXiv 2022

  8. [8]

    Balachandar, N

    S. Balachandar, N. Garg, and E. Pierson. Domain constraints improve risk prediction when outcome data is missing. In The Twelfth International Conference on Learning Representa- tions, 2024

    2024

Show all 100 references
  1. [9]

    Barron, J

    A. Barron, J. Rissanen, and B. Yu. The minimum description length principle in coding and modeling. IEEE transactions on information theory , 44(6):2743–2760, 1998

    1998

  2. [10]

    A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind. Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research , 2017

    2017

  3. [11]

    Bengio, N

    Y. Bengio, N. L´ eonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv:1308.3432 [cs.LG] , 2013

    2013 arXiv

  4. [12]

    Bertsekas and J

    D. Bertsekas and J. N. Tsitsiklis. Neuro-dynamic programming. Athena Scientific, 1996

    1996

  5. [13]

    Blondel, Q

    M. Blondel, Q. Berthet, M. Cuturi, R. Frostig, S. Hoyer, F. Llinares-Lopez, F. Pedregosa, and J.-P. Vert. Efficient and modular implicit differentiation. In Advances in Neural Information Processing Systems 22, 2022

    2022

  6. [14]

    Blundell, J

    C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In Proceedings of the 32nd International Conference on Machine Learning , pages 1613–1622. PMLR, 2015

    2015

  7. [15]

    X.-R. Cao. Convergence of parameter sensitivity estimates in a stochastic experiment. IEEE Transactions on Automatic Control , 1985

    1985

  8. [16]

    E. Che, J. Dong, and H. Namkoong. Differentiable discrete event simulation for queuing network control. arXiv:2409.03740 [cs.LG] , 2024

    2024 arXiv

  9. [17]

    C.-H. Chen, J. Lin, E. Yucesan, and S. E. Chick. Simulation budget allocation for further enhancing the efficiency of ordinal optimization. Discrete Event Dynamic Systems , 10:251– 270, 2000

    2000

  10. [18]

    C.-H. Chen, S. E. Chick, L. H. Lee, and N. A. Pujowidianto. Ranking and selection: efficient simulation budget allocation. Handbook of Simulation Optimization , pages 45–80, 2015

    2015

  11. [19]

    A. P. Dawid. Statistical theory: The prequential approach. Journal of the Royal Statistical Society, Series A , 147:278–292, 1984

    1984

  12. [20]

    de Avila Belbute-Peres, K

    F. de Avila Belbute-Peres, K. Smith, K. Allen, J. Tenenbaum, and J. Z. Kolter. End-to-end differentiable physics for learning and control. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2018

    2018

  13. [21]

    F. Ding, M. Hardt, J. Miller, and L. Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems 34 , 34, 2021. 26

    2021

  14. [22]

    T. Du, Y. Li, J. Xu, A. Spielberg, K. Wu, D. Rus, and W. Matusik. D3 {pg}: Deep differen- tiable deterministic policy gradients, 2020

    2020

  15. [23]

    Dwaracherla, X

    V. Dwaracherla, X. Lu, M. Ibrahimi, I. Osband, Z. Wen, and B. V. Roy. Hypermodels for exploration. In International Conference on Learning Representations , 2020

    2020

  16. [24]

    D. J. Eckman and S. G. Henderson. Biased gradient estimators in simulation optimization. In 2020 Winter Simulation Conference (WSC) , pages 2935–2946, 2020

    2020

  17. [25]

    P. S. Efraimidis and P. G. Spirakis. Weighted random sampling with a reservoir. Information processing letters, 2006

    2006

  18. [26]

    Efroni, G

    Y. Efroni, G. Dalal, B. Scherrer, and S. Mannor. Multiple-step greedy policies in approximate and online reinforcement learning. In Advances in Neural Information Processing Systems , 2018

    2018

  19. [27]

    Efroni, M

    Y. Efroni, M. Ghavamzadeh, and S. Mannor. Online planning with lookahead policies. In Advances in Neural Information Processing Systems , 2020

    2020

  20. [28]

    Fortini and S

    S. Fortini and S. Petrone. Prediction-based uncertainty quantification for exchangeable se- quences. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 381(2247):20220142, 2023

    2023

  21. [29]

    P. I. Frazier. Bayesian Optimization , pages 255–278. 2018

    2018

  22. [30]

    P. I. Frazier, W. B. Powell, and S. Dayanik. A knowledge-gradient policy for sequential information collection. SIAM Journal on Control and Optimization , 47(5):2410–2439, 2008. doi: 10.1137/070693424

    2008 doi

  23. [31]

    M. C. Fu and S. D. Hill. Optimization of discrete event systems via simultaneous perturbation stochastic approximation. IIE transactions, pages 233–243, 1997

    1997

  24. [32]

    M. C. Fu and J.-Q. Hu. Conditional Monte Carlo: Gradient estimation and optimization applications. Springer Science & Business Media, 2012

    2012

  25. [33]

    Gal and Z

    Y. Gal and Z. Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. InProceedings of The 33rd International Conference on Machine Learning, pages 1050–1059, 2016

    2016

  26. [34]

    Ghadimi and G

    S. Ghadimi and G. Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341–2368, 2013

    2013

  27. [35]

    J. C. Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society, Series B , 41(2):148–177, 1979

    1979

  28. [36]

    Glasserman

    P. Glasserman. Gradient estimation via perturbation analysis . Springer Science & Business Media, 1990

    1990

  29. [37]

    Glasserman

    P. Glasserman. Derivative estimates from simulation of continuous-time Markov chains. Op- erations Research, 40(2):292–308, 1992. 27

    1992

  30. [38]

    Glasserman

    P. Glasserman. Monte Carlo methods in financial engineering , volume 53. Springer, 2004

    2004

  31. [39]

    P. Glynn. Optimization of stochastic systems via simulation. In 1989 Winter Simulation Conference Proceedings, 1989

    1989

  32. [40]

    P. W. Glynn. Likelilood ratio gradient estimation: an overview. In Proceedings of the 19th Conference on Winter Simulation , 1987

    1987

  33. [41]

    P. W. Glynn and S. Juneja. A large deviations perspective on ordinal optimization. In Proceedings of the 2004 Winter Simulation Conference , pages 577–586. IEEE, 2004

    2004

  34. [42]

    Greensmith, P

    E. Greensmith, P. L. Bartlett, and J. Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research , 2004

    2004

  35. [43]

    Grefenstette, B

    E. Grefenstette, B. Amos, D. Yarats, P. M. Htut, A. Molchanov, F. Meier, D. Kiela, K. Cho, and S. Chintala. Generalized inner loop meta-learning. arXiv preprint arXiv:1910.01727 , 2019

    1910 arXiv

  36. [44]

    Grover, J

    P. Grover, J. Xu, J. Tittelfitz, A. Cheng, Z. Li, J. Zablocki, J. Liu, and H. Zhou. Fraud dataset benchmark and applications. arXiv:2208.14417 [cs.LG] , 2022

    2022 arXiv

  37. [45]

    Heidelberger, X.-R

    P. Heidelberger, X.-R. Cao, M. A. Zazanis, and R. Suri. Convergence properties of infinites- imal perturbation analysis estimates. Management Science, 1988

    1988

  38. [46]

    Y. Ho, X. Cao, and C. Cassandras. Infinitesimal and finite perturbation analysis for queueing networks. Automatica, 19(4):439–445, 1983

    1983

  39. [47]

    Y. C. Ho, M. A. Eyler, and T. T. Chien. A new approach to determine parameter sensitivities of transfer lines. Management Science, 29(6), 1983

    1983

  40. [48]

    L. J. Hong, B. L. Nelson, and J. Xu. Discrete optimization via simulation. In Handbook of Simulation Optimization , pages 9–44. Springer, 2015

    2015

  41. [49]

    Houlsby, F

    N. Houlsby, F. Husz´ ar, Z. Ghahramani, and M. Lengyel. Bayesian active learning for classi- fication and preference learning. arXiv:1112.5745 [cs.CV] , 2011

    2011 arXiv

  42. [50]

    Huang, R

    S. Huang, R. F. J. Dossa, A. Raffin, A. Kanervisto, and W. Wang. The 37 implementation details of proximal policy optimization, 2022. URL https://iclr-blog-track.github.io/ 2022/03/25/ppo-implementation-details/. ICLR 2022 Blog Track

    2022

  43. [51]

    Huang, Y

    Z. Huang, Y. Hu, T. Du, S. Zhou, H. Su, J. B. Tenenbaum, and C. Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. In International Conference on Learning Representations, 2021

    2021

  44. [52]

    E. Jang, S. Gu, and B. Poole. Categorical reparameterization with Gumbel-softmax. In International Conference on Learning Representations , 2017

    2017

  45. [53]

    M. E. Johnson and J. Jackman. Infinitesimal perturbation analysis: a tool for simulation. Journal of the Operational Research Society , 40(3):243–254, 1989

    1989

  46. [54]

    Kim and B

    S.-H. Kim and B. L. Nelson. Recent advances in ranking and selection. In Proceedings of the 2007 Winter Simulation Conference , 2007. 28

    2007

  47. [55]

    D. P. Kingma and M. Welling. Auto-encoding variational bayes, 2014

    2014

  48. [56]

    Kirsch, J

    A. Kirsch, J. van Amersfoort, and Y. Gal. BatchBALD: Efficient and diverse batch acquisition for deep bayesian active learning. In Advances in Neural Information Processing Systems , volume 32, 2019

    2019

  49. [57]

    Kotary, M.-H

    J. Kotary, M.-H. N. Dinh, and F. Fioretto. Backpropagation of unrolled solvers with folded optimization. In International Joint Conference on Artificial Intelligence , 2023

    2023

  50. [58]

    Lakshminarayanan, A

    B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31st International Conference on Neural Information Processing Systems , page 6405–6416, 2017

    2017

  51. [59]

    Lorraine, P

    J. Lorraine, P. Vicol, and D. Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics , 2020

    2020

  52. [60]

    C. J. Maddison, A. Mnih, and Y. W. Teh. The concrete distribution: A continuous relaxation of discrete random variables. In Proceedings of the Fifth International Conference on Learning Representations, 2017

    2017

  53. [61]

    Madeka, K

    D. Madeka, K. Torkkola, C. Eisenach, A. Luo, D. P. Foster, and S. M. Kakade. Deep inventory management. arXiv:2210.03137 [cs.LG] , 2022

    2022 arXiv

  54. [62]

    C. C. Margossian. A review of automatic differentiation and its efficient implementation. WIREs Data Mining and Knowledge Discovery , 2019

    2019

  55. [63]

    Mnih and K

    A. Mnih and K. Gregor. Neural variational inference and learning in belief networks. In Pro- ceedings of the 31st International Conference on Machine Learning , volume 32 of Proceedings of Machine Learning Research, pages 1791–1799, 2014

    2014

  56. [64]

    Mohamed, M

    S. Mohamed, M. Rosca, M. Figurnov, and A. Mnih. Monte Carlo gradient estimation in machine learning. Journal of Machine Learning Research , 21(1), 2020

    2020

  57. [65]

    M. A. Z. Mora, M. Peychev, S. Ha, M. Vechev, and S. Coros. Pods: Policy optimization via differentiable simulation. In Proceedings of the 38th International Conference on Machine Learning, 2021

    2021

  58. [66]

    Mullainathan and Z

    S. Mullainathan and Z. Obermeyer. Diagnosing physician error: A machine learning approach to low-value health care. The Quarterly Journal of Economics , 2022

    2022

  59. [67]

    M¨ uller, N

    S. M¨ uller, N. Hollmann, S. P. Arango, J. Grabocka, and F. Hutter. Transformers can do bayesian inference. In Proceedings of the Tenth International Conference on Learning Repre- sentations, 2022

    2022

  60. [68]

    Z. Nado, N. Band, M. Collier, J. Djolonga, M. Dusenberry, S. Farquhar, A. Filos, M. Havasi, R. Jenatton, G. Jerfel, J. Liu, Z. Mariet, J. Nixon, S. Padhy, J. Ren, T. Rudner, Y. Wen, F. Wenzel, K. Murphy, D. Sculley, B. Lakshminarayanan, J. Snoek, Y. Gal, and D. Tran. Uncertain...

    2021 arXiv

  61. [69]

    Namkoong, S

    H. Namkoong, S. Daulton, and E. Bakshy. Distilled thompson sampling: Practical and efficient thompson sampling via imitation learning. arXiv:2011.14266 [cs.LG] , 2020

    2011 arXiv

  62. [70]

    Nguyen and A

    T. Nguyen and A. Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. In Proceedings of the 39th International Conference on Machine Learning, 2022

    2022

  63. [71]

    of Medicine (US) Committee on Understanding, E

    I. of Medicine (US) Committee on Understanding, E. Racial, and E. D. in Health Care. Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care . 2003

    2003

  64. [72]

    Osband and B

    I. Osband and B. Van Roy. Bootstrapped Thompson sampling and deep exploration. arXiv:1507.00300 [stat.ML] , 2015

    2015 arXiv

  65. [73]

    Osband, J

    I. Osband, J. Aslanides, and A. Cassirer. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems 31 , volume 31, 2018

    2018

  66. [74]

    Osband, Z

    I. Osband, Z. Wen, S. M. Asghari, V. Dwaracherla, X. Lu, M. Ibrahimi, D. Lawson, B. Hao, B. O’Donoghue, and B. V. Roy. The Neural Testbed: Evaluating Joint Predictions. In Advances in Neural Information Processing Systems , 2022

    2022

  67. [75]

    Osband, Z

    I. Osband, Z. Wen, S. M. Asghari, V. Dwaracherla, X. Lu, and B. Van Roy. Evaluating high- order predictive distributions in deep learning. In Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence , pages 1552–1560, 2022

    2022

  68. [76]

    Osband, Z

    I. Osband, Z. Wen, S. M. Asghari, V. Dwaracherla, M. Ibrahimi, X. Lu, and B. V. Roy. Epistemic neural networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  69. [77]

    Ovadia, E

    Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshmi- narayanan, and J. Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. In Proceedings of the 33rd International Conference on Neu- ral Info...

    2019

  70. [78]

    Papini, D

    M. Papini, D. Binaghi, G. Canonaco, M. Pirotta, and M. Restelli. Stochastic variance-reduced policy gradient. In Proceedings of the 35th International Conference on Machine Learning , pages 4026–4035, 2018

    2018

  71. [79]

    M. B. Paulus, C. J. Maddison, and A. Krause. Rao-blackwellizing the straight-through gumbel-softmax gradient estimator. In International Conference on Learning Representa- tions, 2021

    2021

  72. [80]

    T. J. Pollard, A. E. Johnson, J. D. Raffa, L. A. Celi, R. G. Mark, and O. Badawi. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Scientific data , 5(1):1–13, 2018

    2018

  73. [81]

    C. E. Rasmussen. Gaussian processes in machine learning. In Summer School on Machine Learning, pages 63–71. Springer, 2003

    2003

  74. [82]

    C. E. Rasmussen. Gaussian processes for machine learning. pages 63–71, 2006. 30

    2006

  75. [83]

    J. Ren, X. Feng, B. Liu, X. Pan*, Y. Fu, L. Mai, and Y. Yang. TorchOpt: An Efficient Library for Differentiable Optimization. Journal of Machine Learning Research , 24(367):1–14, 2023

    2023

  76. [84]

    D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning, pages 1278–1286, 2014

    2014

  77. [85]

    H. V. Roberts. Probabilistic prediction. Journal of the American Statistical Association , 60 (309):50–62, 1965

    1965

  78. [86]

    Scieur, G

    D. Scieur, G. Gidel, Q. Bertrand, and F. Pedregosa. The curse of unrolling: Rate of dif- ferentiating through optimization. In Advances in Neural Information Processing Systems , 2022

    2022

  79. [87]

    B. Settles. Active learning literature survey. 2009

    2009

  80. [88]

    Seyyed-Kalantari, H

    L. Seyyed-Kalantari, H. Zhang, M. B. McDermott, I. Y. Chen, and M. Ghassemi. Under- diagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature medicine, 2021

    2021

  81. [89]

    Shaban, C.-A

    A. Shaban, C.-A. Cheng, N. Hatch, and B. Boots. Truncated back-propagation for bilevel optimization. In Proceedings of the 21st International Conference on Artificial Intelligence and Statistics , 2018

    2018

  82. [90]

    Straw and H

    I. Straw and H. Wu. Investigating for bias in healthcare algorithms: a sex-stratified analysis of supervised machine learning models in liver disease prediction.BMJ health & care informatics, 2022

    2022

  83. [91]

    H. J. Suh, M. Simchowitz, K. Zhang, and R. Tedrake. Do differentiable simulators give better policy gradients? In Proceedings of the 39th International Conference on Machine Learning , volume 162, 2022

    2022

  84. [92]

    R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  85. [93]

    R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforce- ment learning with function approximation. In Advances in Neural Information Processing Systems, volume 12, 1999

    1999

  86. [94]

    Tucker, A

    G. Tucker, A. Mnih, C. J. Maddison, J. Lawson, and J. Sohl-Dickstein. Rebar: Low-variance, unbiased gradient estimates for discrete latent variable models. In Advances in Neural Infor- mation Processing Systems, 2017

    2017

  87. [95]

    C. Wang, S. Sun, and R. Grosse. Beyond marginal uncertainty: How accurately can Bayesian regression models estimate posterior predictive correlations? In International Conference on Artificial Intelligence and Statistics , pages 2476–2484, 2021

    2021

  88. [96]

    R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8:229–256, 1992. 31

    1992

  89. [97]

    C.-J. Wu, D. Brooks, K. Chen, D. Chen, S. Choudhury, M. Dukhan, K. Hazelwood, E. Isaac, Y. Jia, B. Jia, et al. Machine learning at facebook: Understanding inference at the edge. In 2019 IEEE international symposium on high performance computer architecture (HPCA) , pages 331–3...

    2019

  90. [98]

    First-order perturbation analysis of a simple multi-class finite source queue

    Xi-Ren Cao. First-order perturbation analysis of a simple multi-class finite source queue. Performance Evaluation, 7(1):31–41, 1987. ISSN 0166-5316

    1987

  91. [99]

    S. M. Xie and S. Ermon. Reparameterizable subset sampling via continuous relaxations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence , IJCAI’19, page 3919–3925, 2019

    2019

  92. [100]

    Z m+τY i=m pϵ(yi − f (Xi)) ! dµ(f |D0) # . (19) Osband et al. [75] referred to the above metric as “ marginal log-loss

    J. Xu, V. Makoviychuk, Y. Narang, F. Ramos, W. Matusik, A. Garg, and M. Macklin. Ac- celerated policy learning with parallel differentiable simulation. In International Conference on Learning Representations, 2021. 32 A Smoothing the recall objective Performance metric, such a...

    2021

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.