Pith. sign in

REVIEW 2 major objections 5 minor 28 references

When three experiments are better than two: Avoiding intractable correlated aleatoric uncertainty by leveraging a novel bias--variance tradeoff

T0 review · 2 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read To beat BALD under correlated noise, compare the error between rounds and estimate the bias quadratically.

desk verdict Correct core identity and a promising difference trick, but the estimator has an unanalyzed noise artifact and the batched experiments don't actually compare to the claimed baselines. read the letter →

arxiv 2509.04363 v1 pith:QWMZASMA submitted 2025-09-04 cs.LG

classification cs.LG
keywords activelearningbias-variancetradeoffaleatoricuncertaintyheteroskedasticnoisecorrelateddeepensemblesacquisitionfunctionsbatchedqueryselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that in regression active learning, the expected mean squared error splits into epistemic variance, squared bias, and aleatoric variance, and that the aleatoric term cancels when you compare two consecutive experimental rounds. This makes the change in bias between rounds a usable acquisition signal, which the authors exploit in a third round of experiments. To estimate the bias without knowing the true function, they introduce a cobias-covariance matrix whose off-diagonal entries pair historical data points quadratically, and they use its eigendecomposition to select whole batches of experiments. In batched settings with correlated heteroskedastic noise, their difference-based method with a quadratic estimator outperforms BALD and Least Confidence. The practical upshot would be an active learning rule that targets reducible error directly rather than only model uncertainty.

What carries the argument

The central object is the cobias-covariance matrix Omega_k = Sigma_Fk + Delta_k + Sigma_Y, where Delta_k is the rank-1 outer product of the bias vector and Sigma_Y is the possibly correlated aleatoric covariance. The bias-variance tradeoff makes the trace of Omega_k equal to the expected mean squared error; the difference operator between rounds cancels Sigma_Y, so reducible error can be targeted without modeling intractable correlated noise; and the eigendecomposition of Omega_k or its round-to-round difference provides a principled batch-selection rule by choosing the coordinate with largest eigenvector magnitude for each dominant eigenvalue. Quadratic estimation predicts entries of Delta_

What would settle it

Take a synthetic regression task with known true function and strong heteroskedastic noise, give the algorithm only one replicate per queried point, compute the bias estimate from the paper's Eq. (35) on a grid, and compare its ranking against the true squared bias. If selecting the top-ranked points does not reduce mean squared error faster than random selection in a batched run, the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that the expected mean squared error between an ensemble predictor and a noisy oracle decomposes as epistemic variance plus squared bias plus aleatoric variance, and that the aleatoric part drops out of the difference between consecutive rounds. The authors exploit this by building acquisition functions on the difference operator applied to variance, bias-squared, or full pointwise expected mean squared error. To make this practical, they introduce a cobias-covariance matrix whose trace is the expected mean squared error, estimate the bias contribution quadratically through a symmetric neural network rather than squaring per-coordinate bias estimates, and select batches

Load-bearing premise

The load-bearing premise is that the bias, the gap between the model's expected prediction and the true signal, can be read off from historical noisy labels even when only a few replicate measurements exist.

Editorial extensions

If this is right

  • Bias-based acquisition functions reduce mean squared error faster than Least Confidence, BALD, or random selection on the studied noisy and noiseless toy systems.
  • The aleatoric term cancels in the difference between rounds, so correlated noise does not need to be explicitly modeled to guide acquisition.
  • Quadratic estimation of the cobias matrix stabilizes downstream eigendecompositions relative to direct per-coordinate bias estimation, especially when only a few initial data points are available.
  • In the low-initial-data batched regime, quadratic estimation with difference-based acquisition and eigendecomposition is the only non-oracle method that beats random selection in the paper's experiments.
  • The method is developed for regression problems; extending to classification would require handling non-symmetric Bregman divergences.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: using higher-order finite differences over more than two past rounds could sharpen the estimate of how fast the error decreases, potentially improving the single-point acquisition regime where the paper reports difference methods give little benefit.
  • Editorial inference: the same cobias-covariance machinery could transfer to other symmetric bilinear losses, but not to KL-based Bregman divergences, whose dual means lack a symmetric bilinear structure.
  • Editorial inference: weighting the quadratic estimator's training pairs by replicate counts, something the paper notes is unaccounted for, would likely reduce bias-estimate contamination and narrow the gap to the oracle curves.
  • Editorial inference: when the model class cannot represent the true signal, the acquisition rule will keep querying irreducible-bias regions; adding a stopping rule based on estimated residual bias would conserve experimental budget.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes an active-learning framework, AICAU, for regression with heteroskedastic and correlated aleatoric noise. It derives the pointwise bias-variance decomposition (Eq. 5), extends it to a cobias-covariance matrix (Eq. 11), and introduces a difference acquisition function that cancels the aleatoric term by subtracting consecutive rounds (Eqs. 8-9). To make the method practical, the authors train a symmetric neural-network estimator for the bias-squared matrix Δ_k (Eq. 13) and batch queries via the eigendecomposition of the cobias-covariance matrix (Eq. 20). Experiments on a 2D toy problem compare the method against BALD, least confidence, and random selection for noiseless (Type I), uncorrelated-noise (Type II), and correlated-noise (Type III) settings. The headline claim is that, in the batched setting, quadratic estimation combined with difference-PEMSE and eigendecomposition outperforms canonical active-learning methods.

Significance. The mathematical decomposition in Sections 2.2 and 3.1 is sound under the stated independence assumptions, and the cobias-covariance formulation is a natural and potentially useful extension for batched active learning. The difference operator elegantly cancels the intractable correlated aleatoric term, which is a genuine conceptual advance. The paper is also commendably transparent about its derivations and provides an appendix with numerical details and a reproducibility statement. However, the practical bias estimator used in the experiments contains a finite-replicate artifact that is not acknowledged in the main text. Because the reported advantage over BALD and least confidence may be driven by this artifact rather than by the cobias structure, the empirical claims are not yet established.

major comments (2)
  1. [Appendix D.3, Eq. (35); §3.2 and §4.2] The plug-in estimator δhat_i = (1/K)Σ_k f_k(x_i) − (1/N_i)Σ_r y_r(x_i) is unbiased for δ_i, but its square has expectation δ_i^2 + σ_Y^2(x_i)/N_i + Var(μhat_F(x_i)). The diagonal of the estimated Δ_k therefore contains a term that decreases when N_i increases. Under the difference operator κ[·], any point whose replicate count grows between rounds receives a positive contribution to κ(Δ) from variance reduction alone, independent of true bias reduction. In the Section 4.2 batched experiments, most points have N_i=1, and the toy σ_Y^2 is non-negligible near μY=0 (Table 2), so the artifact is comparable in size to the bias signal. The quadratic estimator and eigendecomposition are trained on these contaminated targets, so the reported advantage of quadratic difference-PEMSE over BALD/LC in Figure 5 may be an artifact. The Discussion (Section 5) mentions that the estimator should be weighte
  2. [Section 3.2, Appendix A, and Eq. (12)] The practical pipeline is under-specified regarding how Σ_Y enters the computed Ω_k. Equation (12) defines Ω_k = Σ_Fk + Δ_k + Σ_Y, but the quadratic estimator Q is trained to predict entries of Δ_k, and Appendix D.3 estimates δ_k, not Σ_Y. For Type III problems, Σ_Y is a full correlated matrix that is not estimated anywhere in the main protocol. If the reported method is difference-PEMSE, Σ_Y cancels in Ω_{k−1} − Ω_k, but the PEMSE variants and the eigendecomposition batching of Ω_k in Section 4.2 require Σ_Y. The manuscript should state explicitly whether Σ_Y is assumed known, estimated, or avoided by working exclusively with differences. Without this clarification, the experimental protocol is not fully reproducible and it is unclear what the batching experiments actually compute.
minor comments (5)
  1. [Table 1 caption] Typo: 'Acqusition' should be 'Acquisition'.
  2. [Table 2] The formula for W(x) is ambiguous: is it ε√(1−μY²)/10 or ε√((1−μY²)/10)? Please disambiguate.
  3. [Figures 4 and 5] The experimental plots show means over 10 ensembles but no error bars or statistical significance tests. Given the small reported performance differences (e.g., the 'reproducible small benefits' of eigendecomposition), the paper should include variance estimates or a paired test to support the comparison.
  4. [Section 3.1] Equation (11) silently assumes independence between F_k(x) and Y(x) for all x, x*. This is stated for Eq. (5) but should be explicitly restated for the cobias derivation, since cross terms are dropped.
  5. [Appendix D.3] The title 'Bias-first calculation of ω_k(x,x*)' is misleading because Eq. (35) only defines δhat_i; the relation between δhat_i and ω_k (or Δ_k) is not written out. Please add the explicit formula used to build Δ_k from δhat.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: derivations are algebraic identities from stated definitions; practical estimators are supervised, and self-citations are contextual only.

full rationale

The paper's central theoretical claims are derived from explicit definitions rather than assumed. Equation (5) is the standard bias–variance decomposition obtained by expanding E[(F_k(x)-Y(x))^2] with the independence assumption, and Equation (11) is the same 'add zero' expansion for two distinct points, yielding the cobias–covariance identity ω_k = σ_F + δδ^T + σ_Y. These are algebraic identities, not circular definitions. The acquisition functions in Table 1 and the difference operator κ in Equation (8) are direct definitions of the proposed strategies. The quadratic estimator Q in Equation (13) is a symmetric neural-network matrix-completion model trained on observed (historical) entries of Δ_k; using it to predict missing entries is standard supervised learning, not a fitted parameter renamed as a prediction. The eigendecomposition batching in Section 3.3 follows from the positive-semidefinite structure of Ω^(k) and the rank bound in Equation (18); it is a mechanism derived from the earlier identity, not an imported uniqueness result. Self-citations such as [14] (a software library) and [17] (a unification of AL methods) are contextual and do not carry the paper's load-bearing argument. The skeptic's concern about Equation (35) — that squared bias estimates contain a finite-replicate variance term σ_Y^2/N_i, which can make difference-based acquisition reward replicate-count increases — is a genuine statistical estimation artifact and an important correctness risk for the empirical comparisons, but it is not circularity: the method's estimators are not defined as the quantities they claim to predict, and the manuscript explicitly acknowledges in the Discussion and Limitations that replicate-count variation is not accounted for and that bias estimation 'can be biased.' The claimed advantages over BALD and Least Confidence are empirical outcomes measured against the true μ_Y, not consequences of a definitional identity. Therefore no circular step meeting the stated evidentiary bar is present, and the paper is self-contained against its own derivation chain.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central contribution is a method, not a theory. The mathematical identities are standard, but the practical success depends on treating noisy labels as a proxy for the true function and on a low-rank assumption for the bias matrix.

free parameters (5)
  • Ensemble size K = 5
    Number of neural networks in the deep ensemble; set fixed across experiments.
  • Embedding network hidden dims = 64, 64, 32; output 16
    Architecture for the quadratic estimator ψ(x); chosen without a search and likely tuned on the toy data.
  • Toy noise scale (Type II/III) = sqrt(1 - μY^2)/10
    Heteroskedastic noise model; this makes the problem favorable for bias-based methods.
  • Correlation kernel (Type III) = exp(-2|x-x*|/π)
    Defines correlated noise in the toy system.
  • Evaluation grid = 50 x 50 = 2500 points
    Discretization of the state space.
assumptions (5)
  • standard math Cov(F_k(x), Y(x)) = 0 (independence of model and label noise)
    Required for the bias-variance decomposition in Eq (5) and the cobias-covariance in Eq (11); standard but not always true.
  • domain assumption Aleatoric noise has zero mean and finite variance (Eq 3)
    Assumed for the noise model; standard.
  • ad hoc to paper Bias can be estimated from observed labels via Eq (35)
    Uses the average of observed y's as a proxy for μY; the core practical assumption of the method.
  • domain assumption The bias matrix Δ_k can be approximated by a low-rank embedding Q(x,x*) = ψ(x)^T ψ(x*)
    Exact Δ is rank-1, but the estimator assumes a parametric low-rank structure that must generalize.
  • ad hoc to paper The one-step difference κ[g_k] approximates the negative gradient ∂τ_k/∂k
    Finite-difference heuristic; assumed valid when rounds are close and training is stable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When three experiments are better than two: Avoiding intractable correlated aleatoric uncertainty by leveraging a novel bias--variance tradeoff." pith.science (2026). https://pith.science/paper/QWMZASMA

@misc{pith2026250904363,
  author       = {Pith},
  title        = {Pith review of: When three experiments are better than two: Avoiding intractable correlated aleatoric uncertainty by leveraging a novel bias--variance tradeoff},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QWMZASMA}},
  note         = {Machine review of arXiv:2509.04363}
}
read the original abstract

Real-world experimental scenarios are characterized by the presence of heteroskedastic aleatoric uncertainty, and this uncertainty can be correlated in batched settings. The bias--variance tradeoff can be used to write the expected mean squared error between a model distribution and a ground-truth random variable as the sum of an epistemic uncertainty term, the bias squared, and an aleatoric uncertainty term. We leverage this relationship to propose novel active learning strategies that directly reduce the bias between experimental rounds, considering model systems both with and without noise. Finally, we investigate methods to leverage historical data in a quadratic manner through the use of a novel cobias--covariance relationship, which naturally proposes a mechanism for batching through an eigendecomposition strategy. When our difference-based method leveraging the cobias--covariance relationship is utilized in a batched setting (with a quadratic estimator), we outperform a number of canonical methods including BALD and Least Confidence.

Figures

Figures reproduced from arXiv: 2509.04363 by the authors.

Figure 2
Figure 2. Assessment of acquisition functions proposed in Table 1 with perfect information available. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Diagram of active learning procedure. Alternative cobias–covariance calculation in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Assessment of BR and PEMSE acquisition function using different methods at to estimate [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Assessment of PEMSE and difference-PEMSE acquisition functions using different methods [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 21 canonical work pages

  1. [1]

    A survey of deep active learning

    Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. ACM computing surveys (CSUR), 54 (9):1–40, 2021

  2. [2]

    Active learning literature survey

    Burr Settles. Active learning literature survey. 2009

  3. [3]

    Aleatory or epistemic? does it matter? Structural Safety, 31(2):105–112, 2009

    Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? does it matter? Structural Safety, 31(2):105–112, 2009

  4. [4]

    The future of machine learning within target identification: Causality, reversibility, and druggability

    Jake P Taylor-King, Michael Bronstein, and David Roblin. The future of machine learning within target identification: Causality, reversibility, and druggability. Clinical Pharmacology & Therapeutics, 2024

  5. [5]

    Recover identifies synergistic drug combinations in vitro through sequential model optimization

    Paul Bertin, Jarrid Rector-Brooks, Deepak Sharma, Thomas Gaudelet, Andrew Anighoro, Torsten Gross, Francisco Mart ´ınez-Pe˜na, Eileen L Tang, MS Suraj, Cristian Regep, et al. Recover identifies synergistic drug combinations in vitro through sequential model optimization. Cell Reports Methods, 3(10), 2023

  6. [6]

    No foundations without foundations–why semi-mechanistic models are essential for regulatory biology

    Luka Kovaˇcevi´c, Thomas Gaudelet, James Opzoomer, Hagen Triendl, John Whittaker, Caroline Uhler, Lindsay Edwards, and Jake P Taylor-King. No foundations without foundations–why semi-mechanistic models are essential for regulatory biology. arXiv preprint arXiv:2501.19178, 2025

  7. [7]

    scperturb: harmonized single-cell perturbation data

    Stefan Peidli, Tessa D Green, Ciyue Shen, Torsten Gross, Joseph Min, Samuele Garda, Bo Yuan, Linus J Schumacher, Jake P Taylor-King, Debora S Marks, et al. scperturb: harmonized single-cell perturbation data. Nature Methods, 21(3):531–540, 2024

  8. [8]

    sctrends: A living review of commercial single-cell and spatial’omic technologies

    Joachim De Jonghe, James W Opzoomer, Amaia Vilas-Zornoza, Benedikt S Nilges, Peter Crane, Marco Vicari, Hower Lee, David Lara-Astiaso, Torsten Gross, J¨org Morf, et al. sctrends: A living review of commercial single-cell and spatial’omic technologies. Cell Genomics, 4(12), 2024

Show all 28 references
  1. [9]

    A community effort to track commercial single-cell and spatial’omic technologies and business trends

    Joachim De Jonghe, James W Opzoomer, Amaia Vilas-Zornoza, Peter Crane, Benedikt S Nilges, Marco Vicari, Hower Lee, David Lara-Astiaso, Torsten Gross, J¨org Morf, et al. A community effort to track commercial single-cell and spatial’omic technologies and business trends. nature...

  2. [10]

    Sequential exploration of unknown multi-dimensional functions as an aid to optimization

    IP Schagen. Sequential exploration of unknown multi-dimensional functions as an aid to optimization. IMA Journal of Numerical Analysis, 4(3):337–347, 1984

  3. [11]

    Jones, Matthias Schonlau, and William J

    Donald R. Jones, Matthias Schonlau, and William J. Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization , 1998. doi: 10.1023/a: 1008306431147

  4. [12]

    On a measure of the information provided by an experiment

    Dennis V Lindley. On a measure of the information provided by an experiment. The Annals of Mathematical Statistics, pages 986–1005, 1956. 11

  5. [13]

    Query by committee

    H Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee. In Pro- ceedings of the fifth annual workshop on Computational learning theory , pages 287–294, 1992

  6. [14]

    Pyrelational: A library for active learning research and development

    Paul Scherer, Thomas Gaudelet, Alison Pouplin, Jyothish Soman, Lindsay Edwards, Jake P Taylor-King, et al. Pyrelational: A library for active learning research and development. arXiv preprint arXiv:2205.11117, 2022

  7. [15]

    What uncertainties do we need in bayesian deep learning for computer vision? In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M

    Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett, editors,Advances in Neural Information Processi...

  8. [16]

    Understanding measures of uncertainty for adversarial example detection

    Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Conference on Uncertainty in Artificial Intelligence , pages 560–569. AUAI Press, 2018. URL http: //auai.org/uai...

  9. [17]

    Unifying approaches in active learning and active sampling via fisher information and information-theoretic quantities

    Andreas Kirsch and Yarin Gal. Unifying approaches in active learning and active sampling via fisher information and information-theoretic quantities. Transactions on Machine Learning Research, 2022

  10. [18]

    A generalized bias-variance decomposition for bregman divergences

    David Pfau. A generalized bias-variance decomposition for bregman divergences. Unpublished Manuscript, 2013

  11. [19]

    Understanding the bias-variance tradeoff of bregman divergences

    Ben Adlam, Neha Gupta, Zelda Mariet, and Jamie Smith. Understanding the bias-variance tradeoff of bregman divergences. arXiv preprint arXiv:2202.04167, 2022

  12. [20]

    Bayesian active learning for classification and preference learning, 2011

    Neil Houlsby, Ferenc Husz´ar, Zoubin Ghahramani, and M´at´e Lengyel. Bayesian active learning for classification and preference learning, 2011

  13. [21]

    Speedup matrix completion with side information: Application to multi-label learning

    Miao Xu, Rong Jin, and Zhi-Hua Zhou. Speedup matrix completion with side information: Application to multi-label learning. Advances in neural information processing systems, 26, 2013

  14. [22]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

  15. [23]

    Uncertainty estimates of predictions via a general bias-variance decomposition

    Sebastian G Gruber and Florian Buettner. Uncertainty estimates of predictions via a general bias-variance decomposition. arXiv preprint arXiv:2210.12256, 2022

  16. [24]

    Rethinking bias-variance trade-off for generalization of neural networks

    Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade-off for generalization of neural networks. In International Conference on Machine Learning, pages 10767–10777. PMLR, 2020

  17. [25]

    Bias-variance decompositions: the exclusive privilege of bregman divergences

    Tom Heskes. Bias-variance decompositions: the exclusive privilege of bregman divergences. arXiv preprint arXiv:2501.18581, 2025. 12 A Further details of numerical results For the purposes of evaluating different strategies, we do not evaluateLk, but the key performance metric ...

  18. [26]

    Strategies with perfect information on unknown PEMSE and bias, ‘cheating’, to motivate theoretical results in Section 2.4

  19. [27]

    Strategies which use direct estimation of the PEMSE and bias via a Gaussian process

  20. [28]

    add zero

    Strategies which use quadratic estimation of the PEMSE and bias via symmetric neural network for matrix completion. (1) In the first case of perfect information or ‘cheating’, we call upon the oracle to provide realisations y ∼ Y (x) which we use to impute the PEMSE or bias re...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.