REVIEW 3 major objections 4 minor 47 references
A Heisenberg-esque Uncertainty Principle for Simultaneous (Machine) Learning and Error Assessment?
T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper proves that under squared loss the squared correlation between an unbiased learner's actual error and any unbiased estimate of that error is no more than the learner's relative regret, making optimal unbiased learning…
desk verdict The central inequality is real but elementary—a clean restatement of UMVUE in learning language, honestly labeled as low-hanging fruit. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The proof runs on a one-parameter family of learners $\hat{Q}_\lambda = \hat{Q} - \lambda\,\hat{\delta}_{\hat{Q}}$. Unbiasedness of both components keeps every $\hat{Q}_\lambda$ in the allowed learner class, so its risk cannot fall below the optimal risk $R^{\mathrm{opt}}_s$; minimizing the resulting quadratic in $\lambda$ yields $$$R^{{\mathrm{opt}}$}_s \le V_s(\delta_{\hat{Q}})\left[1-\$rho_s^{2}$(\delta_{\hat{Q}},\hat{\delta}_{\hat{Q}})\right],$$ which rearranges to the relative-regret bound. The other load-bearing object is relative regret itself, which measures how much of a learner's squared-error risk is excess over the best attainable; in the paper's regression example this quantity is exactly the squared correlation between the learner and its residual-based error assessor.
What would settle it
A direct check is available in the paper's own $n=2$ weighted-regression setting: across a grid of weights, the squared correlation between the learner and its residual-based assessor must equal the relative regret exactly, so any numerical discrepancy would indicate a flaw in the derivation. To test whether the unbiasedness assumption is load-bearing, simulate a biased learner (for example ridge regression) and compute whether squared correlation between actual and estimated error ever exceeds relative regret; if it does, the exact theorem cannot extend to biased learners.
Extended reading notes
Core claim
The central claim is inequality (14): for any learner $\hat{Q}$ that is unbiased for its target $Q$ and any assessor $\hat{\delta}_{\hat{Q}}$ that is unbiased for the actual error $\delta_{\hat{Q}} = \hat{Q}-Q$, under squared loss, $$\$rho_s^{2}$(\delta_{\hat{Q}}, \hat{\delta}_{\hat{Q}}) \le \mathrm{RR}_s(\hat{Q}) = \frac{R_s(\hat{Q}) - $R^{{\mathrm{opt}}$}_s}{R_s(\hat{Q})}$$ for every distribution $P_s$ in the family. Since the relative regret vanishes only at an optimal learner, an optimal unbiased learner has zero squared correlation with every unbiased error assessor. The paper derives the inequality from the optimality of $R^{\mathrm{opt}}$ applied to the perturbed learner $\hat{Q} - \lambda\,\hat{\delta}_{\hat{Q}}$, and shows in the $n=2$ weighted-regression example that the bound is tight: squared correlation equals relative regret exactly for any weights.
Load-bearing premise
The bound collapses without the requirement that both the learner and the error assessor are unbiased for every data-generating distribution in the family; the paper relaxes this only asymptotically, not exactly.
Editorial extensions
If this is right
- An optimal unbiased learner admits no unbiased error assessor that is correlated with the actual error; the squared correlation is exactly zero.
- Independence between estimated and actual prediction errors, as found for cross-validation and similar methods, is a predicted consequence of near-optimal learning rather than a defect of those methods.
- Any unbiased error assessor with nonzero correlation with the actual error certifies that the learner is suboptimal and can be exploited to improve the learner by subtracting a suitable multiple of the assessor.
- Quantitatively, raising the squared correlation of an error assessor by a given amount forces the learner's relative regret to be at least that large.
- In practice the principle favors deliberately leaving some information unused—for example by introducing randomness or suboptimal regularization—if reliable error assessment is the goal.
Reading between the lines
- Because the exact theorem needs unbiasedness, biased learners such as ridge or lasso fall outside its scope; the paper's asymptotic version, rather than the exact bound, is the relevant statement for them.
- A direct empirical test: in a simulated regression with known truth, vary the regularization strength and estimate the squared correlation between cross-validated error and true error; the correlation should decline as the learner approaches the empirical risk minimum.
- The same trade-off may extend to Bayesian or decision-theoretic settings where "relevance" is measured by dependence rather than correlation, but the paper does not establish such extensions.
- If the principle is general, data splitting is not just a computational convenience but a necessity: part of the data must be spent on error assessment because the part spent on optimization carries no independent information about the remaining error.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper, framed as an essay for a special issue in memory of C. R. Rao, proposes a Heisenberg-like uncertainty principle for simultaneous learning and error assessment under squared loss. The central mathematical result is Theorem 1 (Section 5): for any unbiased learner Qhat and any unbiased zero-mean error assessor delta_hat_Qhat, the squared correlation between the actual error delta_Qhat and the assessor is bounded above by the learner's relative regret, RR_s(Qhat) = (R_s(Qhat) - R_opt_s)/R_s(Qhat). The proof constructs the perturbed learner Qhat - lambda*delta_hat, which remains unbiased, and minimizes its variance. A corollary states that an optimal unbiased learner cannot have an unbiased, correlated error assessor. Section 6 extends the result asymptotically by allowing O(e_iota) bias in the learner and assessor. The paper also includes a heteroscedastic regression example, a joint-replication calculation for the normal mean problem, and a discussion of quantum covariance mechanisms to connect the inequality with the Cramer-Rao bound and the Heisenberg uncertainty principle. Sections 10-12 are primarily philosophical and expository.
Significance. If Theorem 1 is taken as a statement about the class of unbiased learners, it is correct and provides a clean quantitative reformulation of the classical UMVUE zero-correlation property: unbiasedness of a learner is exactly what permits the variance decomposition that yields inequality (14). The regression identity (8) is a useful sharp example, and the derivation is parameter-free and transparent. The paper is candid about its scope and invites extensions, which is commendable. However, the significance as a 'general uncertainty principle' for learning is limited by the fact that unbiasedness of both the learner and the assessor is load-bearing, and the motivating examples from cross-validation and S^2/n are not, in general, unbiased zero-mean assessors of the actual additive error. The paper is best viewed as a pedagogical and conceptual essay uniting classical ideas rather than a broadly applicable new bound.
major comments (3)
- [Section 5, Theorem 1, Eq. (14)] The unbiasedness of the learner is load-bearing, not a regularity condition. The proof of Eq. (14) uses E_s(delta_Qhat)=0 twice: it identifies R_s(Qhat) with V_s(delta_Qhat), and it ensures that Qhat_lambda = Qhat - lambda*delta_hat remains in the class Q. If E_s(delta_Qhat) is nonzero, the bias term survives in the risk decomposition and the bound can fail. For example, let X1,X2 be iid N(mu,1), take Qhat = 0.5*X1 and delta_hat = X1 - X2, which is unbiased for zero. For mu = sqrt(2), direct calculation gives rho_s^2 = 0.5 while RR_s(Qhat) = 1/3, violating (14). Thus the 'no free lunch' claim as stated in the abstract and Section 5 is not established for biased learners; the theorem should be explicitly framed as applying to the class of unbiased learners, and any claim about broader classes requires additional arguments.
- [Section 6, Theorem 2, proof of (18)] The displayed derivation does not justify the stated order of the remainder. From Ropt_s <= R_s(Qhat) [1 - rho_s^2] + O(e_iota^2), dividing by R_s(Qhat) gives rho_s^2 <= RR_s(Qhat) + O(e_iota^2)/R_s(Qhat). The paper omits this division and the needed assumption that R_s(Qhat) is bounded away from zero uniformly in s (or some alternative control of the remainder). If R_s(Qhat) tends to zero as e_iota does, the remainder is O(e_iota), not O(e_iota^2). The theorem as stated is therefore not proved. Additionally, the asymptotic assumption E_s(delta_hat)=O(e_iota) is not satisfied by the motivating examples such as S^2/n or typical cross-validation estimates, whose expectations are positive constants rather than vanishing quantities; Theorem 2 does not bridge the gap between Theorem 1 and those examples.
- [Sections 2, 4, and 10] The motivating examples cited in the abstract and Section 2 concern the independence between the squared actual error delta^2 and an estimator such as S^2/n, or between prediction errors and cross-validation estimates. These are estimators of expected loss or variance with positive mean, not zero-mean additive error assessors as required by Theorem 1. Theorem 1 concerns the additive error delta_Qhat = Qhat - Q and assessors satisfying E_s(delta_hat)=0. Consequently, the paper does not formally establish that inequality (14) explains the Bates et al. independence results, except for the special regression construction in Section 3 where the residual-based assessor has mean zero. The manuscript should explicitly delimit the theorem's applicability to these motivating examples or supply a separate result for squared errors and positive-mean assessors.
minor comments (4)
- [Abstract and Section 2] There are several typographical issues to correct, including 'Bo otstrap' in the abstract and the date 'Revised: 31 November, 2024', which is not a valid calendar date.
- [Section 8, Eqs. (28)-(29)] As printed, Eq. (29) appears to state Cov(hat_p, hat_x) = Cov(hat_x, hat_p), which contradicts Eq. (30). The intended relation is that the two mechanism-level covariances are complex conjugates of each other; this should be stated explicitly.
- [Section 4, Eq. (11)] The symbol gamma_sigma^2 in Eq. (11) is not formally defined; it should be introduced as the coefficient of variation of sigma^2, namely SD(sigma^2)/E(sigma^2), to make the formula self-contained.
- [Appendix C] The quasi-score analogy is interesting but somewhat disconnected from the main theorem; a sentence in Section 7 or 8 linking the symmetry failure in quasi-scores to the non-commutativity of the operator covariances would help the reader see why it is included.
Circularity Check
No significant circularity: the central inequality is derived from a self-contained variance decomposition; self-citations are contextual and non-load-bearing.
full rationale
Theorem 1's bound (14) is proved in Section 5 without importing the conclusion. The proof constructs Qhat_lambda = Qhat - lambda delta_hat_Qhat, notes that unbiasedness of both components keeps Qhat_lambda in the class (12), uses the existence of the optimal learner to write R_opt <= R(Qhat_lambda), and then minimizes the quadratic in lambda; the displayed algebra yields rho^2 <= RR directly. This is a first-principles variance/projection argument, not a fit, and it does not assume the inequality it establishes. The same decomposition drives Theorem 2, with the bias terms O(e_iota^2) entering explicitly. The UMVUE orthogonality result is cited to classical textbooks (Lehmann and Casella), not to any of the author's prior papers, and the author explicitly describes the contribution as 'a low-hanging fruit' and 'recasts a classical result regarding UMVUE,' which is a novelty disclaimer rather than a circularity. The paper's self-citations (Meng 2018, 2021, 2024; Liu and Meng 2014, 2016; Gong and Meng 2021; Berger et al. 2024) occur in illustrative, editorial, or contextual passages; none is load-bearing for Theorems 1-2. No fitted parameter is later renamed a prediction, and no uniqueness theorem from the author's own work is invoked to force the conclusion. The only mild concern is that the 'no free lunch' interpretation equates relevance with linear correlation and unbiasedness, but the paper states these restrictions explicitly in (12)-(13) and in Section 6, so the framing does not make the derivation circular. Overall the derivation chain is self-contained against external benchmarks, and no circular step can be exhibited.
Assumptions & free parameters
assumptions (3)
- domain assumption The class Q of unbiased learners is closed under subtracting a multiple of any unbiased error assessor.
- standard math All random quantities have finite second moments and the target Q has a well-defined distribution under each P_s.
- domain assumption An optimal learner Qhat_opt exists in Q with finite risk R_opt_s.
invented entities (1)
-
Mechanism-level co-variance for quantum operators (Cov(hat_x, hat_p))
Cite this review
Pith. "Pith review of A Heisenberg-esque Uncertainty Principle for Simultaneous (Machine) Learning and Error Assessment?." pith.science (2026). https://pith.science/paper/VPHYPADS
@misc{pith2026250101475,
author = {Pith},
title = {Pith review of: A Heisenberg-esque Uncertainty Principle for Simultaneous (Machine) Learning and Error Assessment?},
year = {2026},
howpublished = {\url{https://pith.science/paper/VPHYPADS}},
note = {Machine review of arXiv:2501.01475}
}
abstract
A highly cited and inspiring article by Bates et al (2024) demonstrates that the prediction errors estimated through cross-validation, Bootstrap or Mallow's $C_P$ can all be independent of the actual prediction errors. This essay hypothesizes that these occurrences signify a broader, Heisenberg-like uncertainty principle for learning: optimizing learning and assessing actual errors using the same data are fundamentally at odds. Only suboptimal learning preserves untapped information for actual error assessments, and vice versa, reinforcing the `no free lunch' principle. To substantiate this intuition, a Cramer-Rao-style lower bound is established under the squared loss, which shows that the relative regret in learning is bounded below by the square of the correlation between any unbiased error assessor and the actual learning error. Readers are invited to explore generalizations, develop variations, or even uncover genuine `free lunches.' The connection with the Heisenberg uncertainty principle is more than metaphorical, because both share an essence of the Cramer-Rao inequality: marginal variations cannot manifest individually to arbitrary degrees when their underlying co-variation is constrained, whether the co-variation is about individual states or their generating mechanisms, as in the quantum realm. A practical takeaway of such a learning principle is that it may be prudent to reserve some information specifically for error assessment rather than pursue full optimization in learning, particularly when intentional randomness is introduced to mitigate overfitting.
Reference graph
Works this paper leans on
-
[1]
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[2]
Abba, M. A., Williams, J. P., and Reich, B. J. (2024). A Bayesian shrinkage estimator for transfer learning. arXiv:2403.17321 ,
arXiv 2024
-
[3]
Bates, S., Hastie, T., and Tibshirani, R. (2024). Cross-validation: what does it estimate and how well does it do it? Journal of the American Statistical Association , 119 , 1434--1445
work page 2024
-
[4]
Berger, J., Meng, X.-L., Reid, N., and Xie, M.-g. (2024). Handbook of Bayesian, Fiducial, and Frequentist Inference . CRC Press
work page 2024
-
[5]
Blitzstein, J. K. and Hwang, J. (2014). Introduction to Probability . CRC Press, Boca Raton, FL, 1st edition
work page 2014
-
[6]
and Berger, R
Casella, G. and Berger, R. (2024). Statistical Inference . CRC Press
2024
-
[7]
Cash , W. (1979). Parameter estimation in astronomy through application of the likelihood ratio. The Astrophysical Journal , 228 , 939
work page 1979
-
[8]
Chapman, D. G. and Robbins, H. (1951). Minimum variance estimation without regularity assumptions. The Annals of Mathematical Statistics , 22 , 581--586
work page 1951
Show all 47 references
-
[9]
A., Bonamente, M., and Kashyap, V
Chen, Y., Li, X., Meng, X.-L., van Dyk, D. A., Bonamente, M., and Kashyap, V. (2024). Boosting C-statistics in astronomy via conditioning: More power, less computation. Technical report, Department of Statistics, University of Michigan
2024
-
[10]
Cox, D. R. (1980). Local ancillarity. Biometrika , 67 , 279--286
1980
-
[11]
V., Gong, R., and Meng, X.-L
Craiu, R. V., Gong, R., and Meng, X.-L. (2023). Six statistical senses. Annual Review of Statistics and Its Application , 10 , 699--725
2023
-
[12]
Dembo, A. (1990). Information inequalities and uncertainty principles. Department of Statistics, Stanford University., Stanford, CA, Technical Report , 75
1990
-
[13]
M., and Thomas, J
Dembo, A., Cover, T. M., and Thomas, J. A. (1991). Information inequalities and uncertainty principles. IEEE Transactions on Information Theory , 37 , 1501--1518
1991
-
[14]
Dempster, A. P. (1963). Further examples of inconsistencies in the fiducial argument. The Annals of Mathematical Statistics , 34 , 884--891
1963
-
[15]
Efron, B. (2020). Prediction, estimation, and attribution. International Statistical Review , 88 , S28--S59
2020
-
[16]
and Hinkley, D
Efron, B. and Hinkley, D. V. (1978). Assessing the accuracy of the maximum likelihood estimator: Observed versus expected fisher information. Biometrika , 65 , 457--483
1978
-
[17]
and Betancourt, M
Gelman, A. and Betancourt, M. (2013). Does quantum uncertainty have a place in everyday applied statistics. Behavioral and Brain Sciences , 36 , 285
2013
-
[18]
and Meng, X.-L
Gong, R. and Meng, X.-L. (2021). Judicious judgment meets unsettling updating: Dilation, sure loss, and simpson’s paradox. Statistical Science , 36 , 169--214. Discussion article with rejoinder
2021
-
[19]
Griffiths, D. J. and Schroeter, D. F. (2018). Introduction to Quantum Mechanics . Cambridge University Press
2018
-
[20]
Heisenberg, W. (1927). Über den anschaulichen inhalt der quantentheoretischen kinematik und mechanik. Zeitschrift für Physik , 43 , 172--198
1927
-
[21]
and Uffink, J
Hilgevoord, J. and Uffink, J. (2024). The uncertainty principle. In Zalta, E. N. and Nodelman, U., editors, The Stanford Encyclopedia of Philosophy . Stanford University. Spring 2024 edition
2024
-
[22]
F., Scully, M
Hillery, M., O'Connell, R. F., Scully, M. O., and Wigner, E. P. (1984). Distribution functions in physics: Fundamentals. Physics reports , 106 , 121--167
1984
-
[23]
ur Angewandte Mathematik der Universit\
Hoeffding, W. (1940). Ma tabinvariante K orrelatiostheorie. Schriften des Mathematischen Instituts und des Instituts f\"ur Angewandte Mathematik der Universit\" a t Berlin , 5 , 179--233
1940
-
[24]
Kennard, E. H. (1927). Zur quantenmechanik einfacher bewegungstypen. Zeitschrift für Physik , 44 , 326--352
1927
-
[25]
Landau, L. D. and Lifshitz, E. M. (2013). Quantum Mechanics: Non-relativistic Theory , volume 3. Elsevier
2013
-
[26]
Le Cam, L. (1956). On the asymptotic theory of estimation and testing hypotheses. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics , volume 3, pages 129--157. University of California Press
1956
-
[27]
Lehmann, E. L. and Casella, G. (2006). Theory of Point Estimation . Springer Science & Business Media
2006
-
[28]
Lin, H. (2024a). Bayesian epistemology. In Zalta, E. N. and Nodelman, U., editors, The Stanford Encyclopedia of Philosophy . 2024 Edition, originally published 2022
2024
-
[29]
Lin, H. (2024b). To be a Frequentist or Bayesian ? Five positions in a spectrum . Harvard Data Science Review , 6 . https://hdsr.mitpress.mit.edu/pub/axvcupj4
2024
-
[30]
and Meng, X.-L
Liu, K. and Meng, X.-L. (2014). Comment: A fruitful resolution to simpson’s paradox via multiresolution inference. The American Statistician , 68 , 17--29
2014
-
[31]
and Meng, X.-L
Liu, K. and Meng, X.-L. (2016). There is individualized treatment. Why not individualized inference? Annual Review of Statistics and Its Application , 3 , 79--111
2016
-
[32]
and Pasquini, B
Lorce, C. and Pasquini, B. (2011). Quark wigner distributions and orbital angular momentum. Physical Review D—Particles, Fields, Gravitation, and Cosmology , 84 , 014015
2011
-
[33]
Some statistical heresies
McCullagh, P. (1999). Discussion on Lindsey, J.K. (1999). "Some statistical heresies" . Journal of the Royal Statistical Society: Series D (The Statistician) , 48 , 34--35
1999
-
[34]
and Nelder, J
McCullagh, P. and Nelder, J. A. (1989). Generalized Linear Models , volume 37 of Monographs on Statistics and Applied Probability . Chapman & Hall/CRC, London, 2nd edition
1989
-
[35]
Meng, X.-L. (1994). Posterior predictive p -values. The annals of statistics , 22 , 1142--1160
1994
-
[36]
Meng, X.-L. (2018). Statistical paradises and paradoxes in big data (I) : law of large populations, big data paradox, and the 2016 us presidential election. The Annals of Applied Statistics , 12 , 685--726
2018
-
[37]
Meng, X.-L. (2021). Data science: A happy marriage of quantitative and qualitative thinking? Harvard Data Science Review , 3 . https://hdsr.mitpress.mit.edu/pub/pger71uh
2021
-
[38]
Meng, X.-L. (2024). A BFFer’s exploration with nuisance constructs: Bayesian p-value, H-likelihood, and Cauchyanity. In Handbook of Bayesian, Fiducial, and Frequentist Inference, Eds J. Berger, XL. Meng, N. Reid and M. Xie , pages 161--187. Chapman and Hall/CRC
2024
-
[39]
Rao, C. R. (1945). Information and the accuracy attainable in the estimation of statistical parameters. Bulletin of the Calcutta Mathematical Society , 37 , 81--91
1945
-
[40]
Rao, C. R. (1962). Efficient estimates and optimum inference procedures in large samples. Journal of the Royal Statistical Society: Series B (Methodological) , 24 , 46--63
1962
-
[41]
Severini, T. A. (1993). Local ancillarity in the presence of a nuisance parameter. Biometrika , 80 , 305--320
1993
-
[42]
Stam, A. J. (1959). Some inequalities satisfied by the quantities of information of fisher and shannon. Information and Control , 2 , 101--112
1959
-
[43]
K., Krafft, P., and Dreier, S
Tanweer, A., Gade, E. K., Krafft, P., and Dreier, S. (2021). Why the data revolution needs qualitative thinking. Harvard Data Science Review , 3 . https://hdsr.mitpress.mit.edu/pub/u9s6f22y
2021
-
[44]
and Fr \"o wis, F
T \'o th, G. and Fr \"o wis, F. (2022). Uncertainty relations with the variance and the quantum fisher information based on convex decompositions of density matrices. Physical Review Research , 4 , 013075
2022
-
[45]
and Petz, D
T \'o th, G. and Petz, D. (2013). Extremal properties of the variance and the quantum fisher information. Physical Review A—Atomic, Molecular, and Optical Physics , 87 , 032324
2013
-
[46]
Wald, A. (1943). Tests of statistical hypotheses concerning several parameters when the number of observations is large. Transactions of the American Mathematical society , 54 , 426--482
1943
-
[47]
Wang, Y. (2022). When quantum computation meets data science: Making data science quantum. Harvard Data Science Review , 4 . https://hdsr.mitpress.mit.edu/pub/kpn45eyx
2022
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.