REVIEW 1 major objections 4 minor 47 references
Hybrid least squares for learning functions from highly noisy data
T0 review · 1 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A hybrid least-squares algorithm that combines Christoffel sampling with variance-aware allocation provably reduces the sample budget needed to learn functions from heavily noisy data.
desk verdict Solid new hybrid least-squares method; the main bound is correct, but the practical gain rests on a variance-estimation step that is only guaranteed under subgaussian assumptions away from zero variance. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Christoffel function $\Phi_n(x)=\sum_{i=1}^n v_i(x)^2$ on an orthonormal basis of the approximation space $V_n$, which defines the induced sampling measure $\nu(dx)=\Phi_n(x)\,d\mu/n$ used to draw the $m$ sample points. On top of this, the allocation vector $$p^*_{n,i}=\frac{w(x_i)\$\sigma$(x_i)\sqrt{\Phi_n(x_i)}}{\sum_{j\in[m]} w(x_j)\$\sigma$(x_j)\sqrt{\Phi_n(x_j)}}$$ assigns replicate samples proportionally to the local noise-variance contribution. The analysis is carried by the variance functional $G(p)=\frac{1}{L}\sum_{i=1}^m \frac{w^2(x_i)\sigma^2(x_i)\Phi_n(x_i)}{m^2p_i}$: $p^*_n$ minimizes $G(p)$, and the closed-form expectation of the minimized value produces the sample-complexity formula.
What would settle it
On a problem with known heteroscedastic noise, compare the empirical MSE of HLS-1 with standard Christoffel-sampled least squares at a fixed total budget $L$ with $L/n$ large: if the MSE ratio does not approach $\left\|\sigma\sqrt{\Phi_n/n}\right\|^2_{L^1_\mu}/\left\|\sigma\sqrt{\Phi_n/n}\right\|^2_{L^2_\mu}$ or does not stay at or below one, the asymptotic efficiency claim in (5.5)--(5.6) is contradicted.
Extended reading notes
Core claim
The paper's central result, Theorem 5.1, is that for the non-reweighted allocation $p^*_n$, the conditional expected squared error satisfies $$\mathbb{E}_{X,y}\big[\|\hat f-f\|^2_{$L^{2}$_\mu}\mid A\big] \lesssim \mathrm{OPT} + \mathbb{E}_X[G(p^*_n)],$$ where $A$ is the event that the weighted design matrix has spectrum in $[0.9,1.1]$ and $$\mathbb{E}_X[G(p^*_n)] = \frac{n}{L}\left[\frac{1}{m}\|\$\sigma$\|^2_{$L^{2}$_\mu} + \left(1-\frac{1}{m}\right)\left\|\$\sigma$\sqrt{\Phi_n/n}\right\|^2_{$L^{1}$_\mu}\right].$$ Because $\|\sigma\sqrt{\Phi_n/n}\|^2_{L^1_\mu}/\|\sigma\sqrt{\Phi_n/n}\|^2_{L^2_\mu}\le 1$, the implied total budget for error $\eta\ge\mathrm{OPT}$ is asymptotically at least as efficient as the standard bound $L \sim n\max\{\log n, \|\sigma\|^2_{L^2_\nu}/\eta\}$. The paper also treats a reweighted allocation $q^*_n$ based on A-optimal design, shows that projecting onto a closed convex set preserves the error bounds up to a factor of two, and proves approximation-capacity results for random subspaces built from evaluations of the random field defining $f$.
Load-bearing premise
The guarantees require that the conditional variance $\sigma^2(x)$ be known or estimated accurately enough to set the allocations; if the pilot variance estimates are badly off, or if $\sigma$ vanishes at sampled points, the computed allocations are miscalibrated and the stated sample-complexity improvement is not assured.
Editorial extensions
If this is right
- For a target error $\eta$, the hybrid method needs $L \sim n\log n + \frac{n}{\eta}\left(\frac{1}{m}\|\sigma\|^2_{L^2_\mu} + (1-1/m)\left\|\sigma\sqrt{\Phi_n/n}\right\|^2_{L^1_\mu}\right)$ total evaluations, matching the noiseless $n\log n$ count when noise is small.
- Compared with standard Christoffel-sampled least squares, the large-noise regime is improved: the variance factor is asymptotically no larger than the standard $\|\sigma\|^2_{L^2_\nu}$ factor, so the method is never worse and can be better.
- The reweighted allocation $q^*_n$ can reduce variance further when the approximation space $V_n$ is highly expressive, at the cost of amplifying the oracle bias by a factor that depends on the regularization parameter $\delta$.
- Convexity constraints, such as preserving positivity of a financial pricer, can be enforced by post-projection with the same theoretical guarantees up to a constant factor.
- When $f$ is the expectation of a random field, random subspaces spanned by field realizations give provable approximation capacity, including a bound that depends on tail decay of the covariance eigenvalues.
Reading between the lines
- The allocation $p^*_n$ is essentially a variance-proportional design, so the same two-stage idea could be applied to other estimators where replicate sampling at selected points is cheaper than drawing new design points; the paper does not explore such generalizations.
- The efficiency gain is largest when the noise variance is concentrated at points where $\sigma(x)\sqrt{\Phi_n(x)}$ is large relative to the sampling density; mapping this condition on the domain could lead to a priori rules for when hybrid sampling helps.
- For near-zero variance regions, the paper's pilot estimator can return zero variance and the guarantees are not covered; a shrinkage or regularized variance estimate could fill this gap and is a natural testable extension.
- The random-subspace analysis assumes subgaussian Karhunen-Lo\`eve coefficients; a heavy-tailed generalization would broaden the applicability to more realistic stochastic processes and would be a direct follow-up.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a hybrid least-squares method for approximating a function f from noisy point evaluations y(x) = f(x) + ε(x). The method first draws m points from the Christoffel sampling measure associated with an n-dimensional approximation space V_n, then allocates a total budget L of noisy evaluations among these points according to a Neyman allocation (non-reweighted) or an A-optimal design (reweighted) that accounts for the conditional variance σ²(x). The main theoretical result, Theorem 5.1, gives conditional error bounds showing that the total evaluation budget L required to achieve error η is approximately n[log n + (noise factor)/η], where the noise factor is no larger than the corresponding factor in the standard Christoffel-sampled least squares bound (Theorem 1.1). The paper also extends the results to convexity-constrained approximation and to adaptively chosen random subspaces when f is the expectation of a random field, and it provides numerical experiments on a synthetic polynomial example and on basket option pricing in a Black–Scholes model.
Significance. The paper makes a worthwhile contribution to noisy function approximation. The two-stage idea of using Christoffel sampling for the design and experimental-design-based allocation for repeated evaluations is natural and is analyzed carefully. The proofs are detailed and largely self-contained: Lemma 4.1 rigorously identifies the cond²-approximate optimal allocation, Theorem 5.1 provides explicit error bounds with absolute constants, and the pilot variance estimation analysis in Appendix B.3 gives a concrete guarantee under a subgaussian assumption. The comparison with Theorem 1.1 is quantified through the ratio in (5.6), which is ≤ 1 by Jensen's inequality, and the numerical experiments support the theory. The main weakness is the reliance on known σ²(x) for the central theorem and the restricted conditions under which variance estimation is analyzed; the authors are transparent about this limitation, but it should be more prominently qualified in the main text.
major comments (1)
- [Appendix B.3, Theorem B.1] The practical version of Algorithm 5.1 estimates σ²(x) from Rm pilot samples, and the only rigorous guarantee for this step is Theorem B.1, which requires the relative-error condition (B.9) for every i. This condition is derived under a uniform subgaussian assumption on ε(x)/σ(x) with R ≳ log m/κ². As the authors note at the end of Appendix B.3, when this assumption is violated (e.g., at points where σ(x) is very small and the noise is heavy-tailed), the sample variance can be zero or badly scaled and there is no rigorous analysis. Because the optimal allocations p*_n and q*_n are computed from the estimated variances, the sample-complexity improvement in (5.5) is not guaranteed in this regime. The abstract and Section 1.2 state the improvement without this caveat. I recommend either supplying a rigorous treatment of the small-variance case (for example, using a regularized variance estimator with a floor) or explicitly qualifying the claims in the main text so that the scope of the theoretical guarantees is clear.
minor comments (4)
- [Section 7.1] The sentence 'For HLS-1 and HLS-2. For each x, we estimate σ(x) from R = 50 MC simulations offline' contains an extra period after 'HLS-2'; it should read 'For HLS-1 and HLS-2, for each x, we estimate ...'.
- [Theorem 6.3] The bound '24√rτr+1' is ambiguous; it should be typeset as '24√(r τ_{r+1})' to clarify that the square root applies to the product r·τ_{r+1}.
- [Proof of Theorem 6.3] The constants c' and C1 are not explicitly related; the authors should state that C1 is chosen large enough (e.g., C1 ≥ 2c') so that the constructed function f_n lies in the sparse set \bar{V}_{n,k}, as the proof relies on this implicitly.
- [Section 5.1, Equation (5.5)] The total evaluation complexity L in (5.5) is stated for the case of known σ; a sentence noting that the pilot variance estimation cost Rm is additional (as quantified in Appendix B.3) would improve clarity, since the complexity comparison with Theorem 1.1 otherwise omits this overhead.
Circularity Check
No significant circularity: the hybrid bound is derived from an explicit MSE-variance proxy and an external Christoffel embedding result, not assumed.
full rationale
The central derivation is self-contained. Lemma 4.1 derives two-sided bounds on the conditional MSE of the non-reweighted estimator in terms of the explicit quantity G(p), then identifies p*n as the minimizer of G(p); Theorem 5.1 subsequently combines this with the external subspace-embedding result [11, Theorem 2] and evaluates EX[G(p*n)] by direct quadrature. The error bound is not defined to be G(p*n); it is an upper bound obtained through a genuine inequality chain. Similarly, the reweighted analysis in Lemma 4.5 and Theorem 5.1 uses a bias-variance decomposition and an operator-convexity argument, not the conclusion. The known-σ assumption and the pilot-estimation gap in Appendix B.3 are explicitly acknowledged limitations that affect robustness, but they are not circular: Theorem 5.1 is stated for known σ, and Theorem B.1 gives a perturbation bound showing near-optimality of estimated allocations under a stated relative-error condition. Self-citations ([1], [4], [19], [29]) appear only in background and implementation details; the load-bearing embedding inequality is cited to the independent prior work [11]. No equation reduces by definition to its own input.
Assumptions & free parameters
free parameters (1)
- regularization parameter delta =
0.01/m in experiments
assumptions (4)
- domain assumption Noise epsilon(x) satisfies E[epsilon(x)|x]=0 with finite variance sigma^2(x)>0, and evaluations at different points and replicates are independent.
- domain assumption Conditional variance sigma^2(x) is known or accurately estimated via pilot samples; normalized noise epsilon(x)/sigma(x) is uniformly subgaussian for the pilot guarantee.
- standard math The Christoffel subspace embedding theorem of Cohen and Migliorati (ref [11]) holds with m of order n log n, giving the event A with P(A)>1-n^{-2}.
- domain assumption For the random subspace results, the covariance kernel of the random field is continuous, the Karhunen-Loeve expansion exists, and the KL coefficients are non-atomic and uniformly subgaussian.
Cite this review
Pith. "Pith review of Hybrid least squares for learning functions from highly noisy data." pith.science (2026). https://pith.science/paper/52EERXSI
@misc{pith2026250702215,
author = {Pith},
title = {Pith review of: Hybrid least squares for learning functions from highly noisy data},
year = {2026},
howpublished = {\url{https://pith.science/paper/52EERXSI}},
note = {Machine review of arXiv:2507.02215}
}
read the original abstract
Motivated by the need for efficient estimation of conditional expectations, we consider a least-squares function approximation problem with heavily polluted data. Existing methods that are effective in the small-noise regime are suboptimal when large noise is present. To address this issue, we propose a hybrid approach that combines Christoffel sampling with optimal experimental design. We show that the proposed algorithm enjoys appropriate optimality properties for both sample point generation and noise mollification, leading to improved computational efficiency and sample complexity compared to existing methods. We also extend the algorithm to convexity-constrained settings with similar theoretical guarantees. When the target function is defined as the expectation of a random field, we further extend our approach to leverage adaptive random subspaces and establish results on the approximation capacity of the adaptive procedure. Our theoretical findings are supported by numerical studies on both synthetic data and on a more challenging stochastic simulation problem in computational finance.
Figures
Reference graph
Works this paper leans on
-
[1]
B. Adcock , Optimal sampling for least-squares approximation , Foundations of Computational Mathe- matics, (2025), pp. 1–60
work page 2025
-
[2]
Compressed sensing with sparse corruptions: Fault-tolerant sparse collocation approximations
B. Adcock, A. Bao, J. D. Jakeman, and A. Narayan, Compressed Sensing with Sparse Corruptions: Fault-Tolerant Sparse Collocation Approximations , SIAM/ASA Journal on Uncertainty Quantifica- 22 B. ADCOCK AND B. HIENTZSCH AND A. NARAYAN AND Y. XU tion, 6 (2018), pp. 1424–1453, https://arxiv.org/abs/1703.00135
work page Pith review arXiv 2018
- [3]
-
[4]
B. Adcock and J. M. Cardenas , Near-optimal sampling strategies for multivariate function approxi- mation on general domains , SIAM Journal on Mathematics of Data Science, 2 (2020), pp. 607–630
work page 2020
-
[5]
A. Alla and J. N. Kutz, Randomized model order reduction, Advances in Computational Mathematics, 45 (2019), pp. 1251–1271
work page 2019
- [6]
-
[7]
F. Bach, On the equivalence between kernel quadrature rules and random feature expansions , Journal of machine learning research, 18 (2017), pp. 1–38
work page 2017
-
[8]
J. Bendat and S. Sherman , Monotone and convex operator functions , Transactions of the American Mathematical Society, 79 (1955), pp. 58–71
work page 1955
Show all 47 references
-
[9]
Borwein and A
J. Borwein and A. Lewis , Convex Analysis, Springer, 2006
2006
-
[10]
Cohen, M
A. Cohen, M. A. Davenport, and D. Leviatan, On the stability and accuracy of least squares approx- imations, Foundations of Computational Mathematics, 13 (2013), pp. 819–834
2013
-
[11]
Cohen and G
A. Cohen and G. Migliorati , Optimal weighted least-squares methods , The SMAI Journal of Compu- tational Mathematics, 3 (2017), pp. 181–203
2017
-
[12]
M. B. Cohen, Y. T. Lee, C. Musco, C. Musco, R. Peng, and A. Sidford , Uniform sampling for matrix approximation, in Proceedings of the 2015 conference on innovations in theoretical computer science, 2015, pp. 181–190
2015
-
[13]
M. B. Cohen and R. Peng, Lp row sampling by lewis weights , in Proceedings of the forty-seventh annual ACM symposium on Theory of computing, 2015, pp. 183–192
2015
-
[14]
Glasserman, Monte Carlo methods in financial engineering , vol
P. Glasserman, Monte Carlo methods in financial engineering , vol. 53, Springer, 2004
2004
-
[15]
L. Guo, A. Narayan, L. Yan, and T. Zhou , Weighted approximate fekete points: sampling for least- squares polynomial approximation, SIAM Journal on Scientific Computing, 40 (2018), pp. A366–A387
2018
-
[16]
L. Guo, A. Narayan, and T. Zhou , Constructing least-squares polynomial approximations , SIAM Review, 62 (2020), pp. 483–508
2020
-
[17]
Haberstich, A
C. Haberstich, A. Nouy, and G. Perrin , Boosted optimal weighted least-squares , Mathematics of Computation, 91 (2022), pp. 1281–1315
2022
-
[18]
Hadigol and A
M. Hadigol and A. Doostan , Least squares polynomial chaos expansion: A review of sampling strate- gies, Computer Methods in Applied Mechanics and Engineering, 332 (2018), pp. 382–407
2018
-
[19]
Herremans and B
A. Herremans and B. Adcock , Refinement-based christoffel sampling for least squares approximation in non-orthogonal bases , arXiv preprint arXiv:2510.08461, (2025)
2025
-
[20]
Huge and A
B. Huge and A. Savine , Differential machine learning , arXiv preprint arXiv:2005.02347, (2020)
2020 arXiv
-
[21]
R. A. Johnson and D. W. Wichern , Applied multivariate statistical analysis , (2020)
2020
-
[22]
Lewis, Finite dimensional subspaces of lp, Studia Mathematica, 63 (1978), pp
D. Lewis, Finite dimensional subspaces of lp, Studia Mathematica, 63 (1978), pp. 207–212
1978
-
[23]
Li, Compressed Sensing and Matrix Completion with Constant Proportion of Corruptions, Constructive Approximation, 37 (2012), pp
X. Li, Compressed Sensing and Matrix Completion with Constant Proportion of Corruptions, Constructive Approximation, 37 (2012), pp. 73–99
2012
-
[24]
O. A. Malik and S. Becker , A sampling-based method for tensor ring decomposition , in International Conference on Machine Learning, PMLR, 2021, pp. 7400–7411
2021
-
[25]
O. A. Malik, Y. Xu, N. Cheng, S. Becker, A. Doostan, and A. Narayan , Fast algorithms for monotone lower subsets of kronecker least squares problems , arXiv preprint arXiv:2209.05662, (2022)
2022 arXiv
-
[26]
Martinsson and J
P.-G. Martinsson and J. A. Tropp, Randomized numerical linear algebra: Foundations and algorithms, Acta Numerica, 29 (2020), pp. 403–572
2020
-
[27]
Matsuda and Y
T. Matsuda and Y. Nakatsukasa , Polynomial approximation of noisy functions: T. matsuda, y. nakatsukasa, Numerische Mathematik, 157 (2025), pp. 1285–1311
2025
-
[28]
Murray, J
R. Murray, J. Demmel, M. W. Mahoney, N. B. Erichson, M. Melnichenko, O. A. Malik, L. Grigori, P. Luszczek, M. Derezi ´nski, M. E. Lopes, et al. , Randomized numerical linear algebra: A perspective on the field with an eye to software , arXiv preprint arXiv:2302.11474, (2023)
2023 arXiv
-
[29]
Narayan, J
A. Narayan, J. Jakeman, and T. Zhou , A christoffel function weighted least squares algorithm for collocation approximations, Mathematics of Computation, 86 (2017), pp. 1913–1947
2017
-
[30]
N. H. Nelsen and A. M. Stuart , The random feature model for input-output maps between banach HYBRID LEAST SQUARES 23 spaces, SIAM Journal on Scientific Computing, 43 (2021), pp. A3212–A3243
2021
-
[31]
Nevai, G´ eza freud, orthogonal polynomials and christoffel functions
P. Nevai, G´ eza freud, orthogonal polynomials and christoffel functions. a case study, Journal of Approx- imation Theory, 48 (1986), pp. 3–167
1986
-
[32]
Niederreiter, Random number generation and quasi-Monte Carlo methods , SIAM, 1992
H. Niederreiter, Random number generation and quasi-Monte Carlo methods , SIAM, 1992
1992
-
[33]
Olivares, A
P. Olivares, A. Alvarez, et al. , Pricing basket options by polynomial approximations , Journal of Applied Mathematics, 2016 (2016)
2016
-
[34]
Paszke, S
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer , Automatic differentiation in pytorch , (2017)
2017
-
[35]
Peherstorfer, Breaking the kolmogorov barrier with nonlinear model reduction, Notices of the Amer- ican Mathematical Society, 69 (2022), pp
B. Peherstorfer, Breaking the kolmogorov barrier with nonlinear model reduction, Notices of the Amer- ican Mathematical Society, 69 (2022), pp. 725–733
2022
-
[36]
A. K. Polala and B. Hientzsch , Parametric differential machine learning for pricing and calibration , arXiv preprint arXiv:2302.06682, (2023)
2023 arXiv
-
[37]
Pukelsheim, Optimal design of experiments , SIAM, 2006
F. Pukelsheim, Optimal design of experiments , SIAM, 2006
2006
-
[38]
Rahimi and B
A. Rahimi and B. Recht , Uniform approximation of functions with random bases , in 2008 46th annual allerton conference on communication, control, and computing, IEEE, 2008, pp. 555–561
2008
-
[39]
Reiss and M
M. Reiss and M. W ahl, Nonasymptotic upper bounds for the reconstruction error of pca , The Annals of Statistics, 48 (2020), pp. 1098–1123
2020
-
[40]
Shin and D
Y. Shin and D. Xiu, Correcting Data Corruption Errors for Multivariate Function Approximation, SIAM Journal on Scientific Computing, 38 (2016), pp. A2492–A2511
2016
-
[41]
J. A. Tropp , User-friendly tail bounds for sums of random matrices , Foundations of computational mathematics, 12 (2012), pp. 389–434
2012
-
[42]
V apnik, Principles of risk minimization for learning theory , Advances in Neural Information Process- ing Systems, 4 (1991)
V. V apnik, Principles of risk minimization for learning theory , Advances in Neural Information Process- ing Systems, 4 (1991)
1991
-
[43]
C. K. Williams and C. E. Rasmussen , Gaussian processes for machine learning , vol. 2, MIT press Cambridge, MA, 2006
2006
-
[44]
D. P. Woodruff et al. , Sketching as a tool for numerical linear algebra , Foundations and Trends® in Theoretical Computer Science, 10 (2014), pp. 1–157
2014
-
[45]
Xiu, Numerical methods for stochastic computations: a spectral method approach, Princeton University Press, 2010
D. Xiu, Numerical methods for stochastic computations: a spectral method approach, Princeton University Press, 2010
2010
-
[46]
Xu and A
Y. Xu and A. Narayan, Randomized weakly admissible meshes , Journal of Approximation Theory, 285 (2023), p. 105835
2023
-
[47]
V. Zala, M. Kirby, and A. Narayan , Structure-preserving function approximation via convex opti- mization, SIAM Journal on Scientific Computing, 42 (2020), pp. A3006–A3029. Appendix A. Optimal allocation. A.1. Proof of Lemma 4.1.By a direct computation, Ey[∥ bα(p) − ¯α∥2 2 | X...
2020
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.