REVIEW 2 major objections 4 minor 29 references
A Characterization of Mean Squared Error for Estimator with Bagging
T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Bagging the unbiased sample variance estimator reduces its mean squared error only when the data distribution has kurtosis above 3/2 and the number of bootstrap iterations is large enough; otherwise bagging makes the estimate worse.
desk verdict Genuinely new exact MSE formula for the bagged sample variance estimator and a clean kurtosis threshold; the proof has a fixable factor-of-two gap, but the math checks out on independent re-derivation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine is a symmetric-polynomial counting argument over the finite set of all $m$-subsamples drawn with replacement from the $n$ data points. The bagged estimator is an average over the sampling map $U$; its expectations with respect to the bagging random variable are symmetric polynomials in the squared differences $(X_i-X_j)^2$, so they are completely determined by three sums: $P$ of fourth powers, $Q$ of products of squared differences sharing one index, and $R$ of products over disjoint indices. The coefficients of these sums are obtained by differentiating the moment-generating function of a single bootstrap draw, which yields the exact bias and variance in Theorem 3.1. The $N$-dependence comes from the general variance decomposition $\mathrm{MSE}(\tilde\theta) = \frac{1}{N}E_L\mathrm{Var}_U(\hat\theta(L_U)) + \mathrm{Var}_L(E_U(\hat\theta(L_U))) + \mathrm{bias}^2$, which separates the within-bag spread from the between-bag spread.
What would settle it
Run a Monte Carlo comparison for a distribution with kurtosis below $3/2$, for example the paper's mixture $P(X=\pm1)=p/2$, $P(X=\pm\sqrt{a})=q/2$ with parameters chosen so $\kappa=1.4$, at $n=1000$, $m=n$, and any $N$. Theorem 3.3 predicts the bagged variance estimator's MSE exceeds the plain estimator's; a simulation showing the opposite would refute the central claim. A cheaper check on the derivation is to evaluate the supplement's equation (7) for $n=3,m=2$ by direct enumeration: the printed normalization gives an expectation off by a factor of two from Lemma 2.2's first item.
Extended reading notes
Core claim
The paper's central discovery is an exact, closed-form characterization of when bagging improves the unbiased sample variance estimator $\hat v(L) = \frac{1}{n-1}\sum_i (X_i-\bar X)^2$. For a random variable with finite fourth moment, Theorem 3.1 gives the full bias and variance of the bagged version $\tilde v(L,B)$ in terms of the second and fourth moments $\mu_2,\mu_4$ and the bagging parameters $m,N$. Comparing this with the classical MSE of the plain estimator, Theorem 3.3 states that as $n\to\infty$ the bagged estimator beats the plain one precisely when $-2\mu_4+3\mu_2^2<0$ (equivalently kurtosis $\kappa>3/2$) and $N>\frac{\mu_4-\mu_2^2}{2\mu_4-3\mu_2^2}\frac{n^2}{m}$. The folk intuition that bagging always helps is therefore false: for distributions with kurtosis at or below $3/2$—for instance a Rademacher distribution, where $\kappa=1$—bagging the variance estimator is provably counterproductive, and even in the favorable regime the gain is only $O(1/n^2)$.
Load-bearing premise
The theorem's exact MSE formula depends on counting each unordered index pair exactly once in the symmetric-polynomial expansion; as printed in the supplement (Lemma 2.2, equations (7) and (10)), one intermediate identity counts pairs twice, and the stated kurtosis threshold holds only after that normalization is corrected.
Editorial extensions
If this is right
- For any estimator, increasing the number of bootstrap iterations $N$ can only decrease or leave unchanged the bagged MSE; the limit as $N\to\infty$ is the $N$-independent floor $G$.
- For the sample variance, bagging is asymptotically beneficial only when kurtosis $\kappa>3/2$; for $\kappa\le 3/2$, bagging makes the estimator strictly worse.
- When the kurtosis condition holds, the required number of iterations is $N > \frac{\mu_4-\mu_2^2}{2\mu_4-3\mu_2^2}\frac{n^2}{m}$, which is at least about $n/2$ when $m=n$; common choices like $N\in[10,100]$ can fall below the threshold for large $n$.
- The asymptotic gain from bagging the variance estimator is $O(1/n^2)$, so bagging offers only a second-order correction as the sample size grows.
- The proposed algorithm estimates $\mu_2$ and $\mu_4$ from the data and applies bagging only when the estimated kurtosis condition holds, yielding a variance estimate with lower MSE at $O(Nn)$ computational cost.
Reading between the lines
- The same $F/N+G$ decomposition suggests a general diagnostic for when bagging helps any estimator: bagging pays off when the within-bag variance term $E_L\mathrm{Var}_U(\hat\theta)$ is large relative to the between-bag term, and kurtosis is one concrete proxy for that imbalance in variance estimation.
- An analogous moment computation should yield kurtosis-type thresholds for bagged estimators of higher moments (for example, a condition involving $\mu_6$ for a skewness estimator); that is a testable extension of Theorem 3.3.
- For heavy-tailed data, the $\kappa>3/2$ condition is typically met, so bagging the variance with $m\approx n$ and $N\gtrsim n/2$ is a cheap accuracy boost; for light-tailed data the same recipe is actively harmful, so the decision to bag should be data-dependent rather than automatic.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper analyzes the effect of bagging on the mean squared error (MSE) of statistical estimators, with a detailed study of the unbiased sample variance. It proves a general variance decomposition for bagged estimators (Theorem 2.1), derives exact finite-sample expressions for the bias and variance of the bagged sample variance in terms of the sample size n, the batch size m, and the number of bagging iterations N (Theorem 3.1), and then uses an asymptotic expansion to characterize when bagging improves the MSE of the variance estimator in terms of the kurtosis of the underlying distribution (Theorem 3.3). The paper also proposes an algorithm for higher-precision variance estimation and reports supporting experiments.
Significance. The exact finite-sample MSE formula for the bagged variance estimator is a useful and nontrivial contribution, and Theorem 2.1 gives a clean rigorous proof of the monotone decrease of MSE in N. The kurtosis criterion, if correct, is a crisp falsifiable statement with practical implications: for distributions with kurtosis below 3/2, bagging the variance estimator is provably detrimental. The derivation is self-contained and does not rely on fitted parameters or circular arguments. However, the asymptotic if-and-only-if statement in Theorem 3.3 is not correct as stated for fixed batch size m; the kurtosis half of the criterion survives, but the precise N-threshold needs either an additional term or an explicit m→∞ assumption. The experimental section provides qualitative support but does not fully exercise the threshold condition.
major comments (2)
- [Supplementary Lemma 2.2, Eqs. (3), (10), (11)] The proof uses the summation symbol \sum_{i\ne j} to mean summation over unordered distinct pairs. This is evident from the count in Lemma 2.1 item 4, where P is said to contain n(n−1)/2 terms, but the convention is never stated. If a reader interprets \sum_{i\ne j} as an ordered sum, Eq. (3) is off by a factor 1/2, Eq. (10) implies E_L E_U(\hat v) = 2(n−1)/n times the second moment, and the whole derivation appears internally inconsistent. The algebra is actually consistent under the unordered convention, and the final expressions in Theorem 3.1 pass independent counting checks, so the factor-of-two concern does not survive as a mathematical error. Nevertheless, the notation is a serious defect in the written proof: the convention must be made explicit, for example by writing \sum_{i<j} or by stating that all distinct-index sums are over unordered pairs.
- [Section 3.2, expansion before Theorem 3.3 and Eq. (4)] The simplification (1/N) E_L Var_U(\hat v(L_U)) = (1/(Nm))(\mu_4 − \mu_2^2) + o(1/(Nm)) is not valid for fixed batch size m. From Theorem 3.1, Eq. (2), as n→∞ with m fixed, (1/N) E_L Var_U tends to (1/(Nm))(\mu_4 − \mu_2^2 + 2\mu_2^2/(m−1)). The extra term is O(1/(Nm)) and cannot be absorbed in o(1/(Nm)) unless m also tends to infinity. Consequently the if-and-only-if threshold in Eq. (4) is missing the term 2\mu_2^2/(m−1) for fixed m; for example, when m=2 the required N is multiplied by roughly (\mu_4+\mu_2^2)/(\mu_4−\mu_2^2). The kurtosis criterion κ>3/2 survives, but the stated N-threshold and the asymptotic equivalence are not correct as written for fixed m. Please state explicitly whether m is assumed to grow with n (e.g., m=n) or give the corrected fixed-m threshold, and adjust Theorem 3.3 accordingly.
minor comments (4)
- [Theorem 3.1 and Algorithm 1] The case m=1 is excluded implicitly because the sample variance of a one-point bootstrap is undefined; please state explicitly that m≥2 throughout Section 3.
- [Section 2.3] The notation θ_y(Y,T) = E(T|Y=y) is confusing; θ_y is the population regression function and should be written θ_y = E(T|Y=y), with \hat θ_y(L) as its estimator.
- [References] References [7] and [10] appear to be two versions of the same work, and entries [10] and [11] list the same arXiv identifier 1812.08808; please correct the identifiers.
- [Section 4.1] The text says the code will be released on Github; for reproducibility, please indicate whether code is available with the submission or provide a supplementary archive.
Circularity Check
No significant circularity: the kurtosis criterion is derived algebraically from exact MSE moment calculations, with an external benchmark for the non-bagged MSE; no fitted parameter or load-bearing self-citation is present.
full rationale
The paper's derivation chain is self-contained and does not reduce any claimed prediction to its own inputs. Theorem 2.1, the general MSE decomposition for bagged estimators, is proved in the supplementary material by a counting/symmetry argument (expressing the second moment of the bagged estimator as a symmetric polynomial and evaluating the coefficients C1 and C2), not by assuming the conclusion. Theorem 3.1 then applies this framework to the unbiased sample variance estimator, computing the exact expectations E_L E_U(hat_v(L_U)), E_L(E_U(hat_v(L_U))^2), and E_L E_U(hat_v(L_U)^2) in Lemma 2.2 from moment identities in Lemma 2.1. The kurtosis threshold kappa > 3/2 in Theorem 3.3 follows algebraically by comparing the bagged MSE expression with the standard non-bagged variance estimator MSE taken from an external reference [18]. No parameter is fitted to a subset of data and then renamed as a prediction; no self-citation is load-bearing (the authors' prior work is not invoked to justify the central premise); and no ansatz is smuggled in via citation. The only material concern in the paper is a potential factor-of-two normalization inconsistency in the typeset supplementary proof of Lemma 2.2 (e.g., equation (7) writes the ordered-sum representation without the compensating 1/2 factor, and equation (10) as printed implies a factor of 2 in the first moment). That is a rigor/correctness issue about the printed derivation, not a circularity issue: the theorem is not assumed in the premises, and an independent count-based derivation reproduces the final formula. Because the central claim is derived from stated moment assumptions and an external benchmark rather than from its own conclusion, the circularity score is 0.
Assumptions & free parameters
assumptions (5)
- domain assumption X is a real random variable with finite fourth moment, and after centering E(X) = 0.
- domain assumption The bagging sampling functions U_i are independent and uniform over all ordered m-tuples with replacement, and expectations over L and B can be interchanged via Fubini.
- standard math Standard identities for moments of differences of i.i.d. variables, such as E[(Xi-Xj)^4] = 6*mu2^2 + 2*mu4, and the law of total variance.
- standard math The sample variance identity S^2 = 1/(2m(m-1)) * sum_{j != k} (X_j - X_k)^2 underlies the coefficient counts in Lemma 2.2.
- domain assumption The known MSE formula for the non-bagged sample variance estimator, Var(S^2) = (3-n)/(n(n-1)) * mu2^2 + (1/n) * mu4.
Cite this review
Pith. "Pith review of A Characterization of Mean Squared Error for Estimator with Bagging." pith.science (2026). https://pith.science/paper/7CZ3AWFY
@misc{pith2026190802718,
author = {Pith},
title = {Pith review of: A Characterization of Mean Squared Error for Estimator with Bagging},
year = {2026},
howpublished = {\url{https://pith.science/paper/7CZ3AWFY}},
note = {Machine review of arXiv:1908.02718}
}
abstract
Bagging can significantly improve the generalization performance of unstable machine learning algorithms such as trees or neural networks. Though bagging is now widely used in practice and many empirical studies have explored its behavior, we still know little about the theoretical properties of bagged predictions. In this paper, we theoretically investigate how the bagging method can reduce the Mean Squared Error (MSE) when applied on a statistical estimator. First, we prove that for any estimator, increasing the number of bagged estimators $N$ in the average can only reduce the MSE. This intuitive result, observed empirically and discussed in the literature, has not yet been rigorously proved. Second, we focus on the standard estimator of variance called unbiased sample variance and we develop an exact analytical expression of the MSE for this estimator with bagging. This allows us to rigorously discuss the number of iterations $N$ and the batch size $m$ of the bagging method. From this expression, we state that only if the kurtosis of the distribution is greater than $\frac{3}{2}$, the MSE of the variance estimator can be reduced with bagging. This result is important because it demonstrates that for distribution with low kurtosis, bagging can only deteriorate the performance of a statistical prediction. Finally, we propose a novel general-purpose algorithm to estimate with high precision the variance of a sample.
Figures
Reference graph
Works this paper leans on
-
[1]
Bagging predictors.Machine learning, 24(2):123–140, 1996
Leo Breiman. Bagging predictors.Machine learning, 24(2):123–140, 1996
1996
-
[2]
Geoffrey I Webb and Zijian Zheng. Multistrategy ensemble learning: Reduc- ing error by combining ensemble learning techniques.IEEE Transactions on Knowledge and Data Engineering, 16(8):980–991, 2004
work page 2004
-
[3]
A case study on bagging, boosting and basic ensembles of neural networks for ocr
Jianchang Mao. A case study on bagging, boosting and basic ensembles of neural networks for ocr. In1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No. 98CH36227), volume 3, pages 1828–1833. IEEE, 1998
work page 1998
-
[4]
Bagging equalizes influence.Machine Learning, 55(3):251– 270, 2004
Yves Grandvalet. Bagging equalizes influence.Machine Learning, 55(3):251– 270, 2004
work page 2004
-
[5]
Analyzing bagging.The Annals of Statistics, 30(4):927–961, 2002
Peter Bühlmann, Bin Yu, et al. Analyzing bagging.The Annals of Statistics, 30(4):927–961, 2002
work page 2002
-
[6]
On bagging and nonlinear estimation
Jerome H Friedman and Peter Hall. On bagging and nonlinear estimation. Journal of statistical planning and inference, 137(3):669–683, 2007
work page 2007
-
[7]
Smoothing Effects of Bagging: Von Mises Expansions of Bagged Statistical Functionals
Andreas Buja and Werner Stuetzle. Smoothing effects of bagging: Von mises expansions of bagged statistical functionals. arXiv preprint arXiv:1612.02528, 2016
work page Pith review arXiv 2016
-
[8]
Observations on bagging.Statistica Sinica, 16(2):323, 2006
Andreas Buja and Werner Stuetzle. Observations on bagging.Statistica Sinica, 16(2):323, 2006
work page 2006
Show all 29 references
-
[9]
Effects of bagging and bias correction on estimators defined by estimating equations.Statistica Sinica, 13:97–109, 2003
Songxi Chen and Peter Hall. Effects of bagging and bias correction on estimators defined by estimating equations.Statistica Sinica, 13:97–109, 2003
2003
-
[11]
Reducing sampling ratios and increasing number of estimates improve bagging in sparse regression.arXiv preprint arXiv:1812.08808, 2018
Luoluo Liu, Trac D Tran, et al. Reducing sampling ratios and increasing number of estimates improve bagging in sparse regression.arXiv preprint arXiv:1812.08808, 2018
2018 arXiv
-
[12]
The jackknife estimate of variance.The Annals of Statistics, pages 586–596, 1981
Bradley Efron and Charles Stein. The jackknife estimate of variance.The Annals of Statistics, pages 586–596, 1981
1981
-
[13]
Variance of sample variance with replacement
Eungchun Cho and Moon Jung Cho. Variance of sample variance with replacement. International Journal of Pure and Applied Mathematics, 52(1):43–47, 2009
2009
-
[14]
An experimental comparison of three methods for con- structing ensembles of decision trees: Bagging, boosting, and randomization
Thomas G Dietterich. An experimental comparison of three methods for con- structing ensembles of decision trees: Bagging, boosting, and randomization. Machine learning, 40(2):139–157, 2000. 13
2000
-
[15]
Bagging and boosting classification trees to predict churn.Journal of Marketing Research, 43(2):276–286, 2006
Aurélie Lemmens and Christophe Croux. Bagging and boosting classification trees to predict churn.Journal of Marketing Research, 43(2):276–286, 2006
2006
-
[16]
Additive groves of regression trees
Daria Sorokina, Rich Caruana, and Mirek Riedewald. Additive groves of regression trees. InEuropean Conference on Machine Learning, pages 323–334. Springer, 2007
2007
-
[17]
Bagging for linear classifiers
Marina Skurichina and Robert PW Duin. Bagging for linear classifiers. Pattern Recognition, 31(7):909–930, 1998
1998
-
[18]
Rose and M.D
C. Rose and M.D. Smith. Mathematical Statistics with Mathematica. Springer-Verlag, 2002
2002
-
[19]
Investigating the variation of ensemble size on bagging-based classifier performance in imbalanced bioinformatics datasets
Alireza Fazelpour, Taghi M Khoshgoftaar, David J Dittman, and Amri Naplitano. Investigating the variation of ensemble size on bagging-based classifier performance in imbalanced bioinformatics datasets. In2016 IEEE 17th International Conference on Information Reuse and Integrati...
2016
-
[20]
Scikit-learn: Machine learning in python
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. Journal of machine learning research, 12(Oct):2825–2830, 2011
2011
-
[21]
The numpy array: a structure for efficient numerical computation.Computing in Science & Engineering, 13(2):22, 2011
Stefan Van Der Walt, S Chris Colbert, and Gael Varoquaux. The numpy array: a structure for efficient numerical computation.Computing in Science & Engineering, 13(2):22, 2011
2011
-
[22]
SciPy: Open source scientific tools for Python, 2001–
Eric Jones, Travis Oliphant, Pearu Peterson, et al. SciPy: Open source scientific tools for Python, 2001–. [Online; accessed <today>]
2001
-
[23]
Numba: A llvm-based python jit compiler
Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert. Numba: A llvm-based python jit compiler. InProceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC, page 7. ACM, 2015. 14 Supplementary Material: A Characterization of Mean Squared Error for Estimator wit...
2015 arXiv
-
[24]
Simplifying equation (1) we have: C1 = N− 1 N n−2m + 1 Nn−m
= 1 N 2 ( EB(N ˜θ2)2 ) = 1 N 2 ( N (N− 1)n−2m +Nn−m) . Simplifying equation (1) we have: C1 = N− 1 N n−2m + 1 Nn−m. 2 Together with (2), we deduce : C2 = N− 1 N n−2m. Hence, EB(˜θ2) = N− 1 N n−2m ∑ i ˆθ(Lui )2 + ∑ i⁄=j ˆθ(Lui )ˆθ(Luj ) + 1 Nn−m∑ i ˆθ(Lui )2 = N− 1 N EU ...
-
[25]
E ( (Xi−Xj)4) = 6µ2 2 + 2µ4,
-
[26]
E ( (Xi−Xj)2(Xi−Xk)2) = 3µ2 2 +µ4,
-
[27]
Now denoteP =∑ i,j(Xi−Xj)4, Q =∑ i,j,k(Xi−Xj)2(Xi−Xk)2 and R =∑ i,j,k,l(Xi−Xj)2(Xk−Xl)2
E ( (Xi−Xj)2(Xk−Xl)2) = 4µ2 2, whereµ4 := E(X 4) is the forth moment ofX. Now denoteP =∑ i,j(Xi−Xj)4, Q =∑ i,j,k(Xi−Xj)2(Xi−Xk)2 and R =∑ i,j,k,l(Xi−Xj)2(Xk−Xl)2. Then the following equations hold:
-
[28]
EL(P ) = 3n(n− 1)µ2 2 +n(n− 1)µ4,
-
[29]
EL(Q) = 3 2n(n− 1)(n− 2)µ2 2 + 1 2n(n− 1)(n− 2)µ4,
-
[30]
EL(R) = 1 2n(n− 1)(n− 2)(n− 3)µ2 2. Proof. The proof of the three first items is immediate by independence of Xi,X j,X k,X l. Proof of item 4.P =∑ i,j(Xi−Xj)4 containsn(n− 1)/2 terms. Combining the item 1, the equation holds. Item 5 and item 6 can be deduce by similar arguments...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.