{"id":"ba521cff-7669-40d3-9725-b40e5da9088d","arxiv_id":"1908.02718","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper derives an exact MSE formula for the bagged sample variance estimator and shows bagging beats the non-bagged estimator only when the distribution kurtosis exceeds 3/2 and the number of bagging iterations is large enough.","lead":"Bagging always lowers the mean squared error of an estimator as the number of averaged bootstrap estimates grows, but for the standard sample variance it only helps when the underlying distribution has kurtosis above 3/2. The paper proves this with an exact formula and gives an algorithm for higher-precision variance estimation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The printed proof of the central MSE formula mixes factor-of-two normalizations in the pair-difference representation, so Theorem 3.1 is not rigorously established as written; an independent count-based derivation indicates the final formula and kurtosis threshold are nonetheless correct.","rationale":"The reader's weakest-assumption identification is exactly the factor-of-two normalization inconsistency in the supplementary derivation of Lemma 2.2. I agree this is the most load-bearing point: every coefficient in Theorem 3.1, and hence the kurtosis criterion in Theorem 3.3, depends on the symmetric-polynomial moment counts. I checked whether the inconsistency actually invalidates the theorem. Using the correct normalization, the bagged sample variance on a bootstrap sample is hat_v(L_U) = (1/[2m(m-1)]) * sum_{a != b}(X_{U(a)}-X_{U(b)})^2. In terms of multinomial counts n_i this equals (1/[m(m-1)]) * sum_{r 3/2 survives the normalization check; the concern is a real rigor gap in the exposition rather than a mathematical error in the headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes the effect of bagging on the mean squared error (MSE) of statistical estimators, with a detailed study of the unbiased sample variance. It proves a general variance decomposition for bagged estimators (Theorem 2.1), derives exact finite-sample expressions for the bias and variance of the bagged sample variance in terms of the sample size n, the batch size m, and the number of bagging iterations N (Theorem 3.1), and then uses an asymptotic expansion to characterize when bagging improves the MSE of the variance estimator in terms of the kurtosis of the underlying distribution (Theorem 3.3). The paper also proposes an algorithm for higher-precision variance estimation and reports supporting experiments.","tokens_in":13357,"tokens_out":22528,"duration_ms":214726,"significance":"The exact finite-sample MSE formula for the bagged variance estimator is a useful and nontrivial contribution, and Theorem 2.1 gives a clean rigorous proof of the monotone decrease of MSE in N. The kurtosis criterion, if correct, is a crisp falsifiable statement with practical implications: for distributions with kurtosis below 3/2, bagging the variance estimator is provably detrimental. The derivation is self-contained and does not rely on fitted parameters or circular arguments. However, the asymptotic if-and-only-if statement in Theorem 3.3 is not correct as stated for fixed batch size m; the kurtosis half of the criterion survives, but the precise N-threshold needs either an additional term or an explicit m→∞ assumption. The experimental section provides qualitative support but does not fully exercise the threshold condition.","major_comments":[{"comment":"The proof uses the summation symbol \\sum_{i\\ne j} to mean summation over unordered distinct pairs. This is evident from the count in Lemma 2.1 item 4, where P is said to contain n(n−1)/2 terms, but the convention is never stated. If a reader interprets \\sum_{i\\ne j} as an ordered sum, Eq. (3) is off by a factor 1/2, Eq. (10) implies E_L E_U(\\hat v) = 2(n−1)/n times the second moment, and the whole derivation appears internally inconsistent. The algebra is actually consistent under the unordered convention, and the final expressions in Theorem 3.1 pass independent counting checks, so the factor-of-two concern does not survive as a mathematical error. Nevertheless, the notation is a serious defect in the written proof: the convention must be made explicit, for example by writing \\sum_{i3/2 survives, but the stated N-threshold and the asymptotic equivalence are not correct as written for fixed m. Please state explicitly whether m is assumed to grow with n (e.g., m=n) or give the corrected fixed-m threshold, and adjust Theorem 3.3 accordingly.","section":"Section 3.2, expansion before Theorem 3.3 and Eq. (4)"}],"minor_comments":[{"comment":"The case m=1 is excluded implicitly because the sample variance of a one-point bootstrap is undefined; please state explicitly that m≥2 throughout Section 3.","section":"Theorem 3.1 and Algorithm 1"},{"comment":"The notation θ_y(Y,T) = E(T|Y=y) is confusing; θ_y is the population regression function and should be written θ_y = E(T|Y=y), with \\hat θ_y(L) as its estimator.","section":"Section 2.3"},{"comment":"References [7] and [10] appear to be two versions of the same work, and entries [10] and [11] list the same arXiv identifier 1812.08808; please correct the identifiers.","section":"References"},{"comment":"The text says the code will be released on Github; for reproducibility, please indicate whether code is available with the submission or provide a supplementary archive.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is the exact finite-sample MSE analysis, which is sound. The asymptotic characterization in Theorem 3.3 is the headline result, but it needs a nontrivial correction unless m is assumed to grow with n. This is fixable within the manuscript's scope, so I recommend major revision rather than rejection. The paper might also strengthen its positioning by acknowledging that Theorem 2.1 is a standard conditional-variance decomposition; the proof is clean but the novelty claim is somewhat overstated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague —\n\nThe short version: this paper contains a genuinely new result about when bagging hurts variance estimation. Theorem 3.3 — the kurtosis condition (κ > 3/2) and the iteration threshold for the bagged unbiased sample variance estimator — is the real contribution. An independent derivation using the correct pair-difference representation reproduces their exact MSE formula and the threshold, so the math appears sound. But the supplementary proof of Lemma 2.2 has a factor-of-two normalization error, so Theorem 3.1 is not rigorously established as written.\n\nWhat is new: the exact finite-n MSE expression for the bagged variance estimator and the simple kurtosis criterion. That criterion is genuinely useful for choosing bagging parameters and for understanding that bagging can hurt for low-kurtosis distributions. It goes beyond the cited literature. Theorem 2.1, the general decomposition into F/N + G, is a straightforward law-of-total-variance application; it is correct but elementary, and the authors' claim that this 'has not yet been rigorously proved' is an overstatement.\n\nSoft spots, in order. First, the normalization inconsistency: equation (7) writes E_U(hat_v) with 1/(m(m-1)) rather than 1/(2m(m-1)) for the unordered pair sum, and equation (10) as printed implies E_L E_U(hat_v) = 2(n-1)/n μ2, contradicting their own equation (4). The final P,Q,R expansion implicitly uses the corrected normalization. This is a genuine derivation gap, though not a wrong theorem; a correct counting of terms reproduces equations (5), (6), and Theorem 3.1 exactly. The proof needs to be fixed before publication. Second, the experiments are qualitative, lack error bars, and the code is promised but not released. That's minor in comparison.\n\nBottom line: worth a serious referee. The central claim survives the correction; the paper needs a revision that cleans up the supplementary proof and tones down the novelty claims about Theorem 2.1. I'd cite it once the proof is in order.","headline":"Genuinely new exact MSE formula for the bagged sample variance estimator and a clean kurtosis threshold; the proof has a fixable factor-of-two gap, but the math checks out on independent re-derivation.","tokens_in":13925,"tokens_out":2503,"would_cite":true,"duration_ms":24438,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F10","62G09"],"pacs":[],"model":"deepseek-v4-flash","headline":"Bagging the unbiased sample variance estimator reduces its mean squared error only when the data distribution has kurtosis above 3/2 and the number of bootstrap iterations is large enough; otherwise bagging makes the estimate worse.","keywords":["bagging","mean squared error","sample variance","kurtosis","bootstrap aggregation","ensemble methods","unbiased variance estimator","symmetric polynomials"],"falsifier":"Run a Monte Carlo comparison for a distribution with kurtosis below $3/2$, for example the paper's mixture $P(X=\\pm1)=p/2$, $P(X=\\pm\\sqrt{a})=q/2$ with parameters chosen so $\\kappa=1.4$, at $n=1000$, $m=n$, and any $N$. Theorem 3.3 predicts the bagged variance estimator's MSE exceeds the plain estimator's; a simulation showing the opposite would refute the central claim. A cheaper check on the derivation is to evaluate the supplement's equation (7) for $n=3,m=2$ by direct enumeration: the printed normalization gives an expectation off by a factor of two from Lemma 2.2's first item.","tokens_in":12818,"feed_emoji":"📊","tokens_out":12593,"duration_ms":117466,"temperature":0.7,"pith_summary":"Bagging—averaging an estimator over many bootstrap subsamples—is widely assumed to reduce error, but the paper shows this assumption fails for variance estimation on low-kurtosis data. It first proves a general decomposition: the mean squared error of any bagged estimator is $F/N + G$, with $F$ and $G$ independent of $N$, so extra iterations always reduce or leave unchanged the MSE. Specializing to the unbiased sample variance estimator, it derives an exact formula for the bagged MSE and concludes that, as the sample size grows, bagging helps if and only if the distribution's kurtosis exceeds $3/2$ and the number of iterations satisfies $N > \\frac{\\mu_4-\\mu_2^2}{2\\mu_4-3\\mu_2^2}\\frac{n^2}{m}$. When those conditions hold, the improvement over the plain estimator decays as $O(1/n^2)$, and the paper closes with an algorithm that estimates the kurtosis condition and selects $N$ accordingly.","feed_headline":"Bagging helps variance estimation only when kurtosis exceeds 3/2","feed_subtitle":"An exact MSE formula shows when averaging bootstrap estimates helps, when it hurts, and how many iterations are needed.","key_machinery":"The engine is a symmetric-polynomial counting argument over the finite set of all $m$-subsamples drawn with replacement from the $n$ data points. The bagged estimator is an average over the sampling map $U$; its expectations with respect to the bagging random variable are symmetric polynomials in the squared differences $(X_i-X_j)^2$, so they are completely determined by three sums: $P$ of fourth powers, $Q$ of products of squared differences sharing one index, and $R$ of products over disjoint indices. The coefficients of these sums are obtained by differentiating the moment-generating function of a single bootstrap draw, which yields the exact bias and variance in Theorem 3.1. The $N$-dependence comes from the general variance decomposition $\\mathrm{MSE}(\\tilde\\theta) = \\frac{1}{N}E_L\\mathrm{Var}_U(\\hat\\theta(L_U)) + \\mathrm{Var}_L(E_U(\\hat\\theta(L_U))) + \\mathrm{bias}^2$, which separates the within-bag spread from the between-bag spread.","core_discovery":"The paper's central discovery is an exact, closed-form characterization of when bagging improves the unbiased sample variance estimator $\\hat v(L) = \\frac{1}{n-1}\\sum_i (X_i-\\bar X)^2$. For a random variable with finite fourth moment, Theorem 3.1 gives the full bias and variance of the bagged version $\\tilde v(L,B)$ in terms of the second and fourth moments $\\mu_2,\\mu_4$ and the bagging parameters $m,N$. Comparing this with the classical MSE of the plain estimator, Theorem 3.3 states that as $n\\to\\infty$ the bagged estimator beats the plain one precisely when $-2\\mu_4+3\\mu_2^2<0$ (equivalently kurtosis $\\kappa>3/2$) and $N>\\frac{\\mu_4-\\mu_2^2}{2\\mu_4-3\\mu_2^2}\\frac{n^2}{m}$. The folk intuition that bagging always helps is therefore false: for distributions with kurtosis at or below $3/2$—for instance a Rademacher distribution, where $\\kappa=1$—bagging the variance estimator is provably counterproductive, and even in the favorable regime the gain is only $O(1/n^2)$.","pith_inferences":["The same $F/N+G$ decomposition suggests a general diagnostic for when bagging helps any estimator: bagging pays off when the within-bag variance term $E_L\\mathrm{Var}_U(\\hat\\theta)$ is large relative to the between-bag term, and kurtosis is one concrete proxy for that imbalance in variance estimation.","An analogous moment computation should yield kurtosis-type thresholds for bagged estimators of higher moments (for example, a condition involving $\\mu_6$ for a skewness estimator); that is a testable extension of Theorem 3.3.","For heavy-tailed data, the $\\kappa>3/2$ condition is typically met, so bagging the variance with $m\\approx n$ and $N\\gtrsim n/2$ is a cheap accuracy boost; for light-tailed data the same recipe is actively harmful, so the decision to bag should be data-dependent rather than automatic."],"forward_implications":["For any estimator, increasing the number of bootstrap iterations $N$ can only decrease or leave unchanged the bagged MSE; the limit as $N\\to\\infty$ is the $N$-independent floor $G$.","For the sample variance, bagging is asymptotically beneficial only when kurtosis $\\kappa>3/2$; for $\\kappa\\le 3/2$, bagging makes the estimator strictly worse.","When the kurtosis condition holds, the required number of iterations is $N > \\frac{\\mu_4-\\mu_2^2}{2\\mu_4-3\\mu_2^2}\\frac{n^2}{m}$, which is at least about $n/2$ when $m=n$; common choices like $N\\in[10,100]$ can fall below the threshold for large $n$.","The asymptotic gain from bagging the variance estimator is $O(1/n^2)$, so bagging offers only a second-order correction as the sample size grows.","The proposed algorithm estimates $\\mu_2$ and $\\mu_4$ from the data and applies bagging only when the estimated kurtosis condition holds, yielding a variance estimate with lower MSE at $O(Nn)$ computational cost."],"supporting_citations":[{"why":"Introduces bagging and the instability heuristic; supplies the estimator form and the empirical phenomenon that Theorem 2.1 formalizes.","marker":"[1]"},{"why":"Documents empirical cases in which bagging deteriorates performance; the paper's kurtosis criterion provides a theoretical instance of such deterioration.","marker":"[4]"},{"why":"Supplies the classical MSE formula for the plain unbiased sample variance estimator used as the baseline in Proposition 3.2 and Theorem 3.3.","marker":"[18]"}],"fun_headline_variants":["Bagging variance: only helps if kurtosis exceeds 3/2","Exact MSE formula reveals when bagging beats plain variance","Bagging can hurt variance estimation for low-kurtosis data","Sample variance with bagging: kurtosis > 3/2 required for gain","Bootstrap averaging: can worsen variance estimates if kurtosis low"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The theorem's exact MSE formula depends on counting each unordered index pair exactly once in the symmetric-polynomial expansion; as printed in the supplement (Lemma 2.2, equations (7) and (10)), one intermediate identity counts pairs twice, and the stated kurtosis threshold holds only after that normalization is corrected.","fun_headline_variants_meta":{"raw":{"variants":["Bagging variance: only helps if kurtosis exceeds 3/2","Exact MSE formula reveals when bagging beats plain variance","Bagging can hurt variance estimation for low-kurtosis data","Sample variance with bagging: kurtosis > 3/2 required for gain","Bootstrap averaging: can worsen variance estimates if kurtosis low"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00103,"raw_usage":{"total_tokens":4397,"prompt_tokens":1062,"completion_tokens":3335,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":678,"completion_tokens_details":{"reasoning_tokens":3241}},"tokens_in":678,"tokens_out":3335,"duration_ms":23568,"temperature":1.0,"reasoning_tokens":3241,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:39:19.364412+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a Monte Carlo comparison for a distribution with kurtosis below $3/2$, for example the paper's mixture $P(X=\\pm1)=p/2$, $P(X=\\pm\\sqrt{a})=q/2$ with parameters chosen so $\\kappa=1.4$, at $n=1000$, $m=n$, and any $N$. Theorem 3.3 predicts the bagged variance estimator's MSE exceeds the plain estimator's; a simulation showing the opposite would refute the central claim. A cheaper check on the derivation is to evaluate the supplement's equation (7) for $n=3,m=2$ by direct enumeration: the printed normalization gives an expectation off by a factor of two from Lemma 2.2's first item.","supporting_citations":[{"cited_title":"Bagging equalizes influence.Machine Learning, 55(3):251– 270, 2004","cited_arxiv_id":null,"evidence_quote":"Documents empirical cases in which bagging deteriorates performance; the paper's kurtosis criterion provides a theoretical instance of such deterioration."},{"cited_title":"Rose and M.D","cited_arxiv_id":null,"evidence_quote":"Supplies the classical MSE formula for the plain unbiased sample variance estimator used as the baseline in Proposition 3.2 and Theorem 3.3."}],"review_version":1}