{"id":"5e3e8ca8-b03f-4942-98ff-8197f85551b0","arxiv_id":"2602.09405","paper_version":2,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"For any smooth prior in overparameterized Bayesian linear regression, training error of order J_πσ⁴ (memorization) is forced when σ² ≲ J_π^{-1}, and training error below σ² − V_π (harmful overfitting) is forbidden when σ² ≳ V_π.","lead":"This paper proves that in a Bayesian overparameterized linear model, whether the optimal predictor must memorize its training data or must avoid overfitting is controlled by two numbers computed from the prior: its Fisher information and its variance. The result reconciles two competing lines of work — 'memorize to generalize' and 'overfitting is harmful' — under a single threshold law.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified","rationale":"The reader's weakest_assumption points to the known-prior and positive-density conditions. I agree these are the most consequential limitations, but they are explicitly part of the model and acknowledged in the text; they do not undermine the theorem as stated. My independent check of the central proof chain found no internal inconsistency. The memorization-necessity argument (Corollary 2.1) uses only the upper bound in Theorem 1, and the overfitting-harmfulness argument (Corollary 2.2) uses only the lower bound; both bounds are rigorously derived via Stam's inequality and the Cramér-Rao bound. The high-noise expansion (4) is used for interpretation but is not load-bearing for the thresholds. The RMT computations in Section 3 and the Bakry-Émery identity in Appendix E are standard, and while I did not re-derive every step, they are not central to the paper's main claim. Thus I see no reason to alter the ACCEPT verdict.","tokens_in":32516,"tokens_out":28079,"duration_ms":234037,"concrete_test":"Numerically evaluate Train(σ²) for a heavy-tailed prior with finite variance (e.g., p(θ) ∝ 1/(1+θ⁴) in n=d=1) at large σ² and compare to σ² − V_π; if the difference does not vanish, Eq. (4) fails but Corollary 2.2 remains valid because it uses only the lower bound in Eq. (2).","verdict_should_be":"UNCHANGED","load_bearing_attack":"No significant objection identified. The central claim — that under the Bayesian model M_X(π,σ²) with π known, positive and differentiable, the training error of any near-optimal estimator is controlled by the two thresholds J_π^{-1} and V_π — is well supported by the arguments in Propositions 1, Theorems 1–2, and Corollaries 2.1–2.2. The known-prior and positive-density restrictions are explicitly stated scope conditions, not hidden flaws. The only notable technical gap is in the proof of the high-noise asymptotic (4): the dominated convergence step in Part III of Appendix B.2 is asserted too quickly, since the ratio defining E[θ̂_B θ̂_B^T] may not be dominated merely by ∫‖θ‖²π(dθ)<∞. However, this asymptotic is not needed for the threshold corollaries, which rely only on the non-asymptotic bounds in Eq. (2).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the Bayesian linear model y = Xθ + στ with d ≥ n and a fixed, known prior π on θ. It introduces two prior-dependent quantities, the Fisher information Jπ and the variance Vπ of the push-forward distribution of Xθ, and proves a non-asymptotic sandwich for the training error of the Bayes estimator: σ⁴/(Vπ+σ²) ≤ Train(θ̂_B) ≤ σ⁴/(Jπ^{-1}+σ²) (Theorem 1). Combining this with a Pythagorean decomposition for arbitrary estimators (Proposition 1), it derives two threshold laws: when σ² ≤ Jπ^{-1}, any near-optimal estimator must have training error at most O(Jπσ⁴) = o(σ²), so memorization is necessary (Corollary 2.1); when σ² ≥ Vπ, any near-optimal estimator must have training error at least σ² − Vπ, so overfitting is harmful (Corollary 2.2). The main theorem is non-asymptotic; Section 3 illustrates the thresholds in isotropic Gaussian, approximately low-rank Gaussian, and approximately sparse mixture priors using random matrix asymptotics. Appendices supply proofs and supporting lemmas.","tokens_in":32602,"tokens_out":25806,"duration_ms":228802,"significance":"If correct, the paper gives a clean conceptual unification: memorization and harmful overfitting are not contradictory phenomena but two regimes of a single threshold law set by the prior. The thresholds Jπ^{-1} and Vπ are derived, not fitted; the proof uses classical tools (Cramér–Rao, Stam's Fisher information inequality, lower semicontinuity of Fisher information) and the core bound holds for fixed d and n without asymptotic assumptions. The paper is explicit about its scope (known prior, positive differentiable density) and honestly flags the artifact that exact low-rank/sparse priors are excluded from the main theorem. Strengths include the non-asymptotic nature of Theorem 1, the concrete random-matrix examples, and the availability of reproducible simulation code. The contribution is likely to be of broad interest to the high-dimensional statistics and learning theory communities.","major_comments":[],"minor_comments":[{"comment":"The dominated-convergence step for E[θ̂_B θ̂_B^T] is asserted too quickly. The assertion that ∫∥θ∥²π(dθ)<∞ suffices is not by itself enough: the integrand is a ratio whose denominator can be small. A uniform lower bound on the denominator (e.g., using positivity of the density p and the pointwise limit e^{−∥τ∥²/2}) is needed to justify the interchange. This does not affect Corollaries 2.1–2.2, which rely only on the non-asymptotic bounds in Eq. (2), but the proof of Eq. (4) should be completed.","section":"Appendix B.2, Part III (high-noise asymptotic)"},{"comment":"The claim that J(σ²)=Train(σ²)/σ⁴ is non-increasing is said to follow from Stam's inequality J(t)^{-1} ≥ J(0)^{-1}+t. That inequality gives a decreasing upper bound on J(t), not monotonicity of J itself. The statement is true, but the correct justification is the derivative formula J'(t)=−E∥∇² log p_t∥²_F/n (Appendix E.2); the proof should be corrected.","section":"§4.2, proof of Proposition 6"},{"comment":"The variational characterization of Fisher information is abbreviated. The equality J(t)=sup_b{−2/n∫p_t∇·b − 1/n∫p_t∥b∥²} requires an approximation argument for the choice b=∇log p_t by compactly supported smooth vector fields. The sentence 'ϵ_b can be arbitrarily small' is not a full justification. This affects only the proof of the low-noise l.s.c. step, not the non-asymptotic sandwich.","section":"Appendix B.2, Lemma B.1"},{"comment":"Several inline logical symbols and square-root spans are garbled in the submitted PDF (e.g., 'Leftr⫯g⊸tl⫯ne⇒' in Section 2.1 and broken radicals in Theorem 2 and Appendix B.3). Please ensure the final typesetting is clean; this appears to be a rendering artifact rather than a mathematical issue.","section":"Throughout"}],"recommendation":"minor_revision","confidential_remarks":"I agree with the reader's assessment that the central argument is sound and the threshold corollaries are well supported. The issues I found are local proof-completeness problems (dominated convergence in Appendix B.2 Part III and the monotonicity justification in Proposition 6) that do not undermine the main claims. If the authors prefer, they could relegate the high-noise asymptotic (4) to a remark or add the missing domination argument. The paper fits the journal's scope and is likely to be impactful."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the take: this paper actually resolves the apparent contradiction, for Bayesian linear regression, between memorization-necessary results and overfitting-harmful results. The key is a two-parameter threshold (Jπ, Vπ) derived from the prior, and the non-asymptotic sandwich on the Bayes estimator's training error: σ⁴/(Vπ+σ²) ≤ Train ≤ σ⁴/(Jπ⁻¹+σ²). Corollaries 2.1 and 2.2 follow cleanly: below σ² ≤ Jπ⁻¹, near-optimal estimators must memorize; above σ² ≥ Vπ, overfitting is harmful. Theorem 1 is genuinely new and holds without asymptotic assumptions. The proof backbone—Pythagorean decomposition, Cramér-Rao, Stam's inequality, lower-semicontinuity of Fisher information—is solid. The Gaussian example recovers Cheng et al. as a special case, which is a sensible consistency check.\n\nThe paper is honest about its scope: the positive-density requirement means exact low-rank/sparse priors are excluded, and the small-noise memorization regime for structured priors is flagged in the text as an artifact. The known-prior Bayesian frame is stated as a condition, not hidden. Those are fine.\n\nSoft spots, in proportion: the high-noise asymptotic (4) has a small proof gap—the dominated convergence step for E[θ̂B θ̂B^T] is asserted too quickly, since the ratio may not be dominated by ∫‖θ‖²π alone. But the threshold corollaries don't rely on (4); they come from the non-asymptotic bounds, so this is a caveat, not a load-bearing flaw. The random matrix computations in the examples are not fully verified here, and the only experiment is not bit-reproducible. Some displayed equations in the provided text are corrupted—almost certainly OCR artifacts, not mathematical errors.\n\nOverall: the central claim holds, the limitations are acknowledged, and the paper deserves a serious referee. I'd bring it to a reading group and cite it.","headline":"A genuinely clarifying threshold result for Bayesian linear regression that reconciles the memorization-necessary and overfitting-harmful camps; the proof core holds up on inspection, with one minor gap in a side asymptotic.","tokens_in":33200,"tokens_out":1678,"would_cite":true,"duration_ms":16313,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","62J07","62C10"],"pacs":[],"model":"deepseek-v4-flash","headline":"In an overparameterized Bayesian linear model, the prior's Fisher information and variance set exact thresholds: below 1/J_pi optimal estimators must memorize, above V_pi they must avoid overfitting.","keywords":["memorization","overfitting","Bayesian linear regression","Fisher information","training error","generalization","overparameterization","prior variance"],"falsifier":"Fix any prior with positive differentiable density and any full-row-rank design X; compute Train(theta_hat_B) for many noise levels and check whether sigma^4/(V_pi + sigma^2) <= Train(theta_hat_B) <= sigma^4/(J_pi^{-1} + sigma^2) for all sigma^2. A single violation, or a failure of Train(theta_hat_B)/sigma^4 to tend to J_pi as sigma^2 goes to 0, would refute Theorem 1. Alternatively, in the regime sigma^2 <= J_pi^{-1}, find a near-optimal estimator whose excess risk is small but whose training error is not o(sigma^2).","tokens_in":32287,"feed_emoji":"📊","tokens_out":4811,"duration_ms":41289,"temperature":0.7,"pith_summary":"The paper asks when a near-optimal predictor in overparameterized linear regression must memorize the training data and when memorizing is actively harmful. Its answer is that both behaviors are forced, but in different noise regimes determined entirely by the prior pi. The paper proves a sandwich bound on the Bayes estimator's training error: sigma^4/(V_pi + sigma^2) <= Train(theta_hat_B) <= sigma^4/(J_pi^{-1} + sigma^2), where J_pi is the Fisher information of the prior and V_pi its variance. Consequently, when sigma^2 <= J_pi^{-1}, any estimator with near-optimal prediction error must train near zero in relative terms, and when sigma^2 >= V_pi, any such estimator must keep training error close to sigma^2. The two apparently conflicting intuitions about overfitting are therefore not in tension; the prior's local spikiness and global spread decide which regime applies.","feed_headline":"Two prior numbers set the overfitting threshold","feed_subtitle":"In overparameterized regression, optimal estimators must memorize below 1/J_pi and must not overfit above V_pi.","key_machinery":"The load-bearing object is the Fisher information of the noised pushed-forward distribution, J(sigma^2) = (1/n) E_{y ~ pi' * N(0, sigma^2 I)}[||nabla log p_sigma^2(y)||^2], related to the Bayes training error exactly by Train(theta_hat_B) = sigma^4 * J(sigma^2) through Tweedie's formula. The proof combines Stam's Fisher information inequality, which yields J(sigma^2) <= (J_pi^{-1} + sigma^2)^{-1}, and a Cramer-Rao/variance bound, which yields J(sigma^2) >= (V_pi + sigma^2)^{-1} after a Cauchy-Schwarz integration-by-parts identity, plus lower-semicontinuity of Fisher information to control the sigma^2 -> 0 limit. This identity converts training error into a purely information-theoretic quanti","core_discovery":"Treating y = X*theta + sigma*tau with theta drawn from a prior pi that has a positive differentiable density, the discovery is that optimal generalization is tied to training error through the two prior parameters J_pi = E[||nabla log p(X*theta)||^2/n | X] and V_pi = E[||X*theta||^2/n | X]. Theorem 1 sandwiches the Bayes estimator's mean squared training error between sigma^4/(V_pi + sigma^2) and sigma^4/(J_pi^{-1} + sigma^2). The asymptotic directions show Train(theta_hat_B) = J_pi*sigma^4 + o(sigma^4) as sigma^2 -> 0, and Train(theta_hat_B) = sigma^2 - V_pi + o(1) as sigma^2 -> infinity. Since the posterior mean is the unique prediction-optimal estimator (Proposition 1), any near-optimal e","pith_inferences":["If these thresholds are robust, a practitioner could estimate J_pi and V_pi from the prior or from score statistics of the posterior and decide whether to regularize or interpolate before seeing data; the paper does not develop this diagnostic.","The Bayesian benchmark assumes the true prior is known. For a misspecified prior, the same sandwich may bound the training error of the misspecified-Bayes estimator against the wrong optimum, so the necessity and harm statements would not transfer unchanged; this is an extension question the paper leaves open.","Because a positive density is required, the memorization regime at extremely small noise can be an artifact of local flatness even under strongly structured priors; replacing the density condition with atom-containing or exactly low-rank priors may remove that regime and change the practical advice.","The same Fisher-information identity suggests a finite-sample check: for a fixed design and prior, compute Train(theta_hat_B) = sigma^4 * J(sigma^2) numerically for several noise levels and compare against the sandwich; a sizable violation would indicate the core theorem is not capturing the true training error."],"forward_implications":["If the sandwich holds, an uninformative prior with J_pi and V_pi of order 1 puts any positive-but-small noise in the memorization regime: near-optimal estimators must have training error o(sigma^2).","If the prior is concentrated near a low-dimensional manifold, J_pi^{-1} is tiny, so for fixed noise memorization is not forced; overfitting becomes harmful once sigma^2 >= V_pi.","For any C > 1, an estimator with Train >= C*J_pi*sigma^4 in the low-noise regime suffers excess prediction risk linear in its training error; an estimator with Train <= sigma^2 - C*V_pi in the high-noise regime suffers excess risk quadratic in the overfit amount divided by sigma^2.","The Bayes-optimal training error increases monotonically with noise, while Train/sigma^4 decreases, so the transition from memorization-necessary to overfitting-harmful is a continuous shift rather than a sharp phase flip.","Exact low-rank priors without a density sit outside the main theorem; direct computation gives Train(theta_hat_B) -> sigma^2*(1-rho) + O(sigma^4), so bounded overfitting is harmless there."],"fun_headline_variants":["Prior's Fisher info and variance decide when to memorize","Memorization necessary below J_pi, harmful above V_pi","Bayes optimum memorizes below J_pi not above V_pi","Prior's two stats set memorization and overfitting thresholds"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Everything is measured against the Bayes estimator under the true prior pi, and pi is required to have a positive, differentiable density; if the practitioner's prior is misspecified or the prior puts mass on a lower-dimensional set, the two thresholds describe the wrong benchmark and the memorization/overfitting verdicts need not hold.","fun_headline_variants_meta":{"raw":{"variants":["Prior's Fisher info and variance decide when to memorize","Memorization necessary below J_pi, harmful above V_pi","Bayes optimum memorizes below J_pi not above V_pi","Prior's two stats set memorization and overfitting thresholds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001131,"raw_usage":{"total_tokens":4505,"prompt_tokens":682,"completion_tokens":3823,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":3751}},"tokens_in":426,"tokens_out":3823,"duration_ms":24394,"temperature":1.0,"reasoning_tokens":3751,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T02:50:05.589079+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix any prior with positive differentiable density and any full-row-rank design X; compute Train(theta_hat_B) for many noise levels and check whether sigma^4/(V_pi + sigma^2) <= Train(theta_hat_B) <= sigma^4/(J_pi^{-1} + sigma^2) for all sigma^2. A single violation, or a failure of Train(theta_hat_B)/sigma^4 to tend to J_pi as sigma^2 goes to 0, would refute Theorem 1. Alternatively, in the regime sigma^2 <= J_pi^{-1}, find a near-optimal estimator whose excess risk is small but whose training error is not o(sigma^2).","supporting_citations":[],"review_version":1}