{"id":"5d6f509b-7ca7-4a4d-9f3c-9a6f40db75c2","arxiv_id":"2412.20355","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Residual-based conditional variance estimation with dense ReLU networks achieves non-asymptotic rates under sub-Exponential noise, and a bootstrap interval for the conditional mean is proven to have coverage at least 1-alpha.","lead":"This paper derives non-asymptotic error bounds for estimating conditional mean and variance with dense ReLU networks, and constructs a bootstrap confidence interval for the regression function with a coverage guarantee. A generalist might read it to see whether deep learning prediction intervals can be given statistical guarantees beyond sub-Gaussian noise.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 5's coverage proof conditions on a common random event Ω1 and then treats the bootstrap draws as conditionally independent; that variance decomposition is unjustified, so the coverage guarantee is not established as written.","rationale":"I read the paper in good faith: the variance estimation framework in Corollary 1 is plausible and follows the standard residual-based route, with the rate depending on Theorem 2 and the Kohler-Langer sup-norm approximation assertion. The reader's weakest-assumption concern about Theorem 2 is a reasonable thing to verify, but it is conditional on the cited theorem not being sup-norm; in fact Kohler-Langer (2019) does provide sup-norm approximation results, so that may not be a real defect. The more definite, internal problem is in the proof of Theorem 5, where the coverage guarantee is derived by conditioning on a common random event Ω1 and then using conditional independence of the bootstrap replicas. Because a(α) itself depends on g_hat^{(B+1)} and the validation data, Ω1 couples the supposedly independent draws, so the variance decomposition in Step 2 of Appendix E.1 is not justified. This is a genuine gap in the central claim of guaranteed coverage. It may be repairable by replacing the conditional argument with an unconditional high-probability concentration bound or by defining Ω1 with a deterministic threshold, and the empirical results are encouraging, so a conditional verdict remains appropriate rather than outright rejection. The verdict is therefore unchanged from the reader's CONDITIONAL, but the specific justification should shift from the external approximation theorem to this internal proof gap and the accompanying formal typo in Theorem 5.","tokens_in":44106,"tokens_out":15822,"duration_ms":157828,"concrete_test":"Analytically check the contested variance identity: take two independent bootstrap mean estimates Z1, Z2 and a random threshold A depending on Z1 (as a(α) depends on g_hat^{(B+1)}), define Ω = {Z1 ≤ A, Z2 ≤ A}, and compute Var(Z1 + Z2 | X, Ω, D_{1:3}). If this is not equal to Var(Z1 | X, Ω, D_{1:3}) + Var(Z2 | X, Ω, D_{1:3}), the Step 2 decomposition fails. Then redo the concentration step unconditionally on Ω1; if the resulting bound contains extra dependence terms or requires a substantially larger t, Theorem 5's proof needs a new argument.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central coverage claim rests on Theorem 5 (Appendix E.1). In Step 2 of that proof, Ω1 is defined as the event that ||f_hat^{(j)}_{A_n} - f*||^2_{L2} ≤ a(α) for every j = B+1, ..., B+eB, and then the proof writes Var(Σ_j Z_j | X, Ω1, D_{1:3}) = Σ_j Var(Z_j | X, Ω1, D_{1:3}), claiming the Z_j are independent conditional on Ω1. This is not valid. The threshold a(α) in (14) is a data-dependent quantity that depends on g_hat^{(B+1)}_{A_n} and on the validation set I4; hence Ω1 is not a product event in the independent bootstrap draws. Conditioning on the common event Ω1 induces dependence among the f_hat^{(j)} even when they are unconditionally independent, so the variance decomposition used to obtain the α/4 concentration bound does not follow. The later bound P(Ω1^c) < 0.58α does not repair this step, because the concentration bound itself is applied conditionally on Ω1. Consequently, the proof of Theorem 5 and of Corollary 3 does not currently establish the advertised coverage guarantee. (A separate formal issue is the impossible hypothesis 'An > max{An, Bn}' in Theorem 5, which should be corrected, but the conditional-independence gap is the more substantive problem.)","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies nonparametric regression with dense ReLU networks under heteroscedastic noise. It proposes a residual-based estimator of the conditional variance function g*, derives nonasymptotic L2 error bounds for the conditional mean (Theorem 2) and for the conditional variance in heteroscedastic and homoscedastic settings (Theorems 3–4 and Corollaries 1–2), and then constructs a bootstrap confidence interval for the conditional mean f*(X) with a claimed finite-sample coverage guarantee (Theorem 5 and Corollary 3). The experimental section compares the proposed ReLU variance estimator with random forests and MARS on simulated and real data, and evaluates the coverage and length of the proposed confidence intervals.","tokens_in":44447,"tokens_out":6908,"duration_ms":71859,"significance":"If the results are correct, the paper would be a useful contribution to the theory of neural-network-based uncertainty quantification: it extends dense ReLU mean-estimation rates to sub-Exponential noise, gives the first nonasymptotic conditional-variance rates for ReLU networks under hierarchical composition assumptions, and proposes a bootstrap procedure with a theoretical coverage guarantee. The residual-based variance framework and the explicit nonasymptotic rates in Theorems 1–4 are coherent and follow standard empirical-process and approximation arguments. However, the central coverage claim for the confidence interval is not established as written: Theorem 5 contains an impossible hypothesis, and the proof uses a conditional-independence step that is invalid because it conditions on a common data-dependent event. In addition, the computable version of the interval in Corollary 3 rests on a strong high-probability condition whose derivation from the earlier rates is not supplied. These issues are load-bearing for the paper's headline confidence-interval contribution, so the manuscript needs substantial revision.","major_comments":[{"comment":"The hypothesis 'An > max{An, Bn}' is impossible, since the right-hand side contains An itself. This appears both in Section 2.2 ('we will assume that An is large enough such that An > max{An, Bn}') and in the statement of Theorem 5. Because this is an assumption of the main coverage theorem, Theorem 5 is vacuous as stated. The authors presumably intend a lower bound involving Bn and some norm of f*, but the condition must be corrected and the proof re-checked under the corrected statement.","section":"Section 2.2 and Theorem 5"},{"comment":"The proof conditions on the common event Ω1 = {‖fhat_An^{(j)} - f*‖_{L2}^2 ≤ a(α) for j = B+1,...,B+eB} and then writes Var(Σ_j Z_j | X, Ω1, D_{1:3}) = Σ_j Var(Z_j | X, Ω1, D_{1:3}), claiming the bootstrap fits are conditionally independent given X, Ω1, and D_{1:3}. This is not valid: the bootstrap fits are conditionally independent given the data D_{1:3} and X, but Ω1 is an intersection of events involving all j jointly, and the threshold a(α) is itself data-dependent through the validation set I4 and through ĝ^{(B+1)}. Conditioning on a common, non-product event destroys conditional independence, so the variance decomposition used to obtain the α/4 concentration bound does not follow. The subsequent bound P(Ω1^c) < 0.58α does not repair this step because the concentration inequality is applied conditionally on Ω1. Consequently, the proof of Theorem 5 and Corollary 3 does not currently establish the advertised coverage guarantee (35).","section":"Appendix E.1, Step 2"},{"comment":"The condition n^{q+1} P(max{‖g* - ĝ_An^{(B+1)}‖_{L2}^2, ‖f* - fhat_An^{(B+1)}‖_{L2}^2} > ε_n) → 0 with ε_n ≲ α^2/log^{2s} n is assumed rather than derived. The paper's earlier results, Theorem 2 and Corollary 1, provide only o_P rates of convergence; they do not imply the much stronger tail bound with polynomial factor n^{q+1}. Since Corollary 3 is what converts the oracle quantities a0 and b(α) in Theorem 5 into the computable choices used for the 'NN' method in Section 4.2, the coverage guarantee for the implemented procedure is not established. The authors should either prove (38) from the network approximation and empirical-process bounds or clearly state it as an additional unverified assumption and adjust the claims accordingly.","section":"Corollary 3, condition (38)"},{"comment":"The proof relies on Theorem 3 of Kohler and Langer (2019) to assert the existence of a network fbar in F(L,ν) with uniform approximation error ‖fbar - f*‖∞ ≤ √ϕ_n. The manuscript does not state the hypotheses of that theorem or verify that the hierarchical composition class H(l1,P1) satisfies them. If the cited theorem only establishes approximation in L2(P) rather than in sup-norm, then the rate r_n in (25) does not follow, and the downstream variance rates in Corollary 1 inherit the gap. The authors should provide the precise statement of the external result and confirm that it yields the sup-norm approximation used here.","section":"Appendix E, proof of Theorem 2"}],"minor_comments":[{"comment":"The sentence 'so that ϕn is the approximating error' in the statement of Theorem 3 should refer to ψn, since the approximation is of g*.","section":"Theorem 3"},{"comment":"In the statement of Theorem 4, 'Bno(nb)' is a typo; it should read Bn = o(n^b).","section":"Theorem 4"},{"comment":"Equation (34) has a mismatched parenthesis: the expression should be 32[E((fhat_An^{(B+1)}(X) - Y)^2) - E(g*(X))]^{1/2} / (5α(1 - 0.58α)), not 32[E((fhat_An^{(B+1)}(X) - Y)^2 - E(g*(X))]^{1/2}.","section":"Theorem 5, equation (34)"},{"comment":"The definition of a(α) in (14) uses a nested absolute value and parentheses that are difficult to parse; the authors should clarify the expression, especially which terms are inside the outer absolute value.","section":"Section 2.2, Step 5"},{"comment":"The claim that this is 'the first result for variance estimation using ReLU networks' should be qualified with respect to the specific nonasymptotic, dense-network setting considered here, since the prior literature includes various neural-network variance estimators without such theoretical guarantees.","section":"Abstract and Section 1.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the residual-based variance estimation theory is the real contribution: Corollary 1 gives the first non-asymptotic L2 rates for conditional variance estimation with dense ReLU networks, under sub-Exponential noise, and the general theorem (Theorem 3) is a clean adaptation of Fan-Yao to this setting. Second, the confidence interval result, Theorem 5, has a genuine proof gap that the authors would need to fix before the coverage claim is credible.\n\nThe mean estimation extension (Theorem 2) relaxes sub-Gaussian to sub-Exponential noise and follows from standard empirical process machinery; the stated rate r_n looks plausible given the Kohler-Langer approximation result. I would cite this part. The variance estimator's rate in Corollary 1 is a legitimate first.\n\nNow the soft spots. The stress-test is correct. In the proof of Theorem 5 (Appendix E.1, Step 2), Ω1 is defined as the event that every bootstrap estimate f_hat^(j) is within a(α) of f* in L2. The threshold a(α) is random: it depends on the first B bootstrap fits and on the validation set I4. So Ω1 is not a product event in the independent bootstrap draws. Conditioning on Ω1 couples the draws, and the line Var(Σ_j Z_j | X, Ω1, D_{1:3}) = Σ_j Var(Z_j | X, Ω1, D_{1:3}) is not justified. The later bound P(Ω1^c) < 0.58α does not fix this, because the concentration step is applied conditionally on Ω1. The coverage proof fails at this point.\n\nThere's also the impossible assumption An > max{An, Bn} in Theorem 5 and Section 2.2. That's clearly a typo, but it's the kind of error that makes you want to recheck every display in that section. The reliance on Kohler-Langer (2019, Theorem 3) for a sup-norm approximation is a secondary concern: the paper doesn't verify that the cited theorem gives the uniform error used to define r_n, so the downstream rates inherit that gap. The simulations are encouraging, but no code is shipped, so reproducibility is limited.\n\nBottom line: the variance estimation half is a solid, citable contribution. The bootstrap CI is not established as written; it might be repairable, but the current proof doesn't deliver. This deserves a serious referee, and a conditional accept would be too generous as is.","headline":"Solid variance estimation rates for ReLU networks; the bootstrap coverage theorem is not proven as written.","tokens_in":44919,"tokens_out":4279,"would_cite":true,"duration_ms":40741,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G08","62G15","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"A residual-based ReLU network estimator learns conditional variance at near-parametric rates and builds bootstrap confidence intervals with provable coverage.","keywords":["conditional variance estimation","ReLU networks","nonparametric regression","bootstrap confidence intervals","sub-Exponential noise","heteroscedasticity","uncertainty quantification","dense networks"],"falsifier":"Simulate a known pair $(f^*, g^*)$ in the hierarchical composition class, for example $f^*(x)=\\sin(2\\pi x_1)+x_2^2$ and $g^*(x)=\\exp(-|x_1|-|x_2|)$, with sub-Exponential noise, then measure $\\|\\hat{g}-g^*\\|^2$ on a large test set at $n=10^4,10^5,10^6$; if the error does not decay at the stated $\\mathrm{poly}_1(\\log n)(\\phi_n+\\psi_n+1/n)$ rate, Corollary 1 is falsified. For the confidence statement, fix $f^*$ and repeat the full bootstrap procedure many times, counting whether $f^*(X)$ falls in the interval; if the empirical coverage is systematically below $1-\\alpha$ under the paper's assumptions, Theorem 5 is falsified.","tokens_in":43914,"feed_emoji":"📊","tokens_out":8365,"duration_ms":73544,"temperature":0.7,"pith_summary":"This paper tries to establish that dense ReLU networks can do two things at once: estimate the conditional variance in a nonparametric regression and produce a confidence interval for the conditional mean with a finite-sample coverage guarantee. The method is the classic residual recipe, fit the mean with a ReLU network, square the residuals, and fit a second ReLU network to those squares. The main theoretical claims are a non-asymptotic $L^2$ error bound for the variance estimator, stated as the first of its kind for ReLU networks, and a bootstrap procedure whose interval covers $f^*(X)$ with probability at least $1-\\alpha$. The noise assumption is relaxed to sub-Exponential errors, and simulations show the residual-based network beating random forests and MARS on variance estimation.","feed_headline":"ReLU networks now come with coverage-guaranteed confidence intervals","feed_subtitle":"A bootstrap built on dense ReLU variance estimates reaches finite-sample coverage for the conditional mean","key_machinery":"The load-bearing object is the dense ReLU network class $F(L,\\nu)$, fully connected feedforward networks with ReLU activation and all hidden layers of equal width. Three mechanisms carry the argument: the residual construction (estimate the conditional mean, square the residuals, and regress those squares to estimate the conditional variance), the approximation and entropy properties of deep ReLU networks over hierarchical composition classes, and a bootstrap correction that computes quantiles of prediction errors on a held-out validation set rather than quantiles of fitted values. The coverage proof combines concentration for retrained networks with DKW-type bounds on the empirical quantile adjustment.","core_discovery":"On the paper's own terms, the central discovery is that when the true conditional mean $f^*$ and conditional variance $g^*$ belong to a hierarchical composition class, the residual-based dense ReLU estimator $\\hat{g}_{B_n}$ satisfies $\\|g^* - \\hat{g}_{B_n}\\|^2_{L^2} = o_P(\\mathrm{poly}_1(\\log n)(\\phi_n+\\psi_n+1/n))$, and the bootstrap interval $[\\frac{1}{eB}\\sum_{j=B+1}^{B+eB} \\hat{f}_{A_n}^{(j)}(X) \\pm \\Delta(\\alpha)]$ covers $f^*(X)$ with probability at least $1-\\alpha$. This is claimed as the first non-asymptotic conditional variance bound for ReLU networks and the first coverage-guaranteed bootstrap confidence interval in deep ReLU regression, with sub-Exponential noise allowed.","pith_inferences":["The proof strategy suggests the bounded-response assumption behind the coverage theorem could be replaced by a truncation argument for sub-Exponential $Y$, although the paper's stated result requires $P(|Y|\\le A_n)=1$.","The held-out prediction-error bootstrap is a template that could be adapted to other deep uncertainty-quantification methods currently relying on heuristic ensembles; this adaptation is not part of the paper.","The rate in Corollary 1 implies a practical misspecification diagnostic: if a cross-validated residual-variance plot systematically disagrees with the fitted variance surface, the assumed hierarchical composition structure is likely wrong."],"forward_implications":["Conditional variance estimation in heteroscedastic nonparametric regression becomes feasible with deep ReLU networks at non-asymptotic rates, beyond the usual kernel or local-polynomial settings.","The bootstrap confidence interval can be used when the regression function is a hierarchical composition and the noise is only known to be sub-Exponential, without a parametric error distribution.","The non-asymptotic bounds give concrete guidance for choosing the clipping level, retraining count, and interval width from the closed-form expressions in the theorems.","Because the general variance bound transfers to any mean estimator with a known $L^2$ rate, the same three-step framework applies to other function classes beyond ReLU networks.","In the homoscedastic special case the scalar variance estimator attains rate $\\mathrm{poly}_1(\\log n)(\\phi_n+1/n)$, so constant-variance models are covered as a limit of the main result."],"supporting_citations":[{"why":"Supplies the dense ReLU approximation result for hierarchical composition classes used to locate a network within $\\sqrt{\\phi_n}$ of $f^*$ in sup norm.","marker":"Kohler and Langer (2019)"},{"why":"The residual-based conditional variance estimation template the paper generalizes to ReLU networks.","marker":"Fan and Yao (1998)"},{"why":"Provides the VC-dimension bound for piecewise linear networks used to control the entropy of the ReLU function class.","marker":"Bartlett et al. (2019)"},{"why":"Gives the sharp DKW inequality constant used to bound the bootstrap quantile error in the coverage proof.","marker":"Massart (1990)"},{"why":"Supplies the critical-radius machinery connecting covering numbers to empirical process tail bounds.","marker":"Wainwright (2019)"},{"why":"Provides the symmetrization inequality used in the deviation bounds for the mean and variance estimators.","marker":"Van Der Vaart and Wellner (1996)"},{"why":"Origin of the bootstrap resampling scheme used in the confidence interval construction.","marker":"Efron (1992)"},{"why":"Supplies covering-number and regression-estimation lemmas used in the ReLU entropy calculation.","marker":"Györfi et al. (2002)"}],"fun_headline_variants":["Coverage-guaranteed bootstrap intervals for deep ReLU models","First non-asymptotic variance bounds for ReLU networks","Provable confidence intervals for ReLU regression with bootstrap","ReLU nets get finite-sample coverage for conditional mean","Dense ReLU networks: variance estimation and valid CIs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rates inherit from an external approximation theorem, cited in the paper, asserting that dense ReLU networks uniformly approximate functions in the hierarchical composition class at the advertised speed; if that sup-norm approximation only holds in $L^2$, the stated rate $r_n$ and the downstream variance and coverage results do not follow.","fun_headline_variants_meta":{"raw":{"variants":["Coverage-guaranteed bootstrap intervals for deep ReLU models","First non-asymptotic variance bounds for ReLU networks","Provable confidence intervals for ReLU regression with bootstrap","ReLU nets get finite-sample coverage for conditional mean","Dense ReLU networks: variance estimation and valid CIs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000557,"raw_usage":{"total_tokens":2623,"prompt_tokens":889,"completion_tokens":1734,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":1653}},"tokens_in":505,"tokens_out":1734,"duration_ms":12641,"temperature":1.0,"reasoning_tokens":1653,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:24:08.066628+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate a known pair $(f^*, g^*)$ in the hierarchical composition class, for example $f^*(x)=\\sin(2\\pi x_1)+x_2^2$ and $g^*(x)=\\exp(-|x_1|-|x_2|)$, with sub-Exponential noise, then measure $\\|\\hat{g}-g^*\\|^2$ on a large test set at $n=10^4,10^5,10^6$; if the error does not decay at the stated $\\mathrm{poly}_1(\\log n)(\\phi_n+\\psi_n+1/n)$ rate, Corollary 1 is falsified. For the confidence statement, fix $f^*$ and repeat the full bootstrap procedure many times, counting whether $f^*(X)$ falls in the interval; if the empirical coverage is systematically below $1-\\alpha$ under the paper's assumptions, Theorem 5 is falsified.","supporting_citations":[{"cited_title":"On deep learning as a remedy for the curse of dimensionality in nonparametric regression","cited_arxiv_id":null,"evidence_quote":"Supplies the dense ReLU approximation result for hierarchical composition classes used to locate a network within $\\sqrt{\\phi_n}$ of $f^*$ in sup norm."},{"cited_title":"Efficient estimation of conditional variance functions in stochastic regression","cited_arxiv_id":null,"evidence_quote":"The residual-based conditional variance estimation template the paper generalizes to ReLU networks."},{"cited_title":"Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the VC-dimension bound for piecewise linear networks used to control the entropy of the ReLU function class."},{"cited_title":"The tight constant in the dvoretzky-kiefer-wolfowitz inequality","cited_arxiv_id":null,"evidence_quote":"Gives the sharp DKW inequality constant used to bound the bootstrap quantile error in the coverage proof."},{"cited_title":"Weak convergence","cited_arxiv_id":null,"evidence_quote":"Provides the symmetrization inequality used in the deviation bounds for the mean and variance estimators."}],"review_version":1}