{"id":"474b53b1-0d04-479f-9b52-726761d485fa","arxiv_id":"2509.02735","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The authors construct prediction intervals by calibrating estimated conditional CDF values on a grid, and show asymptotic validity for DNN and kernel estimators, with a finite-sample coverage guarantee only under an oracle DNN assumption.","lead":"This paper introduces a calibration prediction interval (cPI) for regression, built from DNN or kernel estimates of the conditional CDF at grid points, with a monotonicity correction and a search over grid indices. The method aims to fix the undercoverage that plagues plug-in prediction intervals, and the authors prove asymptotic coverage plus conditional finite-sample coverage under strong assumptions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 5.1's proof assumes L2 consistency plus positive density gives pointwise convergence of bH_j at every xf; this implication is false, so the main DNN asymptotic coverage theorem is not established.","rationale":"I read the paper as proposing a calibration construction that converts any consistent conditional-CDF estimator into a PI with asymptotically valid conditional coverage, plus finite/large-sample guarantees. The empirical sections are careful: five replicated simulation models, two real datasets, comparisons with deep generative PIs, and the cPI performs well. These results support practical usefulness. The theoretical core, however, is the proof of Theorem 5.1. The proof is one paragraph, and its key step—getting pointwise consistency bH_j(xf)→H_j(xf) for every xf from the L2 bound in Eq. (11) plus f>0—does not follow. A simple bump sequence is a counterexample to the implication. This is not a disagreement with the consensus; it is an internal gap in the argument. The reader's identified weakest assumption (B4) is also serious: Remark 5.3 admits that taking delta_n=0 is 'not practically accomplishable', so Theorem 5.2 provides no practical finite-sample guarantee. I therefore partially agree with the reader: they flagged the sketchiness of the proofs, but I would elevate the pointwise-convergence gap over B4 as the single load-bearing issue, since Theorem 5.1 is the main claim for DNN cPIs. The proposed check—attempting the implication with a localized ReLU bump—is a decisive analytical test: if the bump is admissible in the DNN class, the proof step is invalid; if some constraint rules it out, that constraint should be stated and used. Because the result may be repairable with a sup-norm convergence theorem, the verdict remains conditional rather than outright rejection; this matches the reader's verdict, so no verdict change is needed beyond the altered justification.","tokens_in":36475,"tokens_out":9367,"duration_ms":86876,"concrete_test":"Test the inference in the proof of Theorem 5.1 by attempting to prove pointwise convergence from Eq. (11) plus B2. Concretely, with X=[0,1], f≡1, H_j≡0, and xf=1/2, set bH_n to be the ReLU-representable triangular spike of height 1 and width 2n^{-1/2} centered at xf; its L2(f) norm tends to 0 while bH_n(xf)=1 for every n. If this spike is admissible in the DNN class of Theorem A.1 (width/depth grow with n; truncation to [−M1,M1] allows height 1), the implication fails and Theorem 5.1 needs an additional uniform-convergence condition. If the class excludes such spikes, identify the exclusion and derive the corresponding sup-norm bound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is the asymptotic coverage guarantee for DNN cPIs (Theorem 5.1). Its proof reduces to Eq. (11), ∫(bH_j−H_j)^2 f dx → 0, and then asserts that under B2 (f(x)>0), bH_j(xf)→H_j(xf) with probability tending to 1 for all xf. This inference is invalid: L2 convergence does not imply pointwise convergence at every fixed point, even if the density is bounded below. Example: on X=[0,1] with f≡1, take H_j≡0 and bH_n the triangular ReLU-representable bump of height 1 and width 2n^{-1/2} centered at xf; then ∫(bH_n−H_j)^2 f →0 but bH_n(xf)=1 for all n. B2 says only that f is positive and bounded; on a compact set this does not give inf f>0, and positivity does not convert L2 convergence into uniform or pointwise convergence. Because Algorithms 3–7 select endpoints from estimated CDF values bH_j(xf) at the fixed future point, the theorem needs convergence at that xf, not just convergence in L2(X). Theorem A.1 supplies only an L2 error bound; no L∞ or pointwise DNN consistency result is cited. Thus the central asymptotic claim is currently unsupported. The reader's B4 concern (Theorem 5.2) and the incompatible condition ln(n)/(nh^{d+1})→∞ in Lemma 5.1/Theorem 5.3 (which should be →0 for the variance term to vanish) are real, but the Theorem 5.1 proof gap is the most load-bearing because it underlies the headline theoretical result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a calibration prediction interval (cPI) for conditional prediction in nonparametric regression. The method estimates the conditional CDF FY|X at fixed grid points using either deep ReLU networks (one network per grid point) or kernel density estimators, applies one of three monotonicity corrections, and then selects interval endpoints so that the estimated CDF mass is at least 1−α. Several variants are proposed (minimal length, symmetric around the estimated mean, asymmetric), including an adjusted version that moves endpoints outward by one grid step. The authors claim asymptotic conditional coverage for both DNN-based and kernel-based cPIs, a high-probability large-sample coverage guarantee for the kernel version, and a finite-sample coverage guarantee for DNN cPIs under oracle-type assumptions. The practical claims are supported by simulations on six nonlinear models and by two wine-quality datasets.","tokens_in":36939,"tokens_out":10039,"duration_ms":92878,"significance":"If the theoretical guarantees were valid, the paper would make a practically useful contribution: the calibration idea is simple, avoids normality and linearity assumptions, compensates for estimation variability, and is parallelizable. The numerical study is substantial (six models, normal/t/skew-normal errors, two sample sizes, several DNN widths), and the comparison on the wine datasets is informative. The monotonicity-correction methods are clearly described and appear to work well in the experiments. However, the main DNN asymptotic theorem is not established by the given proof, the kernel lemma contains a bandwidth condition that is inconsistent with consistency, and the finite-sample guarantee rests on an acknowledged oracle assumption for which no supporting result is provided. The theoretical claims in the abstract are therefore currently stronger than what the manuscript establishes.","major_comments":[{"comment":"The proof asserts that ∫(bH_j−H_j)^2 f dx → 0 with probability tending to 1, together with B2 (f(x)>0), implies bH_j(xf) → H_j(xf) for every xf. This implication is false: L2 convergence does not imply pointwise convergence, and positivity of a density on a compact set does not imply a positive lower bound. A sequence of narrow spike functions centered at xf can have L2 distance tending to zero while remaining equal to 1 at xf. Theorem A.1, the only DNN error bound cited in this paper, supplies L2 convergence only; no pointwise or uniform consistency result for bH_j is given. Since Algorithms 3–7 evaluate bH_j at the fixed future point xf and select endpoints from those values, the theorem requires convergence at that xf, and the claim 'for any xf in its domain X' requires simultaneous control over all xf. The headline asymptotic coverage result for DNN cPIs is therefore unsupported by the current proof.","section":"Section 5.1.1, Eq. (11), proof of Theorem 5.1"},{"comment":"The condition ln(n)/(nh^{d+1}) → ∞ stated in Lemma 5.1 is inconsistent with the displayed error bound. The variance term (ln(n)/(nh^{d+1}))^{1/2} converges to zero only if ln(n)/(nh^{d+1}) → 0; under the stated condition it diverges, so the claimed uniform consistency of fhat(y|x) does not follow. The same sign error appears in Theorem 5.3. Because the kernel cPI's asymptotic and large-sample coverage results are derived directly from Lemma 5.1, this condition must be corrected before those results can be accepted.","section":"Lemma 5.1 and Theorem 5.3, Eq. (8)"},{"comment":"The finite-sample coverage theorem is conditional on B4, which assumes ||bH_j − H*_j||_{L∞(X)} ≤ δ_n with δ_n treated as exactly 0 for all n. Remark 5.3 states that no existing result supports B4 and that achieving it 'may not be practically accomplishable'. The proof of Theorem 5.2 is a non-quantitative description of a diagram rather than a verification that the monotonicity correction and the one-step outward adjustment guarantee eFAvg(q_l~) ≤ α/2 and eFAvg(q_r~) ≥ 1−α/2 under B5. As stated, the theorem does not provide a quantitative finite-sample coverage claim for any realistic estimator, and the abstract's statement that the cPI based on DNN works 'even with finite samples' is not supported.","section":"Section 5.1.2, B4, Remark 5.3, Theorem 5.2"},{"comment":"The proof of Theorem 5.1 does not address the discrete index-selection step. Even if bH_j(xf) converged at each grid point, coverage requires that, for the particular corrected estimators eFLtoR, eFRtoL, and eFAvg, the selected indices l and r satisfy the corresponding inequalities with enough slack to imply true coverage. The monotonicity corrections C1–C3 can move estimated values away from H_j at the endpoint grid points, and the arg max/arg min operations in Algorithms 6 and 7 have no defined behavior when no grid point satisfies the required inequality. A rigorous proof would need to control these quantities simultaneously, for all grid points and for all xf in the domain; the current one-paragraph proof does not do so.","section":"Section 5.1.1, Algorithms 3–7"}],"minor_comments":[{"comment":"The line '|H0(u) − H0(v)| ≤ ε·M·∞' appears to be a typo; it should be ε·M.","section":"Proof of Lemma 2.1"},{"comment":"The sentence 'all cPIs with DNN estimators guarantee at least 1−α conditional coverage asymptotically' should refer to kernel estimators, since the surrounding argument concerns the kernel-based estimators bHk_j.","section":"Section 5.2.1, paragraph after Lemma 5.1"},{"comment":"'Fine the index c' should read 'Find the index c'.","section":"Algorithm 4, Step 3"},{"comment":"Please define the convention when no grid point satisfies eFAvg(qk) ≤ α/2 or eFAvg(qk) ≥ 1−α/2; otherwise the arg max and arg min are taken over an empty set.","section":"Algorithms 6 and 7"},{"comment":"The statement should specify whether the number of grid points g is fixed or grows with n, and it should specify the probability space for P(Y ∈ bI | Xf = xf), since bI depends on the training sample.","section":"Theorem 5.1 statement"},{"comment":"The symbol '√–' used for the DNN-based large-sample coverage entry is not defined in the table note; please explain it in the caption.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper's practical idea is attractive and the empirical work is extensive, but the theoretical claims are currently much stronger than what is proved. The main DNN asymptotic theorem needs either a uniform or pointwise DNN consistency result, or a weakened statement; the kernel lemma needs a corrected bandwidth condition; and the finite-sample theorem should be rephrased as a conditional statement under an explicitly unproved oracle assumption. I would not require code for revision, but detailed derivations for the index-selection step would help."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about arXiv:2509.02735. First, the core idea is genuinely useful: estimate the conditional CDF at a fixed grid of y-values, monotonize, then search for an interval with estimated mass at least 1-alpha. That reverse approach is a clean way to absorb estimation variability without conformal or generative machinery. Second, the headline theory as written does not prove what it claims. The stress-test note is correct: Theorem 5.1's proof reduces to an L2 error bound, then asserts pointwise convergence of the DNN at the fixed future point because the design density is positive. L2 convergence does not imply pointwise convergence, and the DNN class with ReLU can produce a spike that is invisible in L2 but undermines the coverage argument. So the asymptotic guarantee for the DNN cPI is currently unsupported.\n\nThe paper does some things well. The monotonicity corrections C1-C3 are sensible and are shown to work in simulations. The empirical comparison on wine data is honest and the method appears competitive with deep generative PIs. The authors are also explicit that B4 (the oracle DNN assumption behind the finite-sample theorem) 'is possible but may not be practically accomplishable' — that is good honesty, but it means Theorem 5.2 is a conditional statement with no verified condition.\n\nOther soft spots are real but less damaging. Theorem 5.2's proof is essentially a diagram and an assertion. Lemma 5.1 and Theorem 5.3 state ln(n)/(nh^{d+1}) -> infinity, which would make the variance term blow up; the correct condition for the stated rate is ln(n)/(nh^{d+1}) -> 0. That is an internal inconsistency with real teeth. No code is shipped, which is minor since the datasets and seeds are specified.\n\nOverall, the practical method has value, and the paper deserves a serious referee. But a referee should ask for a real proof of Theorem 5.1 — either uniform or pointwise DNN convergence over the relevant class, or a statement that only L2-consistency-driven coverage is claimed. The finite-sample claims should be downgraded or given non-oracle conditions. I would send it to review, because the idea is worth the referee time and the paper is salvageable.","headline":"The calibration PI idea is practical and worth engaging with, but the main DNN coverage theorem is not proven as written and the finite-sample guarantee rests on an admitted oracle assumption.","tokens_in":37345,"tokens_out":2104,"would_cite":true,"duration_ms":19863,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G08","62G15","62G20","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"A prediction interval built by calibrating conditional CDF estimates at grid points can restore nominal coverage in nonparametric regression.","keywords":["prediction interval","calibration","deep neural networks","kernel estimation","uncertainty quantification","coverage guarantee","conditional coverage","nonparametric regression"],"falsifier":"Train the DNNs of Algorithm 1 on simulated data from Model-1 with $n = 2000$, construct $PI_{aaa}$ at $g = 200$ using standard gradient descent, and compare the empirical conditional coverage over 500 replications with the nominal $1-\\alpha$. If coverage systematically falls below nominal, the finite-sample guarantee has no quantitative support because Assumption B4, that the trained $\\hat H_j$ matches the oracle $H_j^*$ in $L_\\infty$ with error exactly zero, is violated. Directly computing $\\|\\hat H_j - H_j^*\\|_{L_\\infty}$ on these trained networks would settle whether the assumed oracle condition holds.","tokens_in":36182,"feed_emoji":"🎯","tokens_out":9170,"duration_ms":77903,"temperature":0.7,"pith_summary":"The paper aims to establish that prediction intervals in nonparametric regression can be made conditionally valid at the nominal $1-\\alpha$ level by a calibration step that explicitly accounts for estimation variability. Rather than assuming normality or linearity, it estimates the conditional distribution function at many fixed thresholds using deep neural networks or kernel estimators, then chooses interval endpoints from those thresholds so that the estimated probability mass inside the interval is at least $1-\\alpha$. The paper proves asymptotic coverage for the DNN and kernel versions, a high-probability large-sample coverage guarantee for the kernel version, and a conditional coverage guarantee for the DNN version in finite samples under strong oracle and grid-spacing assumptions. A sympathetic reader would care because plug-in intervals, including the normal-based one, systematically undercover when the estimated mean and variance carry estimation error, and the proposed cPI is a simple, parallelizable correction.","feed_headline":"Calibration step restores nominal prediction-interval coverage","feed_subtitle":"By estimating the conditional CDF on a grid and widening the interval, the method offsets estimation variability in nonparametric…","key_machinery":"The central machinery is a grid of threshold indicators $Z_j = \\mathbf 1\\{Y \\le q_j\\}$ for $j = 1,\\dots,g$, each regressed on $X$ by its own DNN, or, for the kernel variant, obtained by integrating a conditional density estimate. This turns conditional CDF estimation into $g$ binary regression problems whose estimators $\\hat H_j(X_f)$ are continuous functions of $x_f$ by Lemma 2.2. The paper's methodological contribution is the subsequent calibration: three monotonicity corrections, $\\hat F_{LtoR}$, $\\hat F_{RtoL}$, and their average $\\hat F_{Avg}$, render the estimated CDF nondecreasing, and an index-selection step picks endpoints with estimated mass at least $1-\\alpha$. The adjusted version $PI_{aaa}$ shifts both endpoints outward one grid cell to compensate for worst-case estimation error, which is the mechanism behind the finite-sample coverage claim. The proof engine is a non-asymptotic DNN risk bound, stated as Theorem A.1, that makes $\\hat H_j$ consistent, together with a DNN approximation bound, Lemma A.1, expressed through the modulus of continuity.","core_discovery":"The core claim is that the so-called calibration PI (cPI) computed by Algorithms 3 to 7 satisfies $P(Y \\in \\hat I \\mid X_f = x_f) \\ge 1-\\alpha$ as $n \\to \\infty$ (Theorem 5.1), and that the finite-sample version $PI_{aaa}$ satisfies the same inequality under oracle DNN training and a sufficiently large grid spacing (Theorem 5.2). The reason a naive interval fails is that it replaces the true conditional mean and variance by plug-in estimates and ignores how much those estimates vary; the cPI instead estimates the conditional CDF $F_{Y|X_f}(q_j)$ at grid points $q_j$, corrects the estimates for monotonicity, and then selects $l$ and $r$ so that $\\hat F(q_r) - \\hat F(q_l) \\ge 1-\\alpha$. By construction the interval is widened just enough to absorb the estimation error, and the adjusted version widens it one further grid step. The paper also claims the kernel-based cPI attains coverage with high probability for large samples (Theorem 5.3), a stronger property than plain asymptotic validity.","pith_inferences":["Inference: the same grid-of-thresholds calibration could be applied to any base probabilistic regression model, such as gradient boosting, Gaussian processes, or random forests, so the method's scope is wider than DNN and kernel estimators.","Inference: the one-step outward adjustment in $PI_{aaa}$ is a generic conservative device; if a base estimator has a known uniform error bound $\\delta_n$, moving each endpoint one grid step should restore coverage whenever the grid spacing exceeds $2\\delta_n$, a condition analogous to B5 that could be tested empirically.","Inference: a data-driven choice of $g$, for instance the smallest grid whose validation coverage is at least $1-\\alpha$, would remove the user-set grid hyperparameter and is a natural extension the paper does not pursue.","Inference: if Assumption B4 fails, as the authors concede is likely, a bootstrap estimate of the $L_\\infty$ error of $\\hat H_j$ could substitute for the oracle condition and yield an adjusted interval whose length reflects actual estimation variability."],"forward_implications":["Plug-in intervals that ignore estimation variability undercover even when the error is normal; the paper's simulations show the normal-based $PI_b$ stays below the nominal level at $n = 10{,}000$.","Any consistent estimator of the conditional CDF can be plugged into the calibration procedure; DNN and kernel estimators are the two instantiations the paper develops.","Because the $g+1$ DNNs in Algorithm 1 can be trained in parallel, the calibration step need not add significant wall-clock time over a single point-prediction DNN.","The kernel-based cPI gives a high-probability large-sample coverage guarantee, a middle ground between asymptotic validity and the finite-sample guarantee.","The finite-sample coverage claim holds only under Assumptions B4 and B5; without oracle training or a wide grid, the guarantee has no quantitative support."],"supporting_citations":[{"why":"Supplies the DNN risk bound restated as Theorem A.1, giving the $L_2$ consistency of the conditional mean and CDF estimators that underpins Theorem 5.1.","marker":"Wu and Politis (2024)"},{"why":"Introduces the pertinent PI idea of widening intervals to capture estimation variability, which the cPI's calibration step operationalizes.","marker":"Politis (2015)"},{"why":"Develops the pertinent PI in regression that cPI extends, providing the benchmark against which the undercoverage correction is motivated.","marker":"Wang and Politis (2021)"},{"why":"Provides the monotonicity-correction ideas adapted as C1 and C2 in the paper, used to make the estimated conditional CDF nondecreasing.","marker":"Das and Politis (2020)"},{"why":"Supplies the DNN approximation bound through the modulus of continuity, stated as Lemma A.1, which underlies Assumption B5 and the finite-sample argument.","marker":"Shen et al. (2021)"},{"why":"Gives the impossibility of distribution-free finite-sample conditional coverage, motivating the distributional restrictions under which Theorem 5.2 operates.","marker":"Lei and Wasserman (2014)"},{"why":"Provides the kernel density estimation framework and proof techniques used in Lemma 5.1 and the kernel cPI theory.","marker":"Li and Racine (2007)"},{"why":"Gives pseudo-dimension bounds for ReLU networks, used in Theorem 5.1's sample-size requirement.","marker":"Bartlett et al. (2019)"}],"fun_headline_variants":["Calibration widens intervals to restore nominal coverage","Estimation-aware calibration yields valid prediction intervals","Nonparametric intervals achieve coverage via calibration","Calibrated intervals absorb estimation error for true coverage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The finite-sample coverage guarantee in Theorem 5.2 assumes the trained DNN estimates are as good as the best DNN in the class in $L_\\infty$ norm, with the estimation error sequence $\\delta_n$ treated as exactly zero, an assumption the authors say is possible but may not be practically accomplishable.","fun_headline_variants_meta":{"raw":{"variants":["Calibration widens intervals to restore nominal coverage","Estimation-aware calibration yields valid prediction intervals","Nonparametric intervals achieve coverage via calibration","Calibrated intervals absorb estimation error for true coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000825,"raw_usage":{"total_tokens":3633,"prompt_tokens":994,"completion_tokens":2639,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":2594}},"tokens_in":610,"tokens_out":2639,"duration_ms":19612,"temperature":1.0,"reasoning_tokens":2594,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:34:26.919208+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the DNNs of Algorithm 1 on simulated data from Model-1 with $n = 2000$, construct $PI_{aaa}$ at $g = 200$ using standard gradient descent, and compare the empirical conditional coverage over 500 replications with the nominal $1-\\alpha$. If coverage systematically falls below nominal, the finite-sample guarantee has no quantitative support because Assumption B4, that the trained $\\hat H_j$ matches the oracle $H_j^*$ in $L_\\infty$ with error exactly zero, is violated. Directly computing $\\|\\hat H_j - H_j^*\\|_{L_\\infty}$ on these trained networks would settle whether the assumed oracle condition holds.","supporting_citations":[{"cited_title":"Model-Free Prediction and Regression: A Transformation-Based Approach to Inference","cited_arxiv_id":null,"evidence_quote":"Introduces the pertinent PI idea of widening intervals to capture estimation variability, which the cPI's calibration step operationalizes."},{"cited_title":"Model-free Bootstrap and Conformal Prediction in Regression: Conditionality, Conjecture Testing, and Pertinent Prediction Intervals","cited_arxiv_id":"2109.12156","evidence_quote":"Develops the pertinent PI in regression that cPI extends, providing the benchmark against which the undercoverage correction is motivated."},{"cited_title":"Nonparametric estimation of the conditional distribution at regression boundary points","cited_arxiv_id":null,"evidence_quote":"Provides the monotonicity-correction ideas adapted as C1 and C2 in the paper, used to make the estimated conditional CDF nondecreasing."},{"cited_title":"Nonparametric econometrics: theory and practice","cited_arxiv_id":null,"evidence_quote":"Provides the kernel density estimation framework and proof techniques used in Lemma 5.1 and the kernel cPI theory."}],"review_version":2}