{"id":"aa8761b8-343a-4ab9-9dfc-c4c32c1076a7","arxiv_id":"2506.21460","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Wild refitting resamples residuals with random signs, refits the predictor once, and bounds the instance-wise mean-squared error via the resulting wild optimism.","lead":"This paper introduces a one-refit procedure that converts a black-box predictor into an upper bound on its own mean-squared prediction error. It matters because it offers model selection and error reporting for complex predictors such as deep image denoisers with a single additional fit.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline bound is not a usable upper bound: the pilot error term A_n is left unquantified, and for the natural pilot fhat it equals 2r||fhat - f*||_n, making the inequality self-referential.","rationale":"I read the paper's main theorem and proof carefully. The central claim is that a single wild refit yields a high-probability upper bound on the instance-wise excess risk. For this to be a genuine upper bound, every term on the right-hand side of Theorem 1 must be either computable or controlled. The pilot error term A_n(tilde f) is neither. The paper's discussion after Eq. (15b) is explicitly heuristic ('we can expect'), and the manuscript contains no theorem that bounds A_n in terms of known quantities. This is a missing support that the reviewing rules require flagging.\n\nThe concern is not merely abstract. For the default pilot choice tilde f = fhat, the supremum in Eq. (15b) is over the full ball B_{2r}(fhat), so the pilot error is exactly 2r ||fhat - f*||_n. This follows from the fact that the supremum of the linear functional (1/n) sum_i epsilon_i (fhat_i - f*_i) g_i over g with ||g||_n <= 2r is attained at the boundary and equals the norm of the coefficient vector. Hence the main bound becomes Opt* <= gOpt + H + 2r ||fhat - f*||_n. Since the excess risk E = ||fhat - f*||_n^2, the bound is self-referential: the right-hand side contains the square root of the very quantity being bounded. The paper does not solve this implicit inequality and does not state conditions on the signal-to-noise ratio under which 2r||fhat - f*||_n is dominated by gOpt. In high-SNR settings, residuals y - fhat are small and this term can dominate, contradicting the paper's heuristic.\n\nTheorem 2 inherits the same unquantified A_n, so the estimation-error bound used to select the radius r is also not a closed-form upper bound. Thus even if the firm non-expansiveness condition (11a) were verified for every method, the central claim would still fail to deliver a usable, non-asymptotic upper bound unless A_n is controlled. The reader's weakest assumption, firm non-expansiveness, is a real gap for the plug-and-play deep denoiser and the Lagrangian Tikhonov/TV examples, but the pilot-error issue is more fundamental because it affects the main theorem directly and is present even in the convex constrained setting where (11a) holds. Therefore I partially agree with the reader: the reader correctly flagged (11a) as fragile, but the more load-bearing concern is the unquantified pilot error term.\n\nA concrete check is to compute A_n for the paper's own synthetic experiments, where ground truth is known. Because A_n for pilot = fhat has the closed form 2r ||fhat - f*||_n, it can be evaluated exactly once r is selected by the envelope method. Comparing A_n to gOpt and to E will show whether the paper's heuristic claim holds in the demonstrated settings. If A_n is a substantial fraction of E, the paper should either provide a rigorous bound for A_n or restrict the claim to regimes where A_n is provably small. This supports the reader's conditional verdict without changing it.","tokens_in":23842,"tokens_out":18425,"duration_ms":196458,"concrete_test":"Use the known ground truth in the NRSFM experiment (Section 4.1) or the denoising experiments (Section 4.2) to compute, for the radius r selected by the envelope method (18), the value A_n with pilot = fhat, i.e., A_n = 2r ||fhat - f*||_n. Compare this to the wild optimism gOpt(f^diamond_rho) and to the true excess risk E = ||fhat - f*||_n^2. If A_n is comparable to or larger than gOpt (e.g., ratio > 0.5), the paper's assertion that the pilot error is dominated fails for its own examples, and the main bound is vacuous or severely inflated. Additionally, solve the implicit inequality E <= bE_n + 2gOpt + 2H + 4r sqrt(E) for these examples and report the inflation factor of the resulting bound relative to E.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 1 (Eq. 14) bounds the true optimism by the wild optimism plus H_n(t) plus the pilot error A_n(tilde f) (Eq. 15b). The paper does not bound A_n; Section 3.2 only says that 'as long as the pilot function tilde f is a reasonable estimate of f* ... we can expect that the pilot error is dominated by gOpt.' This is a heuristic, not a theorem, and it is false in general. For the simplest and default pilot choice tilde f = fhat, the pilot error can be computed exactly: because the supremum in (15b) is over the full ball B_{2r}(fhat), it equals 2r ||fhat - f*||_n. Thus Theorem 1 takes the form Opt* <= gOpt + H + 2r||fhat - f*||_n, and since the excess risk E = ||fhat - f*||_n^2, the added term is of order r sqrt(E). The bound is self-referential: the RHS depends on the very quantity E being bounded, through ||fhat - f*||_n. The paper neither solves this implicit inequality nor supplies conditions under which A_n is dominated. Theorem 2 (Eq. 16a) suffers from the same difficulty, since A_n appears there too, so the estimation-error bound used to choose r is also not explicit. Consequently, the central claim that wild refitting 'gives an upper bound on prediction error' is not a closed-form, high-probability upper bound unless A_n is controlled; the demonstrations in Section 4 never report A_n.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces 'wild refitting,' a black-box procedure for producing non-asymptotic, high-probability upper bounds on the instance-wise excess risk of penalized least-squares estimators. The procedure forms residuals from a recentering pilot estimator, multiplies them by Rademacher signs and a scale rho, refits the original method to these wild responses, and uses the resulting wild optimism to bound the true optimism. The main theoretical results are Theorem 1, which bounds true optimism by wild optimism plus a deviation term and a pilot error term A_n(ef), and Theorem 2, which uses the same machinery to bound the estimation error ||fhat - fdagger||_n. Numerical demonstrations are given for low-rank NRSFM recovery, plug-and-play image denoising with deep priors, and kernel sketching.","tokens_in":24204,"tokens_out":4879,"duration_ms":62594,"significance":"The idea is attractive: a single black-box refit with Rademacher-symmetrized residuals, combined with sharp concentration for Lipschitz functions of Rademacher variables, could give finite-sample upper bounds under heteroskedastic and heavy-tailed noise. If the advertised bound were fully explicit, the paper would be a useful contribution to risk estimation for complex estimators where Stein-type or cross-validation methods are not available. The paper also gives an explicit algorithmic recipe and empirically illustrates the method on several nontrivial problems. The main current weakness is that the central theorems contain an unquantified pilot error term, which for the default pilot choice is proportional to the square root of the very excess risk being bounded; this prevents the stated results from being the closed-form high-probability upper bounds promised in the abstract and in Section 1.2.","major_comments":[{"comment":"The advertised upper bound is not explicit because the pilot error A_n(ef) is never bounded. For the default and simplest choice ef = fhat, the supremum in (15b) over the full ball B_{2r}(fhat) is exactly 2r ||fhat - f*||_n, so the theorem reduces to Opt*(fhat) <= wild optimism + H_n(t) + 2r ||fhat - f*||_n. Since the excess risk is E = ||fhat - f*||_n^2, the RHS depends on sqrt(E), yielding an implicit inequality that the paper neither solves nor supplements with conditions under which the pilot error is dominated. Section 3.2 only says that 'we can expect' the pilot error to be dominated when ef is a reasonable estimate; this is a heuristic, not a theorem. The abstract's claim of a high-probability upper bound on prediction error is therefore not established by the stated results.","section":"Theorem 1, Eq. (14)-(15b)"},{"comment":"The estimation-error bound suffers from the same unquantified A_n(ef) term and is also not a closed-form bound because brn^2 appears on both sides. In (16a), the term 6||w||_infty/t * brn^2 is retained without any condition ensuring t large enough relative to ||w||_infty to make the coefficient less than 1; the assumption t >= 3 alone does not suffice. Thus the statement that one can 'compute upper bounds on the true error' by varying rho in Section 3.4, as illustrated in Figure 3, is not justified by the displayed inequalities unless additional control on A_n and on ||w||_infty/t is supplied.","section":"Theorem 2, Eq. (16a)-(16b)"},{"comment":"The central assumption of firm non-expansiveness is not verified for the estimators used in the experiments. Eq. (10) restricts the theory to hard constraint sets C (indicator penalties), and the paper explicitly defers Lagrangian regularization to future work. Yet Section 2.4 uses Tikhonov and TV penalties of the form P_q(f) = sum |f(x_{i+1}) - f(x_i)|^q, and Section 4.2 uses a plug-and-play deep denoiser. For these procedures, (11a) is neither proved nor numerically checked, and Lemma 5 and Theorem 2 depend directly on it. Consequently, the numerical demonstrations in Figures 5-8 are not covered by the theory as written.","section":"Section 3.1.1, assumption (11a)"},{"comment":"The experiments report MSE curves for different rho but never report the value of the pilot error A_n(ef), never check whether the radius condition ||fhat - fdagger||_n <= r is met, and do not assess coverage of the stated 1 - 4e^{-t^2} guarantee. The statements 'consistent with our theory' in Section 4.2 are therefore not supported by the numerical evidence in the paper.","section":"Section 4"}],"minor_comments":[{"comment":"There are typographical errors: 'computionally' in the abstract and 'block-box' in the abstract and Section 1.2; these should be corrected.","section":"Abstract and Section 1"},{"comment":"In the last sentence of the proof of Lemma 4, the phrase 'noise scale rho chosen to ensure ||f^D_rho - fhat||_n = 2rho' should read '= 2r', to match the lemma statement.","section":"Section 5.2.3"},{"comment":"The term H_n(t) in Eq. (15a) is said to scale as t||w||_infty / sqrt(n), but the displayed expression has 2||w||_infty t / sqrt(n) multiplied by (3r + ||fdagger - f*||_n); the scaling is fine, but the constant is slightly larger than the text suggests.","section":"Section 3.2"},{"comment":"The text says wild noise scales rho in {1.0, 1.1, 2.2} were used, while Figure 5 is described as showing rho in {1.0, 1.1, 1.2}; this apparent mismatch should be clarified.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be influential if the pilot error term can be controlled or if the theorems are restated as implicit inequalities together with conditions under which they can be solved. I would not reject the manuscript, because the underlying symmetrization idea is sound and the main proof structure appears correct; however, the central advertised guarantee is not yet delivered as stated. A revision should either prove an explicit bound on A_n(ef) for a suitably chosen pilot (e.g., using Theorem 2 plus an approximation inequality) or clearly restrict the claims to settings where A_n is guaranteed to be dominated. The experiments should also report A_n and verify the non-expansiveness assumption for at least one non-projection estimator."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper has a genuinely new idea—a single black-box refit with Rademacher-signed residuals can estimate the optimism of a penalized least-squares estimator—but the headline theorem does not actually deliver a closed-form upper bound, because of the unquantified pilot error A_n. For the natural pilot choice tilde f = fhat, A_n equals 2r ||fhat - f*||_n, which makes the bound in Theorem 1 self-referential: the RHS contains the very excess risk it purports to bound. The paper's own discussion (Section 3.2) merely says that if the pilot is a reasonable estimate, \"we can expect\" A_n to be dominated by the wild optimism; that is a heuristic, not a theorem. The same issue infects Theorem 2, since A_n appears there too, so the estimation-error bound used to set r is not explicit either. The experiments never report A_n.\n\nWhat's genuinely good: the construction is elegant. The connection between the wild optimism and the wild complexity (Lemma 1) is clean, and the use of concentration for Lipschitz functions of Rademacher variables gives sharp deviation bounds that handle heavy-tailed, heteroskedastic noise. The empirical section demonstrates the method works in practice when rho is appropriately tuned, which suggests the idea has real potential for model selection in NRSFM and plug-and-play deep denoising.\n\nThe soft spots beyond A_n: the firm non-expansiveness assumption (11a) is verified for projections onto convex sets, but the demonstrations include plug-and-play deep denoisers and Lagrangian Tikhonov/TV estimators, for which it is not established. The experiments choose rho from a small grid and plot the values that happen to give upper bounds; there's no data-driven rule for selecting rho, despite the theory purportedly giving guidance. So the empirical claims outrun the theoretical scope.\n\nOverall: this is a thought-provoking paper with a real gap at the center. The gap is addressable—one could try to control A_n via an independent pilot or via a more careful two-stage argument—but as written, the central theorem is conditional on an unquantified quantity. It deserves a serious referee, but the referee should require a fix for A_n before the advertised guarantee can be taken as proven. If I were asked, I'd send it to review but with a strong request to address the self-referential issue.","headline":"A clever one-refit method for optimism estimation that is undermined by an unquantified, self-referential pilot error term in the main theorems.","tokens_in":24683,"tokens_out":3260,"would_cite":false,"duration_ms":37541,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G08","62J07","62G09"],"pacs":[],"model":"deepseek-v4-flash","headline":"Single wild refit bounds a black-box predictor's excess risk.","keywords":["wild refitting","black-box prediction","excess risk upper bound","Rademacher symmetrization","nonparametric regression","wild bootstrap","plug-and-play denoising","estimation error"],"falsifier":"Take a deep plug-and-play denoiser on a fixed image, draw a set of perturbation vectors u, and numerically check inequality (11a); if any perturbation yields a squared refit distance noticeably larger than the inner product, the estimation-error bound in Theorem 2 and the resulting excess-risk upper bound lose their stated foundation.","tokens_in":23661,"feed_emoji":"📈","tokens_out":4553,"duration_ms":54461,"temperature":0.7,"pith_summary":"This paper introduces a procedure called wild refitting that produces a high-probability upper bound on the instance-wise mean-squared prediction error of a penalized least-squares estimator, using only a single dataset and black-box access to the fitting method. The procedure computes residuals, multiplies them by random signs, rescales them by a noise factor, and refits the same method on the recentered problem; the resulting wild optimism can then be used to upper-bound the true optimism, and hence the excess risk. The advantage over hold-out, cross-validation, and bootstrap methods is that it is non-asymptotic, makes just one refitting call, and tolerates heteroskedastic and heavy-tailed noise. The paper also shows how to upper-bound the unknown estimation error in terms of a quantity called wild complexity, so the bound can be made fully computable.","feed_headline":"One extra refit can bound a predictor's true error","feed_subtitle":"Wild refitting symmetrizes residuals with random signs and needs only one black-box call to upper-bound the MSE of penalized least-squares…","key_machinery":"The central objects are the wild optimism and the wild complexity. The wild optimism is the Rademacher-symmetrized inner product between the residual-based noise vector and the displacement of the refit from the original predictor, and the wild complexity Wn(r) is its supremum over a radius-r ball around the predictor; Lemma 1 shows the two coincide for the refit whose distance from the original predictor equals r. The argument is carried by firm non-expansiveness of the estimator around the regression function, which makes projections onto convex sets admissible, and by concentration inequalities for convex Lipschitz functions of Rademacher variables, which give the exponential tails.","core_discovery":"The central claim is that, for fixed design with symmetric independent noise, the true optimism of a penalized least-squares estimator is bounded by the wild optimism from a single refit, provided the wild noise scale is chosen so the refit lies at exactly twice some radius that dominates the estimation error. More precisely, Theorem 1 gives Opt*(fhat) ≤ gOpt(f_rho) + H_n(t) + A_n(ef) with probability at least 1 − 4 exp(−$t^{2}$), where the deviation term decays like t times the sup-norm of the noise divided by the square root of the sample size, and the pilot error term is small whenever the recentering function is a good estimate of the regression function. Theorem 2 closes the loop by showing the estimation error itself can be upper-bounded by a computable envelope built from the wild complexity, so the procedure yields a self-contained, non-asymptotic, high-probability upper bound on the instance-wise excess risk. The proof rests on Rademacher symmetrization of residuals and on sharp concentration properties of convex Lipschitz functions of Rademacher variables, in contrast to the asymptotic distributional guarantees of the classical wild bootstrap.","pith_inferences":["Beyond the paper, the same recentering and symmetrization scheme could be extended to random-design settings by first conditioning on the realized covariates and then averaging over their distribution, though the fixed-design analysis here does not cover that averaging step.","The firm non-expansiveness condition is verified for convex projections but not for the deep plug-and-play denoiser used in the demonstrations, so a direct numerical check of inequality (11a) on those denoisers would be a natural test of whether the stated guarantee transfers to that setting.","Choosing the recentering function ef differently from the fitted predictor fhat could reduce the pilot error term A_n(ef), suggesting a two-stage wild refitting scheme in which a preliminary pilot estimate is used to shrink that residual term before the main refit.","The method's reliance on least-squares structure suggests that analogous wild-refitting bounds might be derivable for other convex loss-based M-estimators, with the optimism identity replaced by a Bregman-divergence analogue."],"forward_implications":["One black-box call to the fitting method is enough to obtain a non-asymptotic upper bound on the excess risk, so risk estimation no longer requires repeated refitting as in a full bootstrap.","Model selection can be performed by minimizing the wild-refitting upper bound over tuning parameters, and the paper demonstrates this on nuclear-norm-constrained matrix recovery and on deep plug-and-play image denoisers.","The bound holds for heteroskedastic and heavy-tailed noise, with the noise entering only through its sup-norm, so the method remains meaningful when only a finite polynomial moment is available.","The estimation-error bound can be optimized over the wild noise scale by balancing the two terms in the envelope max(r, Wn(r)/r), giving a concrete rule for choosing rho.","If the estimator is firmly non-expansive and the pilot error term is controlled, the wild-refitting upper bound is valid for a single realization of the random signs, rather than only in distribution or on average."],"supporting_citations":[{"why":"Supplies the concentration inequality for convex Lipschitz functions of Rademacher variables used to obtain the exponential tails in Theorems 1 and 2.","marker":"[28]"},{"why":"Provides the standard optimization-theoretic fact that projections onto convex sets are firmly non-expansive, which is the source of condition (11a).","marker":"[12]"},{"why":"Establishes the wild bootstrap residual-symmetrization idea that the wild refitting procedure adapts.","marker":"[29]"},{"why":"Gives the wild bootstrap variant with Rademacher signs that motivates the symmetrization step and the handling of non-i.i.d. data.","marker":"[17]"},{"why":"Provides the concentration-of-measure results for Lipschitz functions of independent Rademacher variables that underlie the sharp probability bound 1 − 4 exp(−t^2).","marker":"[15]"},{"why":"Defines the low-rank nuclear-norm-constrained estimator used in the non-rigid structure-from-motion demonstration.","marker":"[7]"},{"why":"Supplies the proximal plug-and-play deep denoiser used in the image-restoration experiments.","marker":"[13]"},{"why":"Provides the randomized sketching kernel regression method used in the numerical illustration of the estimation-error envelope.","marker":"[30]"}],"fun_headline_variants":["One wild refit upper-bounds prediction error","Single refit with random signs certifies MSE","Wild refit: a tight bound from one extra call","Bounding true error without extra data","Refit once, bound the error everywhere"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the black-box estimator is firmly non-expansive around the true regression function: for any perturbation of the response vector, the squared distance between the two refits must be no larger than the inner product of that perturbation with their difference.","fun_headline_variants_meta":{"raw":{"variants":["One wild refit upper-bounds prediction error","Single refit with random signs certifies MSE","Wild refit: a tight bound from one extra call","Bounding true error without extra data","Refit once, bound the error everywhere"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1470,"prompt_tokens":1018,"completion_tokens":452,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":382}},"tokens_in":634,"tokens_out":452,"duration_ms":5479,"temperature":1.0,"reasoning_tokens":382,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:25:58.032628+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a deep plug-and-play denoiser on a fixed image, draw a set of perturbation vectors u, and numerically check inequality (11a); if any perturbation yields a squared refit distance noticeably larger than the inner product, the estimation-error bound in Theorem 2 and the resulting excess-risk upper bound lose their stated foundation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the concentration inequality for convex Lipschitz functions of Rademacher variables used to obtain the exponential tails in Theorems 1 and 2."},{"cited_title":"Hiriart-Urruty and C","cited_arxiv_id":null,"evidence_quote":"Provides the standard optimization-theoretic fact that projections onto convex sets are firmly non-expansive, which is the source of condition (11a)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the wild bootstrap residual-symmetrization idea that the wild refitting procedure adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the wild bootstrap variant with Rademacher signs that motivates the symmetrization step and the handling of non-i.i.d. data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the concentration-of-measure results for Lipschitz functions of independent Rademacher variables that underlie the sharp probability bound 1 − 4 exp(−t^2)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the low-rank nuclear-norm-constrained estimator used in the non-rigid structure-from-motion demonstration."},{"cited_title":"Hurault, A","cited_arxiv_id":null,"evidence_quote":"Supplies the proximal plug-and-play deep denoiser used in the image-restoration experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the randomized sketching kernel regression method used in the numerical illustration of the estimation-error envelope."}],"review_version":1}