{"id":"8d4fc2d3-388f-4ae5-8d2f-490b43a70422","arxiv_id":"2505.16923","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"TULiP turns weight perturbations around a converged classifier into an OOD detection score, showing strong near-OOD performance on the OpenOOD benchmark.","lead":"This paper introduces TULiP, a method that estimates how uncertain a trained image classifier is on a new input by adding small random and directional perturbations to the model's weights and averaging the resulting predictions. It is designed for out-of-distribution detection without retraining or accessing the original training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's S does not implement Eq. 6: the trace terms carry an ϵ² factor while the gradient term D carries ϵδ, so the relative weighting is set by OOD-validation-tuned hyperparameters, not by Theorem 1.","rationale":"I read the paper in good faith. The theoretical derivation of Eq. 5 and the perturbe-then-train picture are plausible under the stated assumptions, and the empirical improvements on near-OOD benchmarks, if reproducible, would be valuable. The reader's weakest assumption—that real ResNet training is far from the lazy regime—is real but explicitly acknowledged by the authors, who frame the paper as showing that the method works 'even beyond the ideal regime'. That concern weakens the theory's applicability but does not by itself invalidate the empirical claim. My stress-test identifies a more specific and more damaging issue: the step from the proven bound to the implemented score is not a faithful instantiation of Eq. 6. Proposition 1 and the finite-difference approximation imply that eΘTr and D carry different powers of ϵ, so the combination S = J²(eΘTr+ΘXX−λD) has a relative weighting between the trace and gradient terms that is not the one in the bound. A single multiplicative calibration constant J cannot repair this mismatch; the effective balance is set by λ, δ, and Jscaling, with Jscaling tuned on OOD validation data. Consequently, the paper's central claim that TULiP is 'theoretically-driven' and that its performance follows from Theorem 1 is not currently supported. The issue is concrete and testable: recomputing S with the correctly scaled estimators from Prop. 1 and Eq. 9 would either confirm that the ranking is stable (rescuing the heuristic) or show that the reported AUROC depends on the unscaled hyperparameter combination. The lack of released code and error bars makes this check necessary rather than optional. For these reasons I retain the reader's CONDITIONAL verdict: the paper should not be rejected outright, but the connection between theory and algorithm must be fixed or explicitly relabeled as heuristic, and the experiments should be released with the corrected scaling and uncertainty estimates.","tokens_in":22052,"tokens_out":16536,"duration_ms":125252,"concrete_test":"Reproduce Algorithm 1 on the ImageNet-1K split with the paper's hyperparameters (ϵ=0.005, δ=8, λ=1.25, M=10) for a few hundred ID and OOD samples, recording eΘTr, ΘXX, D, S, and γ. Then recompute the score with the correctly scaled quantities eΘTr/ϵ², ΘXX/ϵ², and D/(ϵδ√o), re-tuning only the overall constants (J or K) on the same validation data. If the near-OOD AUROC ranking changes materially (e.g., by more than 1–2 points), the reported performance is an artifact of the unscaled heuristic combination rather than a consequence of Eq. 6. Also check how often S≤0; if S≤0 for most inputs, γ=0 and TULiP reduces to plain entropy, contradicting Table 1.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The claimed theoretical grounding of TULiP breaks at the step where Eq. 6 is converted into Algorithm 1. Proposition 1 (with Γ=I) states that (1/ϵ²)E_v‖f_emp(z;θ_T+v)−f_emp(z;θ_T)‖² → Tr(Θ(z,z)). Thus the quantity eΘTr computed in line 6 is an estimator of ϵ²TrΘ(z,z), not of TrΘ(z,z), for the stated ϵ=0.005. If ΘXX is estimated by the same finite-difference procedure on the validation set, it also carries the factor ϵ². In contrast, line 7 computes D = √o‖f_emp(z;θ_T+ϵδ(θ_T−θ_ts))−f_emp(z;θ_T)‖ ≈ ϵδ√o‖∇θf_T(z)(θ_T−θ_ts)‖. Line 8 then forms S = J²(eΘTr+ΘXX−λD), which estimates J²[ϵ²(TrΘ(z,z)+E_xTrΘ(x,x)) − λϵδ√o‖∇θf_T(z)(θ_T−θ_ts)‖]. Eq. 6 requires coefficients of order 1 on the trace terms and −2K on the gradient norm. Because J is a single scalar, calibrating J cannot fix this relative weighting: the two z-dependent terms enter with coefficients proportional to ϵ² and ϵ, so as ϵ→0 the gradient term dominates and in the reported setting ϵ/(δ√o) is about 2×10⁻⁵. The assertion that line 8 is an 'Estimation of Eq. 6 up to square root' is therefore not supported by the stated propositions. The actual OOD ranking is controlled by the arbitrary combination λδ and by Jscaling∈{1,...,2} searched on OOD validation data; the theoretical bound plays no demonstrated role in setting the relative scale of the two terms. This severs the link between the reported near-OOD state-of-the-art results and the paper's theoretical argument.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TULiP, a post-hoc OOD detection score for pre-trained classifiers. It models epistemic uncertainty as the effect of a hypothetical functional perturbation applied before convergence under linearized (lazy) training dynamics, derives a bound (Theorem 1, Eqs. 5-6) on the resulting logit difference, and then implements the bound using Gaussian and deterministic weight perturbations. A surrogate-posterior envelope is constructed by matching the variance of perturbed predictions to the estimated bound, and the entropy of the averaged softmax output is used as the OOD score. The method is evaluated on OpenOOD v1.5 for CIFAR-10/100, ImageNet-200, and ImageNet-1K, and it reports consistent near-OOD improvements over post-hoc baselines, with additional experiments on several architectures.","tokens_in":22573,"tokens_out":11940,"duration_ms":106432,"significance":"The paper has a clear strength: it provides a self-contained theoretical framework with a proof of Eq. 5-6 in the appendix, validates Eq. 5 on a synthetic infinite-width regression problem using the neural-tangents library, and evaluates on a large, standardized benchmark across multiple architectures and ID/OOD splits, including a consistency check on ViT, VGG, and RegNet. The near-OOD empirical gains, especially on CIFAR-10 and ImageNet-1K, are promising and would be a useful contribution if the connection between theory and algorithm were made rigorous. As it stands, the significance is conditional: the implemented score is not derived from the bound by a chain of equalities, and the main experimental gains are not shown to be consequences of Theorem 1.","major_comments":[{"comment":"Proposition 1 states that lim_{ε→0} ε^{-2} E_v ||f(z;θ_T+v)-f(z;θ_T)||² = Tr Θ(z,z), yet line 6 of Algorithm 1 computes eΘTr(z,z) = (1/M)Σ_i ||f(z;θ_T+v_i)-f(z;θ_T)||² without the ε^{-2} factor, so for ε=0.005 it estimates 0.005² TrΘ(z,z), not TrΘ(z,z). Line 7 estimates D ≈ εδ√o ||∇θf_T(z)(θ_T−θ_ts)||. Consequently line 8 estimates J²[ε²(TrΘ(z,z)+ΘXX) − λ εδ√o ||...||], whereas Eq. (6) has trace terms and a gradient term with coefficients independent of ε and o. The ratio of the gradient coefficient to the trace coefficient in the implementation is λδ√o/ε ≈ 6×10^4 with the reported hyperparameters (λ=1.25, δ=8, o=1000), so the D-term completely dominates the z-dependent trace term; tuning the single scalar J cannot repair this relative weighting. The claim in §4.1 that line 8 is an \"Estimation of Eq. 6 up to square root\" is therefore not supported. Please either rescale eΘTr by ε^{-2} and D so that the implemented expression matches Eq. (6) up to the stated constants, or explicitly reframe S as a heuristic score and provide evidence that it tracks the predicted bound.","section":"§4.1, Algorithm 1 lines 6-8; Proposition 1; Eq. (6)"},{"comment":"The final score U = H_y(1/M Σ softmax(f̃_i(z))) is not obtained from the bound by any stated inequality. Equation (7) only bounds the trace of the variance of the logits, and the surrogate samples are constructed to match that variance via γ. Entropy of the averaged softmax is a nonlinear function of those samples, and no monotonicity or Lipschitz argument is given to show that low or high entropy corresponds to small or large values of Eq. (6). Thus the near-OOD AUC results in Table 1 are empirical properties of the heuristic surrogate construction, not consequences of Theorem 1. A formal link, or an explicit disclaimer that the entropy step is an empirical design choice, is needed.","section":"§4.3, Algorithm 1 lines 9-13; Eq. (7)"},{"comment":"Theorem 1 is derived for the linearized dynamics of Eqs. (2)-(3) and assumes a constant NTK over training, i.e., the lazy regime. The main experiments use ResNet-18/50 trained with SGD, and Sec. 5 acknowledges that empirical deep networks are typically trained non-linearly. No quantitative evidence is provided that the trained checkpoints remain in the lazy regime, such as the relative change of the empirical NTK during training or the distance between f_emp and the linearized trajectory Eq. (2). Without such a check, the theoretical result does not cover the setting in which the method is evaluated. The paper should add such a measurement, restrict the theoretical claims accordingly, or present the theory as motivation only.","section":"§3.1 and §5"}],"minor_comments":[{"comment":"The table reports averages over three runs but no standard deviations or confidence intervals; many near-OOD differences are small (e.g., CIFAR-100 near AUROC 80.81 vs. 80.27 for ENT), so error bars are needed to support the ranking claims.","section":"Table 1"},{"comment":"The panel labeled \"Verification of A5\" appears to contain no visible curve or axes in the manuscript; please add axis labels, a legend, and a description of the plotted quantity.","section":"Figure 2(c)"},{"comment":"The notation \"≲\" already means \"up to a constant independent of z,\" so saying \"up to constants J and K\" in the same sentence is redundant and ambiguous; please clarify whether J and K are absorbed into ≲ or are explicit.","section":"Equation (6)"},{"comment":"The sentence \"we find the optimal non-negative J* by maximizing the likelihood of 1/M Σ_i softmax(f̃_i(z))\" is incomplete: the probabilistic model, the validation labels, and the loss used for maximum likelihood are not specified.","section":"§4.2"},{"comment":"\"Improved AUROC over 15%\" should be reported as percentage points with the exact numerical improvements, since an AUROC increase from 0.76 to 0.89 is 13 points, not 15 percent in relative terms.","section":"§5.1"},{"comment":"The footnote contains placeholder \"?\" markers for the pretrained model references; these should be completed before publication.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of stat.ML and the empirical study is reasonably extensive, but the central theoretical claim is currently disconnected from the implemented algorithm. The missing ε^{-2} factor in Algorithm 1 is a concrete, load-bearing issue that should be fixed or explicitly demoted to a heuristic. Releasing code would also help verify the perturbation details and the calibration procedure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TULiP is a reasonable weight-perturbation heuristic for OOD detection wrapped in a theory that doesn't quite reach its algorithm. The perturb-then-train bound (Eq. 5) is a real result—the proof under A1-A4 is plausible, the synthetic check in Fig. 2(a) is appropriate, and the authors are honest that practical networks are not in the lazy regime. On the empirical side, the OpenOOD evaluation is broad and the near-OOD numbers are consistently decent. That's the part worth keeping.\n\nThe soft spot is the bridge. The stress-test note is correct: Proposition 1 gives (1/epsilon^2)E||f(theta+v)-f(theta)||^2 -> TrTheta, so line 6's eThetaTr is an estimator of epsilon^2 TrTheta(z,z), not TrTheta(z,z). Line 7's D is approximately epsilon delta sqrt(o)||nabla_theta f(z)(theta_T - theta_ts)||. So the S in line 8 is J^2[epsilon^2(TrTheta(z,z)+Theta_XX) - lambda epsilon delta sqrt(o)||...||], whereas Eq. 6 demands coefficient 1 on the trace terms and -2K on the gradient norm. J is scalar, so it cannot correct the relative weighting; in the reported setting, epsilon/(delta sqrt(o)) is tiny, so the gradient term dominates and S is often negative. The claim that line 8 estimates Eq. 6 'up to square root' is not supported by the paper's own equations. The method then becomes a heuristic combination of Gaussian weight noise and a deterministic step in the theta_T - theta_ts direction, with lambda and Jscaling tuned on OOD validation data. That might be a fine trick, but it is not the bound.\n\nOther gaps are minor by comparison: no error bars in the main tables, no released code, the final entropy score has no theorem attached, and the improvements over ENT are a few AUROC points at several times the inference cost. These are fixable in revision.\n\nWho should read it: anyone working on post-hoc OOD detection will find the empirical recipe worth trying, and the bound is a legitimate contribution to the NTK uncertainty literature even if it does not validate the algorithm. It deserves a serious referee: the core idea is not a restatement of prior work, the experiments are reproducible in principle, and the theory-algorithm gap is precisely the kind of thing a reviewer can ask to be fixed. I would send it to review rather than desk reject, with the expectation of major revisions.","headline":"A promising weight-perturbation OOD heuristic whose stated theoretical grounding breaks at the step where Eq. 6 becomes Algorithm 1; still worth a referee.","tokens_in":23045,"tokens_out":3947,"would_cite":false,"duration_ms":33991,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"TULiP claims entropy of predictions averaged over small Gaussian weight perturbations is a better near-OOD detector than raw softmax entropy.","keywords":["uncertainty estimation","out-of-distribution detection","neural tangent kernel","weight perturbation","lazy training","post-hoc method","epistemic uncertainty","near-distribution shift"],"falsifier":"Train two identical architectures, one in the lazy regime (wide network, small learning rate) and one in the feature-learning regime (narrow network, large learning rate), and compare TULiP's near-OOD AUROC against plain softmax entropy; the paper's bound predicts a larger advantage in the lazy case, so observing the opposite would undermine the theoretical story.","tokens_in":21878,"feed_emoji":"🎯","tokens_out":7262,"duration_ms":59814,"temperature":0.7,"pith_summary":"TULiP is a post-hoc way to estimate how uncertain a trained image classifier should be about a new input, without retraining or access to the original training data. The paper's central claim is that if you slightly perturb the network's weights at test time and average the resulting predictions, the entropy of that average is a better out-of-distribution (OOD) detector than the standard softmax entropy, especially when the new input is only subtly different from the training distribution. The method is justified by a bound on how much a hypothetical pre-convergence perturbation would change the final network, derived under linearized training dynamics. The paper reports top or near-top results on standard image benchmarks, with the clearest gains on near-OOD samples.","feed_headline":"Weight-noise entropy beats plain softmax on near-OOD inputs","feed_subtitle":"A post-hoc score with a kernel-based bound: average predictions over tiny weight perturbations to flag hard distribution shifts.","key_machinery":"The central object is a hypothetical perturbation applied to the network at a time before training converges, followed by the same gradient-flow training. In the linearized (lazy) regime, the effect of that perturbation on the final predictions is bounded by the distance from the test point to the training set in the space of network gradients, which is in turn estimated by the trace of the neural tangent kernel and the norm of the Jacobian times parameter displacement. Algorithm 1 implements this by sampling Gaussian weight perturbations and averaging the resulting softmax outputs, using the calibrated bound as a variance-matching envelope for the surrogate posterior samples.","core_discovery":"The central claim is that epistemic uncertainty at a test point $z$ can be quantified by the variance of predictions produced by perturbed versions of the final network, and that this variance is upper-bounded by an expression involving the neural tangent kernel and the product of the network Jacobian with the parameter displacement during training. Concretely, Theorem 1 bounds $\\|f_T(z)-\\hat{f}_T(z)\\|$ by a term proportional to the gradient distance from $z$ to the training set plus the perturbation size and convergence error. The bound is then converted into an implementable score: sample Gaussian perturbations of the weights, estimate the trace of the neural tangent kernel via a randomized estimator, estimate the Jacobian-displacement term via a finite difference, scale the sample spread by a calibrated constant, and take the entropy of the averaged softmax outputs. The paper claims this score outperforms raw softmax entropy and several other post-hoc detectors on near-OOD inputs across CIFAR-10, CIFAR-100, ImageNet-200, and ImageNet-1K settings.","pith_inferences":["If the bound's linearization assumption is doing the work, architectures closer to the lazy regime should show a larger gain from TULiP; this is a testable prediction the paper does not make.","The calibration step uses a validation set to set one constant; one could attempt to derive that constant from layer-wise weight norms, removing the need for any extra data.","Because TULiP only transforms predictions, it could be stacked with input-level OOD methods such as temperature scaling or input preprocessing, which the paper does not explore.","For non-classification tasks, the entropy step would naturally be replaced by predictive variance or dispersion of the averaged samples; the paper states the construction generalizes but does not demonstrate it."],"forward_implications":["On near-OOD inputs, TULiP's entropy score is reported as the top or second-best AUROC across CIFAR-10, CIFAR-100, ImageNet-200, and ImageNet-1K.","The method requires only the trained model and a small validation set, not the original training data, so it can be added to already-deployed classifiers.","Averaged TULiP predictions can be fed into other scoring rules; the paper shows this pushes MSP scores further apart between in-distribution and OOD data.","TULiP's advantage is consistent across several convolutional and transformer architectures on ImageNet-1K, while some baselines degrade when the training protocol changes."],"supporting_citations":[{"why":"introduces the neural tangent kernel and the lazy-training picture on which the linearization in Eq. 2 rests.","marker":"[28]"},{"why":"shows wide networks evolve as linear models under gradient flow, supplying the dynamics in Eq. 3 and the kernel-ridge view used in Theorem 1.","marker":"[36]"},{"why":"provides the randomized trace-estimation technique used to estimate TrTheta(z,z) from weight perturbations.","marker":"[2]"},{"why":"defines the near/far OOD benchmark protocol, dataset splits, and baseline scores used in the large-scale experiments.","marker":"[56]"},{"why":"introduces maximum softmax probability as the baseline OOD score that TULiP is designed to improve via entropy.","marker":"[23]"},{"why":"motivates interpreting prediction averaging as approximate Bayesian model averaging, which TULiP's surrogate posterior mimics.","marker":"[15]"}],"fun_headline_variants":["Perturb weights, gauge uncertainty: TULiP for OOD","Theoretical bound turns weight noise into OOD score","Variance of perturbed predictions flags near-OOD","Beyond softmax entropy: weight-noise uncertainty","From NTK to entropy: TULiP's uncertainty score"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that real neural network training is well approximated by linearized (lazy) training dynamics, so the bound from Theorem 1 and the variance-matching step in the algorithm both depend on a kernel that changes little during training.","fun_headline_variants_meta":{"raw":{"variants":["Perturb weights, gauge uncertainty: TULiP for OOD","Theoretical bound turns weight noise into OOD score","Variance of perturbed predictions flags near-OOD","Beyond softmax entropy: weight-noise uncertainty","From NTK to entropy: TULiP's uncertainty score"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000489,"raw_usage":{"total_tokens":2384,"prompt_tokens":897,"completion_tokens":1487,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":1405}},"tokens_in":513,"tokens_out":1487,"duration_ms":9993,"temperature":1.0,"reasoning_tokens":1405,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:52:43.018659+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two identical architectures, one in the lazy regime (wide network, small learning rate) and one in the feature-learning regime (narrow network, large learning rate), and compare TULiP's near-OOD AUROC against plain softmax entropy; the paper's bound predicts a larger advantage in the lazy case, so observing the opposite would undermine the theoretical story.","supporting_citations":[{"cited_title":"In: NeurIPS","cited_arxiv_id":null,"evidence_quote":"introduces the neural tangent kernel and the lazy-training picture on which the linearization in Eq. 2 rests."},{"cited_title":"In: NeurIPS","cited_arxiv_id":null,"evidence_quote":"shows wide networks evolve as linear models under gradient flow, supplying the dynamics in Eq. 3 and the kernel-ridge view used in Theorem 1."},{"cited_title":"Journal of the ACM58(2), 1–34 (2011)","cited_arxiv_id":null,"evidence_quote":"provides the randomized trace-estimation technique used to estimate TrTheta(z,z) from weight perturbations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines the near/far OOD benchmark protocol, dataset splits, and baseline scores used in the large-scale experiments."},{"cited_title":"In: ICML","cited_arxiv_id":null,"evidence_quote":"motivates interpreting prediction averaging as approximate Bayesian model averaging, which TULiP's surrogate posterior mimics."}],"review_version":1}