{"id":"72f9c625-8fce-4d01-a9ba-a2da4de68e41","arxiv_id":"2602.01279","paper_version":2,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Enhances Bayesian Last Layers via NTK feature projection and uniform subsampling to provably increase posterior variances and improve empirical uncertainty calibration.","lead":"This paper improves Bayesian Last Layers by projecting subsampled Neural Tangent Kernel features onto the last-layer feature space to capture uncertainty from earlier network layers. The approach keeps inference cheap while producing larger, more accurate uncertainty estimates on regression, classification, and out-of-distribution tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Subsampling approximation bounds may not preserve the claimed variance inequality","rationale":"The reader's weakest assumption correctly flags the projection step itself. The additional and independent risk is that the subsequent uniform subsampling, while bounded, is not shown to preserve the direction of the variance inequality that constitutes the strongest claim. This is a distinct, concrete place where the guarantee could fail without contradicting the abstract.","tokens_in":1681,"tokens_out":306,"duration_ms":41426,"concrete_test":"Take the exact projection matrix P from the non-subsampled derivation; replace it by the uniform subsample estimator P_hat with subsample fraction s; recompute the predictive variance quadratic form at a test point using P_hat versus the standard last-layer features; check whether variance(P_hat) >= variance(BLL) continues to hold for s <= 0.1 on the UCI regression tasks reported in the paper.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim asserts a provable posterior variance >= standard BLL via projection of NTK features onto the last-layer span. The method then applies uniform subsampling both to estimate the projection matrix and for inference, with only separate approximation bounds derived for each. These bounds control matrix error but do not automatically guarantee that the quadratic form determining predictive variance stays above the BLL baseline once the projection is replaced by its subsampled estimate; a sufficiently large error term could reverse the inequality for finite subsample sizes even if the exact-projection case holds.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes enriching Bayesian Last Layers (BLLs) by projecting Neural Tangent Kernel (NTK) features onto the linear span of the last-layer features. This construction is shown to yield posterior variances that are provably at least as large as those of a standard BLL, thereby correcting underestimation of epistemic uncertainty while retaining the computational advantages of last-layer Bayesian inference. Uniform subsampling is introduced both to estimate the projection matrix and to perform inference, accompanied by separate approximation bounds on the resulting matrix errors. Experiments across UCI regression, contextual bandits, image classification, and out-of-distribution detection tasks report improved calibration and uncertainty estimates relative to standard BLLs and several baselines.","tokens_in":1789,"tokens_out":626,"duration_ms":25769,"significance":"If the variance inequality is preserved under the subsampled estimator, the work supplies a practical, theoretically grounded route to richer epistemic uncertainty quantification without incurring the cost of full-network Bayesian inference. The explicit derivation of approximation bounds for both projection estimation and inference, together with the empirical demonstration of improved calibration on regression, bandit, and OOD tasks, constitutes a concrete advance over existing last-layer methods.","major_comments":[{"comment":"§3 (exact-projection case) and §4 (subsampled case): The central claim that posterior variances are provably ≥ those of a standard BLL holds for the exact projection onto the last-layer span. However, the uniform subsampling used both to form the projection matrix and to evaluate the predictive variance introduces separate matrix-norm approximation bounds; these bounds do not automatically guarantee that the quadratic form determining the posterior variance remains above the BLL baseline once the exact projection is replaced by its subsampled estimate. A sufficiently large finite-sample error could reverse the inequality even when the exact case is valid.","section":"§3 and §4"},{"comment":"Abstract and §2 (method): The assumption that the projection of NTK features onto the span of the last-layer features is sufficient to capture epistemic uncertainty induced by earlier layers is stated but not accompanied by a quantitative characterization of the residual uncertainty orthogonal to that span. If this residual component is non-negligible, the claimed correction to BLL underestimation may be only partial.","section":"Abstract and §2"}],"minor_comments":[{"comment":"Notation for the subsampled projection matrix and the resulting approximate kernel should be introduced with an explicit equation number to avoid ambiguity when the approximation bounds are applied.","section":"§4"},{"comment":"The experimental tables would benefit from reporting the effective subsample size (as a fraction of the full feature dimension) alongside the reported metrics so that the computational-accuracy trade-off is immediately visible.","section":"Experiments section"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a good fit for a machine-learning theory venue; the citation pattern is appropriate and the novelty relative to prior BLL and NTK work is clearly stated."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive comments on our work. We address the major concerns point by point below, and we will make revisions to improve the clarity of the theoretical claims and limitations.","responses":[{"response":"We concur that the provable inequality is established strictly for the exact projection case analyzed in §3. For the subsampled estimators in §4, we provide matrix-norm bounds on the approximation errors for both the projection matrix estimation and the inference step. These bounds do not directly imply preservation of the variance inequality for any finite subsample size. We will revise the text in §4 to explicitly acknowledge this point and to clarify that the inequality holds exactly only in the limit of full sampling, while the subsampled version approximates it with controllable error. We will also add a note on the practical implications based on our experimental subsample sizes.","revision_made":"yes","referee_comment":"[§3 and §4] §3 (exact-projection case) and §4 (subsampled case): The central claim that posterior variances are provably ≥ those of a standard BLL holds for the exact projection onto the last-layer span. However, the uniform subsampling used both to form the projection matrix and to evaluate the predictive variance introduces separate matrix-norm approximation bounds; these bounds do not automatically guarantee that the quadratic form determining the posterior variance remains above the BLL baseline once the exact projection is replaced by its subsampled estimate. A sufficiently large finite-sample error could reverse the inequality even when the exact case is valid."},{"response":"The method projects NTK features to capture contributions from the entire network while restricting to the last-layer span for efficiency. We recognize that a quantitative analysis of the residual uncertainty in the orthogonal complement is not provided. Such a characterization would require a more detailed decomposition of the NTK and its interaction with the network architecture, which is outside the scope of the current work. We will revise §2 to include a clearer statement of this modeling assumption and its potential limitations, indicating that the approach addresses a significant portion of the epistemic uncertainty but may leave some residual unaccounted for.","revision_made":"yes","referee_comment":"[Abstract and §2] Abstract and §2 (method): The assumption that the projection of NTK features onto the span of the last-layer features is sufficient to capture epistemic uncertainty induced by earlier layers is stated but not accompanied by a quantitative characterization of the residual uncertainty orthogonal to that span. If this residual component is non-negligible, the claimed correction to BLL underestimation may be only partial."}],"tokens_in":1444,"tokens_out":550,"duration_ms":49545,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing here is that they take a standard Bayesian last layer, project subsampled NTK features onto the span of the last-layer weights, and get posterior variances that are provably at least as large as the plain BLL case. That directly targets the underestimation of epistemic uncertainty while keeping inference cheap. They then add uniform subsampling for both the projection matrix and the posterior, with separate approximation bounds for each.","headline":"NTK projection into BLLs gives a clean variance guarantee over standard last layers, but the subsampling step leaves the inequality unproven for the approximated version.","tokens_in":2292,"tokens_out":162,"would_cite":false,"duration_ms":49396,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"NTK-projection BLL enrichment uses standard linear algebra with no RS cost or ladder structure","alignment":"orthogonal","rationale":"Paper centers on projecting empirical NTK features ϕ_m onto span of last-layer features ϕ_r via least-squares matrix A, Cholesky reparameterization L, and uniform subsampling bounds (Theorems 3.3–3.5) to obtain SB ≽ S_bll. This is ordinary matrix concentration + Woodbury algebra for neural-network uncertainty; it invokes none of the RS forcing chain, J-cost functional equation, φ-ladder, 8-tick periodicity, or parameter-free constant derivations.","tokens_in":60173,"confidence":"high","tokens_out":152,"duration_ms":14256,"cache_read_input_tokens":38528,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Projecting NTK features onto last-layer features corrects underestimation of epistemic uncertainty in Bayesian last layers.","keywords":["Bayesian last layers","Neural Tangent Kernel","epistemic uncertainty","subsampling","uncertainty calibration","neural networks","posterior variance"],"falsifier":"Observing a data point where the enriched Bayesian last layer reports a strictly smaller posterior variance than the standard version would directly contradict the provable inequality; likewise, failure to observe calibration gains on standard benchmark suites would weaken the practical claim.","tokens_in":2601,"feed_emoji":"📊","tokens_out":704,"duration_ms":54243,"temperature":0.7,"pith_summary":"Standard Bayesian last layers treat only the final layer as random and therefore underestimate the epistemic uncertainty coming from the rest of the network. The authors address this by projecting Neural Tangent Kernel features of the full network onto the linear span of the last-layer features. The resulting posterior inference remains linear and cheap yet produces variances that are mathematically guaranteed to be at least as large as those of an ordinary Bayesian last layer. Uniform subsampling of the NTK features keeps the method scalable, and the paper supplies approximation error bounds for both the projection and the posterior. Experiments across regression, bandits, classification, and out-of-distribution detection show improved uncertainty calibration compared with baseline Bayesian last layers.","feed_headline":"NTK projection fixes epistemic underestimation in Bayesian last layers","feed_subtitle":"The enriched posterior variances are provably larger and yield better uncertainty estimates on regression and classification tasks.","key_machinery":"The projection of full-network NTK features onto the column space of the last-layer feature matrix; this linear map lets the Bayesian update incorporate information from all layers without leaving the cheap last-layer inference regime.","core_discovery":"By projecting subsampled NTK features onto the space spanned by the last-layer weights, the method constructs a Bayesian posterior over the last layer whose covariance reflects variability in earlier layers. The authors prove that the marginal posterior variance at any test point is always at least as large as the variance obtained by a conventional Bayesian last layer. This guarantee follows directly from the geometry of the projection: the NTK component orthogonal to the last-layer span is discarded, but the retained component still enlarges the effective prior covariance.","pith_inferences":["The same projection idea could be tested with other kernels that approximate the behavior of deep networks.","Subsampling strategies might be refined by importance sampling rather than uniform selection to further reduce variance in the estimates.","If the method generalizes, it could serve as a drop-in replacement for standard Bayesian last layers in any architecture where NTK features are computable."],"forward_implications":["Posterior variances are provably at least as large as those from a standard Bayesian last layer.","Approximation bounds hold for both the projection matrix and the resulting posterior when features are uniformly subsampled.","The enriched model shows improved calibration on UCI regression tasks and competitive performance on contextual bandit problems.","Uncertainty estimates improve on image classification and out-of-distribution detection in both image and tabular data."],"fun_headline_variants":["Subsampled NTK enlarges posteriors in Bayesian last layers","NTK projection onto last layer space accounts for earlier layers","Subsampled NTK adds earlier layer uncertainty to Bayesian posteriors","Projecting subsampled NTK features enlarges last layer variances"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The projection of NTK features onto the linear span of the last-layer features is sufficient to capture the epistemic uncertainty induced by earlier layers.","fun_headline_variants_meta":{"raw":{"variants":["Subsampled NTK enlarges posteriors in Bayesian last layers","NTK projection onto last layer space accounts for earlier layers","Subsampled NTK adds earlier layer uncertainty to Bayesian posteriors","Projecting subsampled NTK features enlarges last layer variances"]},"model":"grok-4.3","cost_usd":0.015629,"raw_usage":{"total_tokens":6689,"prompt_tokens":660,"num_sources_used":0,"completion_tokens":68,"cost_in_usd_ticks":156287000,"prompt_tokens_details":{"text_tokens":660,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":5961,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":660,"tokens_out":68,"duration_ms":83975,"temperature":1.0,"reasoning_tokens":5961,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-22T10:57:30.226259+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Observing a data point where the enriched Bayesian last layer reports a strictly smaller posterior variance than the standard version would directly contradict the provable inequality; likewise, failure to observe calibration gains on standard benchmark suites would weaken the practical claim.","supporting_citations":[],"review_version":1}