{"id":"62ed07bf-bc27-4261-b684-ff693673837a","arxiv_id":"2412.15758","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A repulsive last-layer ensemble trained with function-space diversity on OOD or augmented samples gives competitive uncertainty estimates at a fraction of deep-ensemble cost.","lead":"This paper shows that uncertainty estimates from neural networks improve if a multi-headed last layer is trained to make diverse predictions on out-of-distribution or heavily augmented images, rather than on the training data itself. The method is cheap enough to retrofit pretrained models, and it competes with much more expensive deep ensembles on several uncertainty benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The method inherits its uncertainty quality from a distance-aware frozen backbone, yet only the authors' own spectral-normalized residual checkpoints are tested; with kernel bandwidth ν unreported and no sensitivity analysis, fs-RLL-E's gains over plain LL-E are not yet robustly established.","rationale":"The paper is a genuine empirical contribution, and the reader's CONDITIONAL verdict is close to right. My independent reading of Tables II and III checked whether the reuse of repulsion-sample datasets as OOD test sets (e.g., fs-RLL-E +Cifar100 evaluated on Cifar100) inflates the headline numbers. The non-matching columns actually still beat MAP and often match DE-5: fs-RLL-E +TinyImagenet on CIFAR100 is above DE-5 on Places365, Texture, and SVHN, and Table X shows partial cross-generalization. So the overlap is a reporting flaw, not the load-bearing failure. The load-bearing assumption is the one the reader identified: the frozen base must preserve input distances so that linear heads can be repelled into diverse functions on OOD inputs. The paper's own Section V-C names feature collapse as the threat, and Appendix B.2 confirms architecture sensitivity (LeNet/VGG degrade). What is missing is proof that the claim extends to genuinely pretrained checkpoints: all experiments use backbones the authors trained with spectral normalization on the same dataset, so 'seamless integration to pretrained networks' is only shown for their own protocol. I elevate the unreported kernel bandwidth ν from a reproducibility nicety to a correctness risk: the repulsion kernel of Section IV-B is the entire mechanism, the method's effect sizes are modest, and the numerically identical RLL-E/LL-E rows in Table I demonstrate that an inactivated kernel reproduces LL-E exactly; without a ν sweep, the claimed improvements could be an artifact of scale tuning. A fixed hyperparameter and a non-author-trained backbone test would settle whether the central claim generalizes. Pending that, keeping the verdict CONDITIONAL is appropriate.","tokens_in":65,"tokens_out":14092,"duration_ms":214195,"concrete_test":"Run a controlled comparison on CIFAR10 with a frozen ResNet-50. (1) Use three frozen backbones: the authors' SN-trained Wide-ResNet-28-10; an off-the-shelf ImageNet-pretrained ResNet-50 with no spectral normalization; and the same network after standard fine-tuning on CIFAR10 without SN. Fix all fs-RLL-E settings, set ν to the median squared pairwise logit distance computed on the repulsion batch, use CIFAR100 as repulsion samples, and report AUROC for CIFAR10 vs {TinyImagenet, Texture, SVHN} for fs-RLL-E, LL-E, MAP, and DE-5. If either non-SN backbone loses the fs-RLL-E gain over LL-E, the retrospective-uncertainty claim is restricted to distance-regularized checkpoints. (2) Repeat the CIFAR10 × CIFAR100-repulsion configuration sweeping ν over {0.1, 1, 10} times the median distance and the repulsion-loss weight over {0.1, 1, 10}.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that freezing a pretrained base and repelling linear heads in function space yields uncertainty competitive with deep ensembles. The load-bearing premise, stated in Sec. V-C, is that the fixed features remain distance-aware (no feature collapse), so that linear heads can express functions that are diverse on OOD or augmented repulsion samples. Two gaps make this premise fragile.\n\nFirst, the evidence covers only backbones trained by the authors under one protocol (App. A: 300 epochs, spectral normalization). App. B.2/B.5 show dependence on architecture and that spectral normalization is irrelevant only when residual connections exist, but no experiment uses a genuine off-the-shelf checkpoint (e.g., an ImageNet-pretrained network with no SN and no fine-tuning protocol): retraining the last layer of an arbitrary deployed network is exactly the advertised use case, and its features may collapse distances, leaving linear heads unable to separate OOD. The authors' own Sec. V-C and future-work text acknowledge this limitation, but it is not quantified.\n\nSecond, the whole effect of the repulsion term is governed by the kernel k(f_i, f_j)=exp(-||f_i - f_j||^p/ν) of Sec. IV-B, and ν (and the repulsion-loss weight) are never reported. The RLL-E and LL-E rows of Table I are numerically identical across all metrics and 10 runs, which is only consistent with a parameter-space kernel that is effectively inactive; this shows the dead-ν regime exists in the paper's own tables, yet no sensitivity analysis is given for the function-space ν. Without it, the observed fs-RLL-E improvements over LL-E (e.g., Table II ECE 0.97→0.66; Table III near-OOD AUROC gains of 2-9 points) cannot be attributed to repulsion rather than to a tuned scale, and the method is not independently reproducible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes function-space repulsive last-layer ensembles (fs-RLL-E): a single network with a shared frozen base and n linear output heads, trained jointly with a function-space repulsion term evaluated on auxiliary repulsion samples (unlabeled OOD data or label-destroying augmented training data). The authors argue that (i) enforcing diversity on training-data samples is detrimental, (ii) parameter-space repulsion is insufficient and function-space diversity should be enforced instead (desideratum D1), and (iii) the multi-headed last-layer parameterization is flexible enough to serve as the particle representation, enabling retrospective uncertainty estimates for pretrained networks (D2). The method is evaluated on DirtyMNIST uncertainty decomposition, an active learning benchmark, OOD detection for CIFAR10/CIFAR100, and covariate-shift calibration on CIFAR10-C/CIFAR100-C, against MAP, DDU, SNGP, last-layer Laplace, and deep ensembles. The appendices add ablations over repulsion-sample choices, base-network architectures, spectral normalization, and feature-space dimensionality.","tokens_in":28923,"tokens_out":14210,"duration_ms":119815,"significance":"If the central claims are correct, the paper's contribution is a practically important one: a cheap post-hoc uncertainty method that achieves OOD detection and calibration competitive with deep ensembles at a fraction of the training and memory cost, plus a transferable design rule (enforce diversity on OOD or augmented samples, not on training data). The empirical work is broad and largely well-controlled: repulsion-sample ablations (Tables II and III), base-architecture sensitivity (Appendix B.2), spectral-normalization sensitivity (Table VIII), and feature-dimension scaling (Appendix B.5). The negative result for training-data repulsion (Table II, fs-RLL-E + Cifar10 with ECE 5.73 and AUROC below 50 on several OOD sets) is a clear, falsifiable finding that future work will have to engage with. However, three issues currently block full confidence: the core kernel hyperparameters are unreported, the parameter-space repulsion ablation appears to run in a dead-kernel regime, and the 'pretrained network' claim has not been tested on off-the-shelf checkpoints. These are fixable revisions rather than fundamental flaws.","major_comments":[{"comment":"The function-space repulsion kernel k(f_i, f_j) = exp(−||f_i − f_j||^p/ν) from Section IV-B is the mechanism that the paper credits for the improvements of fs-RLL-E over LL-E, but neither the bandwidth ν nor the exponent p nor any repulsion-loss weighting is reported, and Table IV (the hyperparameter summary) omits them. Since the experiments show that the repulsion term is strongly active — e.g., Table II, where replacing the repulsion-sample set changes NLL from 14.01 (Patches-32) to 19.08 (Cifar10) — the missing ν is a reproducibility blocker and not a cosmetic omission. I ask the authors to report the exact values used in every experiment and to add a sensitivity study of the main OOD AUROC and ECE numbers to ν (and, if applicable, to the repulsion weight) over at least two orders of magnitude.","section":"§IV-B; Appendix A, Table IV"},{"comment":"In Section VIII-B the authors conclude that 'imposing diversity on the parameters of the last layer particles (RLL-E)... does not provide any additional benefit,' but the supporting rows in Tables I–III are not informative as reported. In Table I the RLL-E row is numerically identical to the LL-E row in every metric and every standard deviation across 10 runs (83.53±0.16, 48.32±0.24, 1.00±0.14, 96.82±0.34, 99.41±0.22, 96.16±1.53), and in Tables II–III the two rows are nearly identical, with most differences well within one standard deviation. Identical statistics over ten runs are consistent with an effectively inactive parameter-space kernel (e.g., ν so large that the repulsion gradient vanishes) rather than with a meaningful test of parameter-space repulsion. The comparison should report ν and the repulsion weight and should demonstrate at least one regime where RLL-E visibly differs from LL-E; otherwise the argument for desideratum D1 (parameter-space diversity is insufficient) rests on a degenerate experiment.","section":"§VIII-B; Tables I–III"},{"comment":"The advertised use case — uncertainty-aware fine-tuning of pretrained networks — is only exercised on base networks trained by the authors under a single protocol (300 epochs, SGD, spectral normalization; Appendix A and Table IV). Appendix B.2 shows that this matters: on LeNet and VGG-16, which lack residual connections, the advantages of fs-RLL-E shrink (e.g., Table VI, LeNet ambig-vs-OOD AUROC of 92.68 vs 99.55 for ResNet-18), and Section V-C itself concedes the feature-collapse risk. Because the method inherits distance-awareness entirely from the frozen backbone, the paper should either add one experiment with a genuine off-the-shelf checkpoint (e.g., an ImageNet-pretrained ResNet without spectral normalization and without the authors' training protocol) or restrict the 'seamless integration to pretrained networks' claim to models trained with distance-preserving regularization. As it stands, the central practical claim is under-tested precisely in the setting it is meant to serve.","section":"§V-C; Appendix A; Appendix B.2"},{"comment":"In the CIFAR10/100 OOD experiments, the repulsion-sample set and the OOD test set coincide for several configurations (fs-RLL-E + TinyImagenet evaluated on TinyImagenet, fs-RLL-E + Texture evaluated on Texture, and likewise for Cifar100 and SVHN in Table X). Table X is helpful evidence that repulsion diversity transfers across OOD datasets, but the main tables report same-source settings without any caveat, so the reader cannot separate genuine OOD detection from detection of the repulsion training signal. Please state explicitly in Section VIII-D which OOD test sets overlap with Drep, and provide the AUROC values with those same-source columns excluded; the conclusions should rest on the cross-generalization numbers.","section":"§VIII-D; Tables II–III; Table X"}],"minor_comments":[{"comment":"The text says 'Euclidean (ℓ2 norm) or Manhattan (ℓ2 norm)', but the Manhattan distance is the ℓ1 norm; the second ℓ2 is a typo.","section":"§IV-A"},{"comment":"The ResNet-18 LL-Laplace AUROC entries are printed as 'nan' with no explanation; either report the values or add a footnote, since missing values in a comparison table block a fair reading of that row.","section":"Table VI"},{"comment":"The phrase 'enhance improve uncertainty estimates' contains a duplicated verb, and 'Section A PPENDIX B.3' has a spacing artifact; both need copy-editing.","section":"§VIII-E"},{"comment":"The legend mixes method names with 'pool (PE)' and 'pool (MI)' variants in a way that is hard to parse; clarify that the parentheticals denote the acquisition score (predictive entropy vs mutual information) and to which curve 'pool' refers.","section":"Fig. 5"},{"comment":"The sentence 'the input samples, where particle predictions are enforced to be diverse, are detrimental to the model performance' is misleading as written, since the paper's headline is that OOD and augmented repulsion samples are beneficial; rephrase to say that enforcing diversity on training-data samples is detrimental.","section":"Abstract"},{"comment":"No link to the authors' own implementation is given; Appendix A links only third-party repositories for [5], [6], and [52]. Since the method's performance hinges on ν and the repulsion-sample protocol, releasing the fs-RLL-E code (or a complete configuration file) is needed for reproducibility.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The exact equality of the LL-E and RLL-E rows in Table I over 10 runs and all metrics is unusual enough that I would double-check whether the RLL-E row is a duplicate or was run with an intentionally inactive kernel; if neither, the runs must have shared seeds, which would explain the identity but still make the row uninformative as an ablation. I would also verify, at revision time, that the authors intend to release code, since the paper currently points only to third-party implementations. The fit with the journal's scope is acceptable: the paper is an empirical methods contribution with a clearly stated design rule, even though it introduces no new theory."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a mostly careful empirical paper with a real finding: the choice of repulsion samples is what makes or breaks function-space POVI, and training-data repulsion is actively harmful while OOD data or label-destroying patch shuffling helps. Second, the headline claim—that function-space repulsion among linear heads on a frozen pretrained base gives deep-ensemble-like uncertainty at a fraction of the cost—is plausible but not yet reproducible as written, because a central hyperparameter, the kernel bandwidth nu, is never reported and there is no sensitivity analysis for it.\n\nWhat is genuinely new is the repulsion-sample story. Previous work drew repulsion samples from the training data; this paper shows that degrades uncertainty estimates, and that OOD or augmented samples fix it. That is a concrete, useful result, and it is backed by broad experiments: DirtyMNIST, CIFAR10/100, multiple base networks, active learning, covariate-shift calibration, and ablations over repulsion-sample type and architecture. The multi-headed last-layer parameterization is not deeply novel—multi-head networks exist—but the combination with function-space repulsion is sensible and the appendix work on base-network inductive biases is more thorough than usual. Credit where due: the paper ships a lot of measurement and the tables are dense with baselines.\n\nThe soft spots, in proportion. Missing nu is the biggest; without it the fs-RLL-E gains cannot be attributed to repulsion rather than tuned scale. The RLL-E rows in Tables I to III are numerically identical to LL-E, which is consistent with a dead parameter-space kernel and only makes the missing nu analysis more urgent. The stress-test note is also right that all tested backbones were trained by the authors under one protocol with spectral normalization and residual connections; the advertised use case is off-the-shelf pretrained checkpoints, and that is not tested. Some headline OOD gains use repulsion datasets as test sets (e.g., fs-RLL-E +Cifar100 evaluated on Cifar100 OOD in Table II), though the authors partly disclose this in Table X. These are fixable issues, not fatal flaws. No circularity problem: the update rule is taken from prior work and the claims are empirical measurements.\n\nWho is this for: uncertainty quantification and active learning researchers, plus practitioners who want cheap post-hoc uncertainty on frozen backbones. It deserves a serious referee. I would accept it for review and ask for the missing hyperparameter, a sensitivity sweep over nu, and at least one genuine off-the-shelf pretrained checkpoint.","headline":"Useful empirical paper on cheap last-layer ensembles; main claims are plausible but the missing kernel bandwidth and partial OOD overlap need fixing before it is independently reproducible.","tokens_in":29589,"tokens_out":1706,"would_cite":true,"duration_ms":17101,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen pretrained network plus a repulsive multi-head last layer yields deep-ensemble-grade uncertainty at a fraction of the cost.","keywords":["uncertainty quantification","function-space inference","particle optimization","last-layer ensembles","deep ensembles","out-of-distribution detection","active learning","pretrained networks"],"falsifier":"Train fs-RLL-E on a base network known to collapse features, such as LeNet or VGG-16 without residual connections on DirtyMNIST with eMNIST as repulsion samples, and measure whether epistemic uncertainty separates ambiguous MNIST from OOD data; the paper's own Appendix B.2 predicts degraded separation, so an AUROC near chance on that separation would falsify the claim that frozen feature extractors suffice.","tokens_in":28415,"feed_emoji":"🎯","tokens_out":5795,"duration_ms":45291,"temperature":0.7,"pith_summary":"This paper argues that the practical gap between function-space particle inference and deep ensembles is not the diversity principle itself but where it is applied. The key move is to freeze a pretrained base network and add a handful of linear heads, then enforce prediction diversity only on repulsion samples drawn from out-of-distribution data or label-destroying augmentations such as shuffled image patches. Diversity on the training data itself is shown to hurt both accuracy and uncertainty, whereas the right repulsion samples let a single multi-headed network approximate deep-ensemble-quality epistemic uncertainty. If the paper is right, uncertainty-aware fine-tuning of already-trained models becomes a cheap last-layer retraining step rather than a full retraining of several networks.","feed_headline":"Last-layer repulsion gives deep-ensemble uncertainty on a budget","feed_subtitle":"Retraining only the final layer with diversity on out-of-distribution data matches deep ensembles for calibration.","key_machinery":"The central object is the function-space repulsive last-layer ensemble (fs-RLL-E): one pretrained base network whose last layer is replaced by $n$ linear heads, each head a particle, with a Gaussian kernel repulsion term on the heads' prediction functions evaluated at a chosen batch of repulsion samples. The repulsion term is what drives the heads to disagree outside the training distribution, turning prediction spread into an epistemic uncertainty estimate via mutual information. The choice of repulsion samples is the second load-bearing mechanism: training data diversity underfits, so the paper uses unlabeled out-of-distribution data or patch-shuffled augmentations that destroy class labels, which suppresses spurious features and makes OOD disagreement generalize.","core_discovery":"The central claim is that a shared frozen feature extractor with a repulsive last-layer ensemble, multiple linear heads trained as particles in function space, can produce uncertainty estimates competitive with deep ensembles while adding only $(d \\times K + K) \\times n$ trainable parameters for feature dimension $d$, $K$ classes, and $n$ particles. The paper shows that function-space repulsion evaluated on unlabeled out-of-distribution data or on label-destroyed augmented samples separates aleatoric from epistemic uncertainty, detects near and far out-of-domain inputs, and calibrates predictions under covariate shift. It also claims that this works retrospectively on pretrained residual networks, provided the base representation is distance-aware enough to avoid feature collapse, so that diversity can be enforced on the last layer alone.","pith_inferences":["If the feature extractor preserves distances, fs-RLL-E could serve as a drop-in uncertainty retrofit for any pretrained residual network, including models fine-tuned for downstream tasks; the paper tests several ResNet sizes but not large language or vision-language backbones.","The choice of repulsion samples is effectively a prior over where the model should express ignorance, suggesting a design rule: point repulsion at the specific failure modes you want to catch, and expect trade-offs when one repulsion set must cover many OOD directions.","A testable extension is to make repulsion samples adaptive during training, for example by mining near-boundary or confusing OOD samples; the paper keeps the repulsion set fixed.","Since parameter-space repulsion gives no benefit over the unregularized last-layer ensemble in the paper's tables, the result implies that diversity must be measured on outputs rather than weights, a lesson that extends to ensemble distillation."],"forward_implications":["Deep-ensemble-level uncertainty becomes available on pretrained models for roughly the cost of retraining a single linear layer.","Active learning can use mutual information from the last-layer heads to pick informative points while ignoring ambiguous ones, matching deep ensembles.","OOD detection improves for both near and far shifts, particularly when repulsion samples are drawn from a related but different dataset like TinyImageNet for CIFAR models.","Calibration under corruption improves over the base network and often matches full deep ensembles.","Diversity on training data itself is counterproductive; label-destroying augmentation is a viable substitute when no OOD data is available."],"supporting_citations":[{"why":"Provides the function-space particle-optimization framework and the kernel repulsion term that the method adapts to a shared base network.","marker":"[5]"},{"why":"Establishes the particle update with attraction and repulsion and the theoretical link between repulsive ensembles and Bayesian inference.","marker":"[6]"},{"why":"Defines deep ensembles, the accuracy and uncertainty baseline that fs-RLL-E aims to match at lower cost.","marker":"[1]"},{"why":"Shows that a multi-headed network can reproduce deep-ensemble diversity, motivating the shared-base parameterization.","marker":"[9]"},{"why":"Documents failure modes of deep-ensemble epistemic uncertainty and pretrained-ensemble degradation, motivating function-space repulsion.","marker":"[10]"},{"why":"Shows that the best deep ensembles sacrifice predictive diversity, supporting the paper's shift away from training-data repulsion.","marker":"[21]"},{"why":"Shows that jointly trained ensembles can collapse due to learner collusion, motivating repulsion on non-training samples.","marker":"[22]"},{"why":"Identifies feature collapse as the obstacle to distance-aware uncertainty, the premise behind the retrospective fine-tuning claim.","marker":"[23]"},{"why":"Provides the DirtyMNIST benchmark for aleatoric/epistemic separation, the density-based baseline, and the spectral-normalization recipe used in the base networks.","marker":"[52]"}],"fun_headline_variants":["Repulsive last-layer ensembles match deep ensembles for less","Uncertainty on a budget: repulsive last-layer ensembles","Deep-ensemble uncertainty from a repulsive last layer","Last-layer repulsion delivers deep-ensemble uncertainty","Pretrained nets get uncertainty from a repulsive last layer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the frozen pretrained feature extractor keeps distant inputs distant in feature space, so that a linear head can actually produce diverse predictions on out-of-distribution or augmented points; if the base network collapses such points together, last-layer repulsion cannot separate them.","fun_headline_variants_meta":{"raw":{"variants":["Repulsive last-layer ensembles match deep ensembles for less","Uncertainty on a budget: repulsive last-layer ensembles","Deep-ensemble uncertainty from a repulsive last layer","Last-layer repulsion delivers deep-ensemble uncertainty","Pretrained nets get uncertainty from a repulsive last layer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000822,"raw_usage":{"total_tokens":3583,"prompt_tokens":920,"completion_tokens":2663,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":2585}},"tokens_in":536,"tokens_out":2663,"duration_ms":15733,"temperature":1.0,"reasoning_tokens":2585,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:06:53.520687+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train fs-RLL-E on a base network known to collapse features, such as LeNet or VGG-16 without residual connections on DirtyMNIST with eMNIST as repulsion samples, and measure whether epistemic uncertainty separates ambiguous MNIST from OOD data; the paper's own Appendix B.2 predicts degraded separation, so an AUROC near chance on that separation would falsify the claim that frozen feature extractors suffice.","supporting_citations":[{"cited_title":"Function Space Particle Optimization for Bayesian Neural Networks","cited_arxiv_id":"1902.09754","evidence_quote":"Provides the function-space particle-optimization framework and the kernel repulsion term that the method adapts to a shared base network."},{"cited_title":"Repulsive deep ensembles are Bayesian,","cited_arxiv_id":null,"evidence_quote":"Establishes the particle update with attraction and repulsion and the theoretical link between repulsive ensembles and Bayesian inference."},{"cited_title":"Quantification of uncertainty with adversarial models,","cited_arxiv_id":null,"evidence_quote":"Documents failure modes of deep-ensemble epistemic uncertainty and pretrained-ensemble degradation, motivating function-space repulsion."},{"cited_title":"The best deep ensembles sacrifice predictive diversity,","cited_arxiv_id":null,"evidence_quote":"Shows that the best deep ensembles sacrifice predictive diversity, supporting the paper's shift away from training-data repulsion."},{"cited_title":"Joint training of deep ensembles fails due to learner collusion,","cited_arxiv_id":null,"evidence_quote":"Shows that jointly trained ensembles can collapse due to learner collusion, motivating repulsion on non-training samples."},{"cited_title":"Uncertainty estimation using a single deep deterministic neural network","cited_arxiv_id":null,"evidence_quote":"Identifies feature collapse as the obstacle to distance-aware uncertainty, the premise behind the retrospective fine-tuning claim."},{"cited_title":"Deep deterministic uncertainty: A new simple baseline,","cited_arxiv_id":null,"evidence_quote":"Provides the DirtyMNIST benchmark for aleatoric/epistemic separation, the density-based baseline, and the spectral-normalization recipe used in the base networks."}],"review_version":1}