{"id":"42dd8665-01e2-4ffc-a0b2-c4ef4c914804","arxiv_id":"2505.15284","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Out-of-distribution images are detected by the reconstruction error of a Cosine-Gaussian kernel PCA subspace learned from in-distribution features.","lead":"This paper detects unusual images by measuring how well they fit into a nonlinear subspace learned from normal training images. It uses a Cosine-Gaussian kernel for the subspace and two fast approximation methods to make the approach practical on ImageNet-scale data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim not supported by clean evaluation protocol: hyperparameters appear tuned on the same OoD test sets and all tables are single-run without error bars.","rationale":"The reader's weakest assumption is the subspace premise, but the truly load-bearing support for the central SOTA claim is the empirical evaluation. The paper's protocol lacks a validation split and error bars, making the headline numbers unverifiable; this is partly anticipated in the reader's rationale but is more severe than the subspace assumption. Separately, Proposition 1 is mathematically suspect: the standard KPCA residual error requires centered kernel evaluations and eigenvalue normalization, roughly ||Lambda_p^{-1/2} U_p^T k_c||^2, not ||U_p^T k_x||^2 as stated, so Figure 6's exact-KPCA comparison is not trustworthy. The main algorithm does not rely on Proposition 1, so I treat that as a secondary issue. The released code is a genuine positive and makes the proposed held-out test feasible. Since the reader already issued a conditional verdict and my concern strengthens the conditions rather than changing the overall assessment, the verdict remains unchanged.","tokens_in":27492,"tokens_out":11646,"duration_ms":105737,"concrete_test":"Run the released code under a held-out protocol: split each of the four OoD test sets into two halves, tune gamma, q, M_r, M_n (and the KPCArff+KPCAnys fusion rule, which is not defined in the paper) on the first half only, then evaluate on the second half across 10 RFF/Nystrom seeds, reporting mean plus or minus standard deviation. If the held-out average FPR/AUROC differs from Tables 1-3 by more than about 2 points, or overlaps the best baseline, the SOTA claim fails this check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the central claim of new SOTA OoD detection results (Section 5.2.3), the paper depends entirely on the ImageNet-1K tables. The reported evaluation protocol does not establish those numbers as fair. Section 5.1 describes no validation split; Sections 5.6 and Figures 8-10 report sensitivity of gamma, the explained-variance ratio (q), M_r, and M_n on the very same four OoD test sets (iNaturalist, SUN, Places, Textures) used to build Tables 1-3. Section 7 itself concedes that the method requires careful parameter tuning, but the paper never states how the final values were chosen or whether OoD labels were consulted. The sensitivity curves are steep: for KPCAnys, average FPR moves by tens of points across the tested gamma range in Figure 9 and by several points across M_n in Figure 10, so selecting a favorable configuration on the evaluation data could easily manufacture the reported margin. In addition, KPCArff depends on random omega,u draws and KPCAnys depends on landmark selection, yet every table reports one run with no error bars. The claimed SOTA is therefore a single untested configuration on the test OoD sets, and the central claim is not robustly established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes an out-of-distribution (OoD) detection method based on Kernel PCA with a composite Cosine-Gaussian kernel. The kernel is motivated by two empirically observed patterns in deep features: an adverse imbalance of feature norms that is mitigated by cosine normalization, and a beneficial l2-distance relation that is captured by a Gaussian kernel on the normalized features. To scale KPCA to large training sets such as ImageNet-1K, the authors approximate the Gaussian part with random Fourier features or a Nyström map, and for the Nyström approximation they select low-energy InD training samples as landmarks. The detection score is the KPCA reconstruction error in the approximate feature space. Experiments on ImageNet-1K with ResNet50, MobileNetV2, and ViT-B/16 report improved detection accuracy and lower inference cost compared with KNN and other baselines.","tokens_in":27719,"tokens_out":7681,"duration_ms":70507,"significance":"If the reported results hold, the paper makes a practical contribution: an OoD detector with O(M_r) or O(M_n) inference cost, a data-dependent landmark selection scheme, and a clear algorithmic pipeline with publicly released code. The complexity analysis and the kernel ablations are useful empirical additions. However, the central SOTA claim is not yet supported by the evaluation protocol as reported, because model selection appears to use the same OoD test sets on which performance is measured and because no variance over runs is reported. In addition, Proposition 1, which underpins the comparison between exact and approximate KPCA, appears incorrect as stated. These issues are locally fixable, but they require re-running or re-reporting experiments under a cleaner protocol before the main claim can be taken at face value.","major_comments":[{"comment":"The evaluation protocol does not separate model selection from evaluation. Section 5.1 describes no validation split, and Section 5.6 reports sensitivity of the Gaussian width gamma, the explained-variance ratio (q), M_r, and M_n averaged over the same four OoD test sets (iNaturalist, SUN, Places, Textures) used in Tables 1–3. Since Section 7 concedes that 'careful parameter tuning' is required, and the sensitivity curves in Figures 8–10 are steep (e.g., KPCArff FPR varies by roughly 30 points across the tested gamma range in Figure 9), selecting a favorable configuration on the evaluation OoD sets could manufacture the reported margins. The final hyperparameter values for gamma and the explained-variance ratio are not stated for Tables 1–3, and all tables report a single run even though KPCArff depends on random omega,u draws. Please report the exact model-selection protocol, add a validation split that does not use OoD labels, and provide means and standard deviations over multiple runs.","section":"§5.1, §5.6, Tables 1–3"},{"comment":"Proposition 1 as stated is not a correct formula for the exact KPCA reconstruction error. In centered KPCA, the projection coordinate onto principal component j is proportional to (1/sqrt(lambda_j)) U_j^T k_c(z), where k_c is the centered kernel vector, and the residual squared error after keeping q components is ||Phi_c(z)||^2 minus the sum of the retained projection energies, i.e., sum_{j=q+1}^N (U_j^T k_c(z))^2 / lambda_j (up to the centering terms). Eq. (10) omits the eigenvalue normalization, the kernel centering, and the norm of the centered test point. The appendix proof only re-derives the explicit-feature-space identity in Eq. (13) and does not bridge to the kernel-space statement in Proposition 1. Because Figure 6 is computed with this formula, the claimed approximation performance of RFF and Nyström relative to exact KPCA is not established by the current derivation.","section":"§4.4, Proposition 1, Eq. (10), Appendix A"}],"minor_comments":[{"comment":"Several entries appear to have missing separators, such as '53.8287.25' and '36.9791.31', which should be corrected to separate FPR and AUROC values.","section":"Table 2"},{"comment":"'KPCnys' is a typo for 'KPCAnys' in the first paragraph of Appendix B.1.","section":"Appendix B.1"},{"comment":"The phrase 'the learned subspace well seizes the InD-OoD disparities' is awkward; 'captures' or 'characterizes' would be clearer.","section":"§5.2.3"},{"comment":"For each sensitivity curve, the values of the fixed hyperparameters are not reported; please state them so that the reader can reproduce the figures.","section":"§5.6, Figures 8–10"},{"comment":"The comparison between exact and approximate KPCA is shown without error bars or repeated-seed details, so the claim of a 'marginal gap' is not quantitatively supported.","section":"Figure 6"},{"comment":"The notation '\\tildeΛ- 1 2' in line 10 is hard to read; please typeset it as \\tilde{\\Lambda}^{-1/2}.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an extension of the authors' NeurIPS 2024 paper, with the main incremental contribution being the energy-based Nyström sampling. The kernel and approximation ideas are reasonable, but the SOTA claim is not yet established because of the evaluation-protocol issues described in the major comments. The Proposition 1 problem is serious because it affects Section 4.4 and Figure 6; if the formula cannot be corrected, that section should be revised or removed. The paper otherwise fits the scope of cs.LG and the code-release policy is a strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: this is a solid extension of the authors' NeurIPS 2024 KPCA-for-OOD paper, and the genuinely new piece is the energy-based Nyström landmark selection. But the paper's \"new SOTA\" claim is not yet supported by the evaluation as reported.\n\nWhat's good. The algorithm is standard and correct: explicit feature maps (random Fourier features or Nyström) followed by PCA on the mapped features. The Cosine-Gaussian kernel has a plausible heuristic motivation from norm imbalance and ℓ2-distance relations, and the ablations in Table 5 do show the cosine part is necessary and the Gaussian part helps. The computational gain over KNN is real and well-documented: 0.2 ms inference and <100 MiB memory on ImageNet-1K, versus 15 ms and 20 GiB for the KNN baseline. Code is released, which makes the method reproducible.\n\nWhere the soft spots are. The evaluation protocol is the load-bearing issue. Section 5.6 reports sensitivity analyses for γ, q, M_r, M_n on the very same four OOD test sets used to build Tables 1-3, and the curves are steep—FPR moves by tens of points across the γ range for KPCAnys. The paper never states how the final hyperparameters were chosen or whether OOD labels were consulted. Section 7 concedes the method requires careful parameter tuning. That means the reported margin over baselines could, in the worst case, be the result of selecting a favorable configuration on the test set. Additionally, every table is single-run with no error bars, even though KPCArff depends on random ω,u draws and KPCAnys on landmark selection. A clean evaluation would use a validation split or a separate hyperparameter selection procedure, and report multiple seeds.\n\nThe other issue is minor: Proposition 1 is under-derived. The appendix proof works in the explicit mapped space and then jumps to the kernel matrix without spelling out kernel centering or eigenvalue normalization. The main algorithm does not depend on this proposition, so it is fixable, but as written it is not a rigorous derivation.\n\nWho this is for. OOD detection researchers who want an efficient post-hoc detector that works with off-the-shelf features. With a revised evaluation protocol, the method would be a useful addition to the toolbox; as it stands, the central SOTA claim is not robustly established.\n\nI would send this to peer review, because the work is substantive and the evaluation flaw is fixable. But a referee should demand a clean protocol before the SOTA claim is accepted.","headline":"A useful, well-engineered extension of the authors' NeurIPS KPCA-OOD work, but the 'new SOTA' claim is not yet supported by a clean evaluation protocol.","tokens_in":28297,"tokens_out":2679,"would_cite":true,"duration_ms":22955,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Kernel PCA with a Cosine-Gaussian kernel detects out-of-distribution inputs by how poorly they reconstruct in a learned non-linear feature subspace, at inference cost independent of training-set size.","keywords":["out-of-distribution detection","kernel PCA","Cosine-Gaussian kernel","random Fourier features","Nyström approximation","reconstruction error","feature subspace","energy-based sampling"],"falsifier":"Take a trained ImageNet-1K model, build the KPCAnys subspace from InD features, and generate OoD test samples by adding perturbations to InD features that stay within the learned non-linear principal subspace, for example by moving along the top eigenvectors in the mapped space. If the reconstruction-error AUROC on such in-manifold OoD samples stays near 50%, the core premise that OoD features fall outside the InD subspace is falsified; if it remains high, the method's sensitivity to distribution shift is confirmed.","tokens_in":27271,"feed_emoji":"🔍","tokens_out":7561,"duration_ms":61919,"temperature":0.7,"pith_summary":"The paper claims that out-of-distribution (OoD) detection can be recast as a non-linear subspace problem: in-distribution (InD) features from a trained network are assumed to concentrate in a low-dimensional non-linear subspace, and OoD features are assumed to fall outside it. Using Kernel PCA, the paper constructs that subspace from InD features and scores new samples by reconstruction error. To make the subspace discriminative, it derives a Cosine-Gaussian kernel: cosine normalization removes the harmful norm imbalance between InD and OoD features, and the Gaussian term preserves the $\\ell^2$ distance structure that separates them. To make KPCA feasible at ImageNet scale, it approximates the kernel with random Fourier features and a data-dependent Nyström mapping whose landmarks are low-confidence InD samples. If correct, this gives a detection method that matches or beats existing feature, logit, and gradient baselines while keeping inference cost independent of training-set size.","feed_headline":"Kernel trick spots out-of-distribution data at ImageNet scale","feed_subtitle":"A Cosine-Gaussian KPCA reconstruction error beats existing detectors while cutting memory to under 100 MB.","key_machinery":"The load-bearing object is the Cosine-Gaussian kernel $k(z_1,z_2)=k_{\\mathrm{gau}}(\\phi_{\\cos}(z_1),\\phi_{\\cos}(z_2))$, where $\\phi_{\\cos}(z)=z/\\|z\\|_2$ is the exact cosine map and $k_{\\mathrm{gau}}$ is the Gaussian kernel $e^{-\\gamma\\|z_1-z_2\\|_2^2}$. This composite kernel is the mechanism: it first projects features onto the unit sphere, neutralizing the norm gap that breaks PCA's centering step, then measures $\\ell^2$ proximity in that normalized space, capturing the distance relation that separates InD and OoD. The detection score is the KPCA reconstruction error in the mapped space; the paper approximates the infinite-dimensional Gaussian part by explicit mappings $\\Phi=\\phi_{\\mathrm{rff}}\\circ\\phi_{\\cos}$ or $\\Phi=\\phi_{\\mathrm{nys}}\\circ\\phi_{\\cos}$, with the Nyström landmarks chosen by lowest energy $E(x;f)=T\\log\\sum_i e^{f_i(x)/T}$, so the principal components anchor near the InD-OoD boundary.","core_discovery":"The central claim is that a non-linear feature subspace learned by KPCA, with a composite Cosine-Gaussian kernel, yields reconstruction errors that separate InD from OoD features more sharply than linear PCA or distance-based scores. The paper identifies two non-linear patterns behind the disparity: imbalanced feature norms, which cosine normalization corrects before centering; and a beneficial $\\ell^2$-distance relation in the normalized space, which the Gaussian kernel preserves. Approximating the Gaussian part with random Fourier features or, better, a Nyström mapping built from low-energy InD support landmarks keeps the computation cheap and even sharpens the subspace around the InD-OoD boundary. On ImageNet-1K with ResNet50 and ViT, the method reports state-of-the-art FPR and AUROC, including when combined with feature rectification.","pith_inferences":["The energy-based landmark selection is effectively a cheap margin-aware subspace learner; one could test whether other uncertainty proxies, such as predictive entropy or gradient norms, sharpen the InD-OoD boundary even further.","Because the kernel is built from generic feature geometry rather than image-specific structure, the same Cosine-Gaussian KPCA recipe may transfer to anomaly detection or open-set recognition on non-image modalities where norm imbalance and $\\ell^2$ distance patterns appear.","A stress test the paper does not run is to draw OoD samples from the same coarse categories as InD or from a second fine-tuned model; those are the cases where the premise that OoD features lie outside the InD subspace is least favorable.","The method can be read as a kernelized version of the nearest-neighbour baseline, with the gains coming from replacing hard nearest-neighbour distances by subspace projection; this interpretation predicts that kernel bandwidth and subspace dimension interact the way the sensitivity analysis shows."],"forward_implications":["OoD detection can be performed without storing or scanning the full training set: inference costs $O(M_r)$ or $O(M_n)$ with $M_r=4096$ and $M_n=2048$, versus $O(N_{tr})$ for nearest-neighbour methods, and memory drops from roughly 20 GiB to under 100 MiB on ImageNet-1K.","The same two-pattern kernel recipe, normalize norms then preserve $\\ell^2$ distance, is shown to beat alternative kernels, since Laplacian, polynomial, and plain Gaussian kernels all hurt detection without the cosine step.","Data-dependent kernel approximation can outperform data-independent approximation for detection: energy-selected landmarks beat uniform and high-energy landmarks, and 1024 low-energy landmarks outperform 4096 random Fourier features.","KPCA reconstruction errors compose with feature rectification, so the subspace score can be stacked with methods like ReAct to improve detection further.","An ensemble of the random-Fourier and Nyström variants reaches the best reported AUROC and FPR on the ViT-B/16 benchmark among the compared methods."],"supporting_citations":[{"why":"Establishes KPCA as a kernel eigenvalue problem and underlies the reconstruction-error formulation used throughout.","marker":"[28]"},{"why":"Introduces random Fourier features, the data-independent approximation used for the Gaussian component.","marker":"[32]"},{"why":"Introduces the Nyström method, the data-dependent low-rank approximation adapted here.","marker":"[33]"},{"why":"Documents the feature-norm imbalance and the $\\ell^2$-distance separability on normalized features, motivating the cosine and Gaussian components; also serves as the main nearest-neighbour baseline.","marker":"[12]"},{"why":"Shows PCA-based reconstruction errors for OoD detection and provides the fusion recipe with feature rectification used in the comparisons.","marker":"[13]"},{"why":"Defines the energy score used to pick low-energy InD landmarks for the Nyström sampling.","marker":"[7]"},{"why":"Is the conference predecessor that introduces the KPCA detection framework and Proposition 1 for exact kernel reconstruction errors.","marker":"[34]"},{"why":"Supplies ReAct feature rectification, which the paper combines with KPCA reconstruction errors for added gains.","marker":"[11]"}],"fun_headline_variants":["Cosine-Gaussian KPCA sharpens OOD detection","Efficient KPCA finds out-of-distribution data","Nonlinear subspace reveals OOD patterns","Kernel choice boosts KPCA OOD accuracy","Reconstruction error catches OOD samples"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes in-distribution features intrinsically lie in a low-dimensional non-linear subspace that out-of-distribution features do not share; if OoD features also sit close to that learned subspace, reconstruction errors will not separate the two distributions.","fun_headline_variants_meta":{"raw":{"variants":["Cosine-Gaussian KPCA sharpens OOD detection","Efficient KPCA finds out-of-distribution data","Nonlinear subspace reveals OOD patterns","Kernel choice boosts KPCA OOD accuracy","Reconstruction error catches OOD samples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00046,"raw_usage":{"total_tokens":2354,"prompt_tokens":1048,"completion_tokens":1306,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":1237}},"tokens_in":664,"tokens_out":1306,"duration_ms":10178,"temperature":1.0,"reasoning_tokens":1237,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:20:26.467979+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained ImageNet-1K model, build the KPCAnys subspace from InD features, and generate OoD test samples by adding perturbations to InD features that stay within the learned non-linear principal subspace, for example by moving along the top eigenvectors in the mapped space. If the reconstruction-error AUROC on such in-manifold OoD samples stays near 50%, the core premise that OoD features fall outside the InD subspace is falsified; if it remains high, the method's sensitivity to distribution shift is confirmed.","supporting_citations":[{"cited_title":"Using the nyström method to speed up kernel machines","cited_arxiv_id":null,"evidence_quote":"Introduces the Nyström method, the data-dependent low-rank approximation adapted here."},{"cited_title":"Kernel pca for out-of- distribution detection","cited_arxiv_id":null,"evidence_quote":"Is the conference predecessor that introduces the KPCA detection framework and Proposition 1 for exact kernel reconstruction errors."}],"review_version":1}