{"id":"60e63075-a0c5-45a8-ac05-409049040271","arxiv_id":"1908.03515","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"KNet jointly optimizes a deep kernel and its spectral embedding through an HSIC objective, yielding strong clustering accuracy on non-convex and real datasets and allowing out-of-sample clustering.","lead":"KNet learns a neural-network embedding and a clustering kernel at the same time, using a spectral-clustering-inspired objective. On six of seven datasets it reports higher clustering accuracy than several deep and classical baselines, and the learned embedding can be applied to new data without rerunning the full algorithm.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Setting λ=0 discards the reconstruction term that the paper calls theoretically necessary for injectivity; the choice is partly grounded in label-based NMI on Wine, so the central unsupervised clustering claim is not yet secure.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: λ=0 is used despite the reconstruction term being described as theoretically necessary for an injective, input-representative embedding. This is the point where the paper's formal support and its practical choice diverge most sharply. The manuscript itself flags the missing support: the theoretical necessity is asserted, then set aside based on empirical behavior and local-minimum intuition, and the empirical grounding of λ=0 in Wine mixes label-based NMI into an allegedly unsupervised hyperparameter choice. If λ=0 fails on new datasets, the reported gains on Moon, Spiral, Cancer, Wine, and Face may not transfer, and the method would lack a principled unsupervised selection rule. The concern is substantial enough to keep the paper at CONDITIONAL rather than ACCEPT, but not severe enough to demand rejection: the algorithm is clearly specified, the HSIC-to-spectral connection is proved in Appendix A, and the experiments are reproducible in principle. The reader's conditional verdict already captures this, so no change in verdict is needed.","tokens_in":14714,"tokens_out":8360,"duration_ms":94379,"concrete_test":"On the six paper datasets plus five held-out labeled datasets, select λ per dataset using only the unsupervised criterion suggested in Sec. 6.1 (HSIC/AE-error ratio near 1 at convergence) and also run λ=0; compute NMI only after selection and measure latent collapse as the fraction of distinct input pairs mapped to near-identical embeddings. If label-free λ selection yields NMI below spectral clustering on any dataset, or if λ=0 collapses any dataset, the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Sec. 4, after Eq. (4.9), the paper states that the autoencoder term is 'theoretically necessary to ensure that the embedding Ψ is injective,' citing Theorem 1 of [30]. The algorithm then sets λ=0 for all reported experiments (Sec. 6.1). The offered justification—that gradient descent from identity initialization makes the final embedding depend on the input—does not imply injectivity or preservation of local structure; an initialization-dependent map can still be non-injective or can distort distances in ways that break out-of-sample transfer. Sec. 6.1 says an unsupervised rule for λ is needed, but the choice λ=0 is 'grounded' in Table 6, which reports NMI (ground-truth agreement) on Wine; no label-free rule is actually used. The central claim is that KNet learns kernels and embeddings that cluster better than spectral clustering and generalize out-of-sample. If λ must be tuned with labels, or if λ=0 permits collapse on datasets where the identity/U0 initialization does not align with true clusters, then neither the unsupervised framing nor the out-of-sample generalization follows from the evidence. RCV already shows the benefit is dataset-dependent, and λ=0 remains the least formally supported part of the pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KernelNet (KNet), a deep kernel learning method for clustering. KNet parameterizes a Gaussian kernel over a neural-network embedding, and trains the network by maximizing the Hilbert-Schmidt Independence Criterion (HSIC) between the embedding matrix and a coupled spectral embedding U, while optionally including an autoencoder reconstruction penalty. The optimization alternates between stochastic gradient ascent on the network parameters and an update of U, either by eigendecomposition of the current normalized kernel Laplacian or by Cayley-based Stiefel manifold ascent. The authors initialize the encoder to approximate the identity and U to the spectral embedding of the original data. After training, k-means is run on the learned embedding, and the embedding can be applied to out-of-sample points without recomputing a spectral decomposition. The paper reports improved NMI over several deep clustering baselines, spectral clustering, and k-means on five of six evaluated datasets, and demonstrates out-of-sample performance after training on subsampled data.","tokens_in":14950,"tokens_out":6737,"duration_ms":67705,"significance":"If the central claims hold, the paper makes a useful contribution: it connects HSIC-based dependence maximization to spectral clustering, provides a deep kernel tailored to clustering, and offers a practical out-of-sample extension. The derivation in Appendix A relating HSIC maximization to the spectral embedding is correct and clearly presented. The paper also ships source code, reports mean and standard deviation over multiple runs, and includes comparisons with several recent deep clustering methods. The main weaknesses are that two load-bearing assertions are not fully supported: the claim that the method is at least as expressive as spectral clustering, and the decision to drop the reconstruction penalty despite its stated theoretical necessity for injectivity. Both issues are addressable, and the empirical evidence is otherwise reasonably strong.","major_comments":[{"comment":"The paper states that the autoencoder penalty is \"theoretically necessary to ensure that the embedding Ψ is injective\" (Sec. 4, after Eq. 4.9), yet all reported experiments set λ=0 (Sec. 6.1). The offered justification, that gradient descent from the identity initialization makes the final embedding \"representative of the input X,\" does not imply injectivity or preservation of local structure; an initialization-dependent map can still be non-injective or can distort distances in ways that break out-of-sample transfer. The empirical grounding in Table 6 uses ground-truth label agreement (NMI) on the Wine dataset and is therefore not an unsupervised selection rule. Because injectivity of the embedding is what connects the learned kernel to the cluster structure of X, the current evidence does not rule out collapse or distance distortion on other datasets, and the out-of-sample generalization claim in Sec. 4 is not secured. Please either retain the reconstruction term, provide a label-free selection rule for λ, or prove or quantify the injectivity of the final embedding.","section":"Sec. 4 (after Eq. 4.9) and Sec. 6.1 (Table 6)"},{"comment":"The claim that KNet is \"at least as expressive as spectral clustering\" is asserted but not proved. The initialization argument (Ψ as identity, U=U0) shows only that the initial feasible point of (4.8) attains the spectral-clustering objective value for the original kernel. Since the optimization then changes both Ψ and U and converges to a local maximum of the HSIC objective, it does not follow that the final embedding separates every pair of clusters that spectral clustering separates, nor that its NMI dominates spectral clustering. A formal statement and proof, or a precise definition of the claimed expressiveness, is needed; otherwise the abstract's headline claim should be weakened.","section":"Sec. 4, Eqs. (4.7)-(4.8); Abstract"},{"comment":"Spiral2 (N=30,000) is listed in Table 1 and used for the illustration in Fig. 1, but no quantitative clustering result or runtime for Spiral2 appears in Tables 2, 3, 4, or 5. The text's claim of results on \"a broad array\" of datasets and the \"5 out of 6 datasets\" statement refer to Table 2, which contains only six of the seven listed datasets. Either report Spiral2 results in the main tables or clarify its role in the evaluation; as written, the experimental record is incomplete.","section":"Table 1 and Tables 2-4"}],"minor_comments":[{"comment":"The term \"Stiefler Manifold Ascent\" is a typo for \"Stiefel Manifold Ascent.\"","section":"Sec. 5 and Algorithm 1"},{"comment":"The notation in Eq. (3.22) is inconsistent with the main text: the kernel matrix is denoted KΨθ(X) in one place and KU in another, and the relationship to the normalized kernel D^{-1/2}KD^{-1/2} from Eq. (3.2) should be made explicit.","section":"Appendix C, Eq. (3.22)"},{"comment":"There are typos in the text, including \"absense\" (Sec. 4) and \"Our choie of λ = 0\" (Sec. 6.1).","section":"Sec. 4 and Sec. 6.1"},{"comment":"The stopping criterion \"until KU has converged\" is not defined; please specify a concrete convergence condition for the outer loop.","section":"Algorithm 1, line 8"},{"comment":"The training subset sizes vary by dataset and include 75% for Wine, which is not a \"small subset\"; the text's claim of using \"as little as 6%\" should be qualified by noting the dataset-dependent percentages.","section":"Table 4"},{"comment":"The sentence \"as observed in [30]\" appears to attribute an empirical observation about clustering autoencoders to a paper on MMD GANs; please either cite a directly relevant source or rephrase to avoid misattribution.","section":"Sec. 4, reference [30]"}],"recommendation":"major_revision","confidential_remarks":"The paper's core HSIC-to-spectral derivation is sound and the empirical comparisons are useful, but the two central claims—expressiveness relative to spectral clustering and the safety of setting λ=0—need substantially more support before publication. The missing Spiral2 results should also be resolved. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this one. KNet learns a deep kernel and its spectral embedding simultaneously by maximizing HSIC, which is a real departure from Niu et al.'s linear subspace and from SpectralNet's fixed similarity matrix. The Appendix A derivation of the HSIC-to-spectral equivalence is sound, the Stiefel-manifold update is a sensible way to avoid repeated eigendecompositions, and the experiments actually deliver: on Moon and Spiral1 it hits NMI 100, on Wine it beats spectral clustering by 3 points, and the out-of-sample results with only 1–35% training data degrade at most 3%. Those are concrete wins, not artifacts.\n\nThe soft spot is exactly the one you flagged. In Sec. 4, after Eq. 4.9, the paper says the autoencoder term is “theoretically necessary to ensure that the embedding Ψ is injective,” then it sets λ=0 for every reported experiment. The stated justification—that gradient descent from identity initialization makes the final embedding depend on the input—does not imply injectivity or preservation of local structure. And the “grounding” in Table 6 uses label-based NMI on Wine, which undercuts the unsupervised framing. That said, the stress-test may be slightly over-weighted: Appendix B shows λ=0 also gives NMI 1 on Spiral1, and the pattern across datasets is consistent. The gap is real but it is a gap in justification, not a demonstrated collapse. A serious revision could close it by proving or at least systematically testing injectivity under λ=0, or by proposing a label-free rule for λ.\n\nThe “at least as expressive as spectral clustering” claim is also looser than what is proven—it is asserted from initialization rather than derived. That is a minor overclaim, not a fatal one. Reproducibility would improve with a commit hash and the missing Stiefel step size τ, but the code link and experimental detail are otherwise adequate.\n\nBottom line: the core method is worth taking seriously, the experiments are informative, and the weaknesses are addressable. This deserves a proper peer review, not a desk reject. I would send it out and ask the authors to fix the λ=0 justification and temper the expressiveness claim.","headline":"KNet is a genuine step forward in deep clustering—jointly learning a kernel and spectral embedding via HSIC works well on non-convex data—but the λ=0 choice is the soft spot that needs a serious fix before publication.","tokens_in":15497,"tokens_out":1824,"would_cite":true,"duration_ms":20694,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T10","68T07","62H30"],"pacs":[],"model":"deepseek-v4-flash","headline":"A neural embedding trained jointly with a spectral target learns clustering kernels that beat spectral clustering and deep-clustering baselines on most tested datasets.","keywords":["deep kernel learning","clustering","Hilbert-Schmidt Independence Criterion","spectral clustering","neural network embedding","Stiefel manifold","out-of-sample generalization","non-convex clusters"],"falsifier":"Train KNet with $\\lambda = 0$ on a dataset where spectral clustering is known to succeed, then check whether distinct inputs from different clusters map to nearly the same point while the initial spectral embedding separates them; a collapse of that kind would violate the claim that the trained embedding is at least as expressive as spectral clustering.","tokens_in":1615,"feed_emoji":"📊","tokens_out":1709,"duration_ms":64014,"temperature":0.7,"pith_summary":"The paper tries to show that a clustering kernel can be learned from data instead of being chosen by hand. KNet trains a neural-network embedding so that a Gaussian kernel evaluated on the embedded points maximizes the Hilbert-Schmidt Independence Criterion (HSIC) against a coupled spectral embedding, initialized at the spectral embedding of the raw data. The joint training makes the learned embedding at least as expressive as spectral clustering while adding out-of-sample generalization and faster updates. Across three synthetic and four real datasets, the paper reports higher normalized mutual information than several deep clustering baselines and spectral clustering on all but one dataset, with the largest gains on non-convex clusters.","feed_headline":"A learned kernel that clusters better than spectral methods","feed_subtitle":"KNet jointly trains an embedding and a spectral target via HSIC, then generalizes to new data from a small sample.","key_machinery":"The engine is the HSIC objective $H(\\Psi_\\theta(X), U) = \\frac{1}{(N-1)^2}\\operatorname{tr}(\\tilde K_{\\Psi} H U U^T H)$, where $\\tilde K_{\\Psi}$ is the degree-normalized Gaussian kernel on the embedded points and $H$ is the centering matrix. Holding the embedding fixed, maximizing over orthonormal $U$ recovers the top eigenvectors of the normalized Laplacian $L = H D^{-1/2} K_{\\Psi} D^{-1/2} H$, which is exactly the spectral clustering step; learning $\\Psi$ and $U$ together generalizes that step. Rewriting the objective as $\\sum_{i,j} \\Gamma_{i,j} e^{-\\|\\psi_\\theta(x_i)-\\psi_\\theta(x_j)\\|^2/2\\sigma^2}$ shows that positive entries of $\\Gamma$ attract embedded points and negative entries repel them, producing convex cluster images. Updates for $U$ are done either by eigendecomposition or by a Cayley-transform ascent on the Stiefel manifold that maintains orthonormality without a full matrix inversion.","core_discovery":"The central claim is that jointly maximizing HSIC between a neural-network embedding $\\Psi_\\theta(X)$ and an orthonormal target $U$, with $U$ initialized to the spectral embedding, discovers a deep kernel that is at least as expressive as spectral clustering and clusters better in practice. The embedding and the spectral target are learned together, so each improves the other: the kernel becomes more informative, producing nearly block-diagonal similarity matrices, and the images of non-convex clusters become convex and linearly separable. Because the embedding is a neural network, it can be applied to new samples directly, and training on a small subset of the data clusters the full dataset with little loss of quality.","pith_inferences":["One implication the paper leaves implicit is that the coupled $U$ update could be made streaming or mini-batch based, since the Stiefel ascent updates orthogonality without eigendecomposition, suggesting an online clustering variant.","A testable extension is to replace the spectral initialization with a more robust affinity embedding on datasets where spectral clustering itself struggles; KNet may inherit spectral clustering's failure modes, such as highly imbalanced cluster sizes.","The same HSIC-plus-coupled-target objective could apply beyond clustering, for example to learn embeddings that preserve graph or manifold structure in semi-supervised settings, though the paper does not claim this."],"forward_implications":["The learned kernel can be applied to out-of-sample data directly, so clustering a new point does not require rerunning spectral clustering on the combined dataset.","Training the embedding on as little as 1 to 35 percent of the data clusters the full dataset with only 0 to 3 percent degradation in normalized mutual information on the tested datasets.","The Stiefel-manifold update avoids repeated eigendecompositions, with complexity $O(N^2 c + c^3)$ per iteration when the number of clusters $c$ is much smaller than the sample count $N$.","On non-convex clusters such as spirals and moons, the embedding turns the clusters into convex, linearly separable shapes, so simple $k$-means on the embedding succeeds where $k$-means on the raw data fails.","Because the identity-map embedding reduces the joint problem to standard spectral clustering, the framework is at least as expressive as spectral clustering by construction."],"supporting_citations":[{"why":"Defines HSIC and its empirical estimator, the dependence measure the training objective is built on.","marker":"[1]"},{"why":"Establishes that maximizing HSIC with a linear kernel on the target recovers the spectral embedding, the key motivation for the objective.","marker":"[13]"},{"why":"Provides the classic spectral clustering algorithm that serves as both the initialization reference and a main comparison baseline.","marker":"[26]"},{"why":"Describes the nearest prior neural spectral-clustering approach, which KNet extends by learning the kernel and embedding jointly rather than fixing the similarity matrix.","marker":"[8]"},{"why":"Supplies the Cayley-transform Stiefel manifold ascent used to update the orthonormal embedding target efficiently.","marker":"[19]"},{"why":"States the theorem that the autoencoder reconstruction term guarantees an injective embedding, which the paper cites when arguing that setting its weight to zero still works in practice.","marker":"[30]"},{"why":"Provides a deep clustering baseline combining an autoencoder with soft cluster assignments, used in the experimental comparison.","marker":"[5]"}],"fun_headline_variants":["Deep kernel that beats spectral clustering","HSIC-trained embedding outperforms spectral methods","Fast neural kernel clustering with out-of-sample reach","Stiefel-optimized deep kernel for better clustering","Joint deep embedding and spectral target for clustering"],"cache_read_input_tokens":17664,"weakest_assumption_plain":"The method assumes that setting the reconstruction penalty to zero ($\\lambda = 0$) does not let the learned embedding collapse onto a non-injective map, even though the paper says that penalty is theoretically necessary for injectivity; if a $\\lambda = 0$ embedding stops being faithful to the input on some dataset, the learned kernel could lose cluster structure.","fun_headline_variants_meta":{"raw":{"variants":["Deep kernel that beats spectral clustering","HSIC-trained embedding outperforms spectral methods","Fast neural kernel clustering with out-of-sample reach","Stiefel-optimized deep kernel for better clustering","Joint deep embedding and spectral target for clustering"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000625,"raw_usage":{"total_tokens":2811,"prompt_tokens":779,"completion_tokens":2032,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":395,"completion_tokens_details":{"reasoning_tokens":1964}},"tokens_in":395,"tokens_out":2032,"duration_ms":15988,"temperature":1.0,"reasoning_tokens":1964,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:10:53.790548+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train KNet with $\\lambda = 0$ on a dataset where spectral clustering is known to succeed, then check whether distinct inputs from different clusters map to nearly the same point while the initial spectral embedding separates them; a collapse of that kind would violate the claim that the trained embedding is at least as expressive as spectral clustering.","supporting_citations":[{"cited_title":"Gretton, O","cited_arxiv_id":null,"evidence_quote":"Defines HSIC and its empirical estimator, the dependence measure the training objective is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that maximizing HSIC with a linear kernel on the target recovers the spectral embedding, the key motivation for the objective."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the classic spectral clustering algorithm that serves as both the initialization reference and a main comparison baseline."},{"cited_title":"Shaham, K","cited_arxiv_id":null,"evidence_quote":"Describes the nearest prior neural spectral-clustering approach, which KNet extends by learning the kernel and embedding jointly rather than fixing the similarity matrix."},{"cited_title":"Wen and W","cited_arxiv_id":null,"evidence_quote":"Supplies the Cayley-transform Stiefel manifold ascent used to update the orthonormal embedding target efficiently."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"States the theorem that the autoencoder reconstruction term guarantees an injective embedding, which the paper cites when arguing that setting its weight to zero still works in practice."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a deep clustering baseline combining an autoencoder with soft cluster assignments, used in the experimental comparison."}],"review_version":1}