{"id":"086b822f-f226-4b98-8eca-9a1630e1af72","arxiv_id":"2506.06231","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SPEC finds and aligns the sample clusters that two embedding models capture differently by analyzing the eigenvectors of the difference of their kernel matrices.","lead":"Researchers introduce SPEC, a method that compares two AI embedding models by examining which groups of data points each model clusters together, and highlights the groups they disagree on. It also provides an optimization step that aligns one model's clustering to another's, with experiments on large image and text datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1 does not establish the principal-eigenvector interpretation; an additional, unverified spectral gap condition is required.","rationale":"The reader identifies the unverifiable spectral-separability conditions as the weakest assumption. I agree with the spirit but sharpen it: even when Conditions 1 and 2 hold, the principal-eigenvector interpretation requires an additional spectral gap that is not implied by those conditions and is never checked. This is a direct logical gap in the central claim, not merely a verification gap. It is load-bearing because if the gap fails or the top eigenvalue is degenerate, the method's output can be arbitrary mixtures that do not correspond to any interpretable cluster. The paper's own concluding caveat that 'the SPEC approach operates based on the assumption that the differently clustered samples can be detected by the spectral method' is honest but does not repair the mismatch between Section 4's unconditional statement and the conditional theorem. Secondary issues noted by the reader—the m=2000 RFF setting that the appendix shows to be unreliable, and inconsistent accuracy numbers—are real but concern experimental support rather than the theory's logical structure. My proposed test would settle whether the theoretical gap is practically severe. The reader's verdict of CONDITIONAL remains appropriate; I do not move it, hence UNCHANGED.","tokens_in":33025,"tokens_out":13671,"duration_ms":136561,"concrete_test":"Run SPEC on a synthetic dataset with two equal-size, identically structured blocks that are strongly clustered by ψ1 and weakly clustered by ψ2, forcing an approximately degenerate top eigenvalue. Inspect whether the two leading eigenvectors are rotations mixing the two block indicators; if they are, the individual-eigenvector interpretation fails. Additionally, on a real dataset, for each reported cluster I, compute λ_top − λ_max(Λ[I^c, I^c]) and compare with the Corollary 1 bound; if the gap is not positive, or if the bound is violated, the claimed cluster interpretation is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that principal eigenvectors of Λ = (1/n)(K_ψ1 − K_ψ2) identify sample groups clustered by ψ1 but not ψ2 is only supported by Theorem 1 under an implicit gap condition that the paper neither states as a formal assumption nor checks empirically. Theorem 1 bounds a sum over all eigenvectors of weighted residuals, but the stated interpretation requires Corollary 1's condition that the eigenvalue λ of interest clears λ_max(Λ[I^c, I^c]) by a positive gap γ. Conditions 1 and 2 alone do not guarantee such a gap: Λ[I^c, I^c] can have its own large positive eigenvalues if the complementary set contains differentially clustered structures, and the top eigenvalue of Λ may then belong to an eigenvector supported substantially on I^c. Moreover, when the top eigenvalue is degenerate—common when two or more differentially clustered groups have identical internal structure—an eigensolver returns an arbitrary orthonormal basis of the eigenspace, and individual eigenvectors can be mixtures of different cluster indicators. Thresholding such a mixture does not recover a meaningful cluster. The paper's conclusion acknowledges a spectral-separability assumption informally, but Section 4 states the interpretation unconditionally, making the claim stronger than the proven theorem.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Spectral Pairwise Embedding Comparison (SPEC) framework, which compares two feature embeddings by forming the normalized difference of their kernel matrices, Λ = (1/n)(K_ψ1 − K_ψ2), and interpreting the principal eigenvectors of Λ as clusters of samples grouped by ψ1 but not by ψ2. The paper proves Theorem 1, a residual bound over all eigenvectors under two conditions on a candidate index set I; Corollary 1, which bounds the entries of a single eigenvector on the complement set I^c under an additional spectral-gap condition; Proposition 1, showing that the nonzero spectrum of Λ can be obtained from a smaller differential covariance matrix with O(n) complexity for bounded feature dimension; Theorem 2, an RFF concentration bound for shift-invariant kernels; and Proposition 2, giving a gradient for a spectral-radius alignment objective. Experiments compare CLIP, DINOv2, Inception, SWAV, RoBERTa, and E5 embeddings on AFHQ, FFHQ, ImageNet, MS-COCO, and text datasets, and apply SPEC-align to finetune CLIP toward DINOv2, reporting improved ImageNet linear-probe accuracy.","tokens_in":33249,"tokens_out":6518,"duration_ms":65253,"significance":"If the central interpretation is justified, SPEC offers a scalable and genuinely interpretable tool for embedding comparison and alignment, complementing existing evaluation metrics that rely on downstream task accuracy. The O(n) differential-covariance implementation and the RFF concentration bound are concrete technical contributions; the large-scale experiments on 100K+ samples demonstrate practical scalability. The paper also makes an honest effort to validate its cluster interpretations beyond label-based metrics, notably the cosine-similarity ranking tests in Figures 16–17. However, the main theoretical claim is stated more strongly than what is proven: the eigenvector interpretation requires a spectral-separability condition that is neither stated as an explicit assumption nor verified empirically.","major_comments":[{"comment":"The paper states unconditionally that “the principal eigenvectors can be interpreted as the clusters of samples assigned by embedding ψ1 that are less strongly grouped by the second embedding ψ2,” but the proven support for this interpretation is conditional on a spectral gap: Corollary 1 requires λ − λmax(Λ[I^c, I^c]) ≥ γ > 0. Conditions 1 and 2 of Theorem 1 do not guarantee this gap; the block Λ[I^c, I^c] may itself have large positive eigenvalues, and the top eigenvalue may be degenerate, in which case an eigensolver returns an arbitrary mixture of cluster indicators. This gap between the stated claim and the proven theorem is load-bearing for the paper’s central contribution. Please restate the main claim as conditional on an explicit spectral-separability assumption, and add an empirical diagnostic (e.g., comparing the top eigenvalues of Λ with estimates of λmax(Λ[I^c, I^c]) for the identified clusters) to check this condition on real data.","section":"Section 4 and Corollary 1"},{"comment":"The AMI/NMI validation compares SPEC-derived cluster labels with K-Means labels computed on the same embedding that produced the eigenvectors. A high AMI for ψ1 only shows that SPEC labels are consistent with one possible clustering of ψ1’s space; it does not directly measure whether ψ2 fails to cluster those samples. The cosine-similarity ranking tests in Figures 16–17 are a much more direct validation of the “clustered by ψ1 but not ψ2” claim; please apply this type of check systematically to the reported clusters, or at least report a clustering-quality metric for ψ2 on the SPEC-identified sample groups.","section":"Section 6, validation methodology"}],"minor_comments":[{"comment":"The proof uses the statement “for every PSD matrix A and vector v” to justify ∥Av − λv∥ ≥ |λ_j − λ| ∥v∥; the matrices Λ11 and Λ22 are not necessarily PSD. The inequality holds for all symmetric matrices, so replacing “PSD” with “symmetric” fixes the statement without changing the argument.","section":"Appendix A.1, final inequality"},{"comment":"The notation “Λψ1,ψ2 ∈ Rn” is misleading; the matrix is n × n, so it should be R^{n×n}.","section":"Definition 1"},{"comment":"The mapped eigenvectors u_i are not normalized before being used for cluster identification; since clustering thresholds entries of u_i, normalization (or at least a stated convention) is needed for interpretability and comparability across eigenvectors.","section":"Algorithm 1, line 11"},{"comment":"The bandwidth selection rule “selected the embeddings bandwidths such that the difference between top eigenvalue is less than 0.01” is data-dependent and under-specified; please provide the exact procedure and report the chosen σ values for reproducibility.","section":"Section 6, Experimental settings"},{"comment":"The Cholesky-based method for symmetric eigendecomposition of Θ is a useful contribution; however, the text should clarify that the correspondence holds only for nonzero eigenvalues and does not recover the null space of Λ, which is important when interpreting zero eigenvalues.","section":"Appendix A.3 and Remark 1"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a scalable and well-engineered method with a mostly correct theoretical core. The main gating issue is the mismatch between the unconditional wording of the central interpretability claim and the conditional nature of Corollary 1; this is fixable by rewording and adding an empirical spectral-gap check. A second, also addressable concern is the partly circular AMI/NMI validation. The RFF concentration bound and the Cholesky symmetric eigendecomposition are solid contributions. I see no reason to doubt the integrity of the results, but the revision needs to bring the claims into alignment with the proven statements before the paper can be recommended for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me tell you what I make of this one. The SPEC framework is a plausible, practical method for comparing two embeddings by eigendecomposing the normalized kernel difference matrix. The genuinely new pieces are the differential covariance construction (which makes the eigenspace computable in O(n) for bounded feature dimension), the SPEC-diff spectral-radius distance, and the gradient-based alignment objective. Proposition 1 is standard linear algebra, and it is a useful observation; Theorem 2's RFF concentration bound is also legitimate. On the empirical side, the qualitative results—CLIP vs DINOv2 on text-overlaid images, dog breeds vs gestures, the similarity-ranking sanity check—do show the method surfaces meaningful clusters. I would credit that.\n\nThe soft spots are real but not fatal. The main one: the paper claims in Section 4 that the principal eigenvectors can be interpreted as clusters assigned by one embedding but not the other. Theorem 1 does not establish that. It bounds a weighted average of residual norms across all eigenvectors, and Corollary 1 needs an explicit gap between the target eigenvalue and the spectrum of the complementary block—a condition that is never stated as an assumption and never checked empirically. With multiple differentially clustered groups, the top eigenvalue can be degenerate and an arbitrary eigensolver basis will mix cluster indicators. The conclusion informally acknowledges a spectral-separability assumption, but the statement in Section 4 is unconditional. That discrepancy should be fixed: either state the gap condition and verify it on real data, or soften the interpretation.\n\nThe validation also has a self-referential element: AMI/NMI is computed against KMeans on the same embedding that produced the SPEC eigenvectors, which inflates agreement. The SPEC-align experiment lacks a control baseline that trains the same CLIP model without the alignment term; the reported accuracies are inconsistent across the main text (67.20 to 73.93) and the appendix table (73.50 to 76.45), which makes me less confident in the headline claim. And the main experiments use m=2000 RFF features, even though the appendix shows clusters are unreliable below r=3000. Those are addressable, but they should be addressed.\n\nOverall: the central idea is sound enough to deserve referee time, and the scalable computation is a genuine practical contribution. I would send it to review, with a clear request to fix the gap condition, add a proper control, and report the RFF ablation in the main paper. It is not a desk reject.","headline":"A useful spectral tool for embedding comparison that overstates its interpretability guarantee—worth reviewing, but the central claim needs a gap condition and the validation needs hardening.","tokens_in":33784,"tokens_out":2488,"would_cite":false,"duration_ms":25619,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The principal eigenvectors of the normalized difference of two embeddings' kernel matrices mark the sample clusters one embedding captures and the other misses, computable in time linear in the number of samples.","keywords":["embedding comparison","kernel difference matrix","spectral method","kernel PCA","random Fourier features","embedding alignment","cluster mismatch detection","explainable AI"],"falsifier":"Build a synthetic dataset with a well-separated cluster $I$ in embedding $\\psi_1$ and make $\\psi_2$ group inside $I$ but not across its boundary, so that the conditions of Theorem 1 hold with small $\\epsilon_1$ and $\\epsilon_2$; if the top eigenvector of $\\Lambda_{\\psi_1,\\psi_2}$ then fails to concentrate on $I$, with its entries on $I^c$ exceeding $2\\sqrt{\\epsilon_1^2+\\epsilon_2}/\\gamma$ from Corollary 1, the central interpretation is false. On real data, the same check is available by computing the block norms for a SPEC-reported cluster and comparing the resulting $\\xi$ with the measured eigenvalue gap.","tokens_in":32812,"feed_emoji":"🔬","tokens_out":14630,"duration_ms":115728,"temperature":0.7,"pith_summary":"This paper claims that the differences between two feature embeddings can be made explainable by eigendecomposing the normalized difference of their kernel matrices. The top eigenvectors of $\\Lambda_{\\psi_1,\\psi_2} = \\frac{1}{n}(K_{\\psi_1} - K_{\\psi_2})$ carry large entries precisely on samples that embedding $\\psi_1$ groups into a cluster that embedding $\\psi_2$ does not strongly group. Because the non-zero eigenspectrum of $\\Lambda_{\\psi_1,\\psi_2}$ is shared by a $(d_1+d_2)\\times(d_1+d_2)$ differential covariance matrix, the computation scales linearly in the sample size for bounded feature dimension, and random Fourier features extend the method to shift-invariant kernels such as the Gaussian RBF kernel. The same spectrum yields a distance (SPEC-diff) and a gradient-based alignment procedure (SPEC-align) that measurably reduces CLIP's mismatch with DINOv2 on ImageNet. A sympathetic reader would care because this replaces black-box accuracy comparisons with named, visualizable clusters of disagreement.","feed_headline":"Eigenvectors unmask clusters one embedding sees, the other misses","feed_subtitle":"Visible clusters expose where embeddings differ; aligning CLIP to DINOv2 lifts ImageNet accuracy to 73.9%.","key_machinery":"The normalized kernel difference matrix $\\Lambda_{\\psi_1,\\psi_2} = \\frac{1}{n}(K_{\\psi_1}-K_{\\psi_2})$ is the object that carries the argument; its non-zero eigenspace is reinterpreted as a set of cluster indicators. The matching computational object is the differential covariance matrix $\\Gamma_{\\psi_1,\\psi_2}$, a $(d_1+d_2)\\times(d_1+d_2)$ block matrix built from the kernel covariance matrices of the two embeddings and their cross-covariance, which shares the non-zero eigenvalues of $\\Lambda_{\\psi_1,\\psi_2}$ while having dimension independent of the sample size. For shift-invariant kernels, random Fourier features replace the infinite-dimensional feature map with a finite proxy and give the approximation error bound of Theorem 2.","core_discovery":"The paper's central claim is a spectral identification result: under two conditions on an index set $I$ — that embedding $\\psi_1$ separates $I$ from its complement, $\\|\\frac{1}{n}K_{\\psi_1}[I,I^c]\\|_F \\le \\epsilon_1$, and that embedding $\\psi_2$ groups $I$ only weakly, $\\|\\frac{1}{n}K_{\\psi_2}[I,I]\\|_2 \\le \\epsilon_2$ — the eigenvectors of $\\Lambda_{\\psi_1,\\psi_2}=\\frac{1}{n}(K_{\\psi_1}-K_{\\psi_2})$ concentrate their mass on $I$ or its complement, with total squared residual bounded by $\\xi=4(\\epsilon_1^2+\\epsilon_2)$. Corollary 1 turns this into a per-eigenvector bound: if an eigenvalue is separated by $\\gamma$ from the spectrum of the complement block, the off-cluster entries of the eigenvector have squared norm at most $4(\\epsilon_1^2+\\epsilon_2)/\\gamma^2$. Proposition 1 supplies the scalable route: $\\Lambda_{\\psi_1,\\psi_2}$ and the differential covariance matrix $\\Gamma_{\\psi_1,\\psi_2}$ share their non-zero eigenvalues, so the eigendecomposition runs in $O(n(d_1+d_2)^2 + (d_1+d_2)^3)$ time, and Algorithm 1 computes the needed covariances in one streaming pass over the samples. The same eigenvalue spectrum defines the distance SPEC-diff, the spectral radius of $\\Lambda_{\\psi_1,\\psi_2}$, and SPEC-align minimizes this distance by gradient descent using the top left and right eigenvectors of $\\Gamma_{\\psi_1,\\psi_2}$.","pith_inferences":["One natural extension the paper leaves implicit is using SPEC as an unsupervised diagnostic for spurious correlations: a leading eigenvector that separates CLIP's text-overlaid clusters from DINOv2's content clusters is exactly the kind of label-free signal useful for auditing a model before deployment.","Because SPEC only needs kernel evaluations, it should apply to cross-modal and cross-architecture pairs beyond CLIP and DINOv2; the paper names cross-modal comparison as a future direction, and the same machinery would work for comparing image and text encoders with different dimensions.","The spectral-separability conditions suggest a practical validator: after a SPEC run, one could compute the measured block norms $\\epsilon_1,\\epsilon_2$ for each reported cluster and use the size of $\\xi$ relative to the eigenvalue gap to flag which clusters are trustworthy.","A robustness check for real deployments would be to perturb the kernel matrices, by resampling the random Fourier features or subsampling the dataset, and ask whether the top clusters persist; the appendix reports RFF-count sensitivity, which supports this as a testable protocol."],"forward_implications":["SPEC turns an embedding comparison into a checklist: each leading eigenvector names a concrete sample cluster, visualizable as images or sentences, that one model captures and the other misses.","Because the eigendecomposition runs in time linear in $n$ for bounded feature dimension, SPEC applies to datasets of 100K samples or more, such as MS-COCO and FFHQ, without subsampling.","SPEC-diff gives a symmetric, kernel-based distance between embeddings of different dimensions, and SPEC-align minimizes it by gradient descent; in the paper's setup, aligning CLIP to DINOv2 raised CLIP's frozen-feature ImageNet accuracy from 67.20% to 73.93%.","Random Fourier features extend the linear-time computation to shift-invariant kernels such as the Gaussian RBF kernel, with an error that decays as $O(1/\\sqrt{m})$ in the number of features.","The comparison is directional: swapping the roles of the two embeddings reports clusters grouped by the second embedding but not the first, so the tool supports pairwise debugging of any two models."],"supporting_citations":[{"why":"This is the kernel-PCA eigendecomposition that SPEC extends to the difference of two kernel matrices.","marker":"Schölkopf et al., 1998"},{"why":"This supplies the random Fourier features that make shift-invariant kernels finite-dimensional for linear-time scaling.","marker":"Rahimi & Recht, 2007"},{"why":"This gives the vector Bernstein inequality used to bound the RFF approximation error in Theorem 2.","marker":"Gross, 2011"},{"why":"This provides the concentration inequality, together with Gross, used in Theorem 2's proof.","marker":"Kohler & Lucchi, 2017"},{"why":"This defines the CLIP embedding that is the primary compared and aligned model in the experiments.","marker":"Radford et al., 2021"},{"why":"This defines the DINOv2 embedding used as the reference for comparison and alignment.","marker":"Oquab et al., 2024"},{"why":"This provides ImageNet, the dataset on which SPEC-align's accuracy improvement is measured.","marker":"Deng et al., 2009"},{"why":"This provides MS-COCO, a large-scale dataset used to demonstrate SPEC's scalability.","marker":"Lin et al., 2015"}],"fun_headline_variants":["Eigenvectors reveal clusters embeddings see differently","Kernel difference spectrum exposes embedding mismatches","Spectral alignment closes embedding gaps quickly","Eigenvalue view shows where embeddings disagree","Linear-time spectral compare and align embeddings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the groups the two embeddings cluster differently are spectrally separable, showing up as a clear, well-gapped direction in the kernel difference matrix, and the paper gives no procedure for checking that this is true on a given real dataset.","fun_headline_variants_meta":{"raw":{"variants":["Eigenvectors reveal clusters embeddings see differently","Kernel difference spectrum exposes embedding mismatches","Spectral alignment closes embedding gaps quickly","Eigenvalue view shows where embeddings disagree","Linear-time spectral compare and align embeddings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1637,"prompt_tokens":1110,"completion_tokens":527,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":726,"completion_tokens_details":{"reasoning_tokens":463}},"tokens_in":726,"tokens_out":527,"duration_ms":6349,"temperature":1.0,"reasoning_tokens":463,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:58:39.966926+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a synthetic dataset with a well-separated cluster $I$ in embedding $\\psi_1$ and make $\\psi_2$ group inside $I$ but not across its boundary, so that the conditions of Theorem 1 hold with small $\\epsilon_1$ and $\\epsilon_2$; if the top eigenvector of $\\Lambda_{\\psi_1,\\psi_2}$ then fails to concentrate on $I$, with its entries on $I^c$ exceeding $2\\sqrt{\\epsilon_1^2+\\epsilon_2}/\\gamma$ from Corollary 1, the central interpretation is false. On real data, the same check is available by computing the block norms for a SPEC-reported cluster and comparing the resulting $\\xi$ with the measured eigenvalue gap.","supporting_citations":[{"cited_title":"Dinov2: Learning robust visual features without supervision, 2024","cited_arxiv_id":null,"evidence_quote":"This defines the DINOv2 embedding used as the reference for comparison and alignment."},{"cited_title":"ImageNet: A Large-Scale Hierarchical Image Database","cited_arxiv_id":null,"evidence_quote":"This provides ImageNet, the dataset on which SPEC-align's accuracy improvement is measured."},{"cited_title":"L., and Dollár, P","cited_arxiv_id":null,"evidence_quote":"This provides MS-COCO, a large-scale dataset used to demonstrate SPEC's scalability."}],"review_version":1}