{"id":"9b9f51f4-7fcf-4379-966f-eb77e14f8215","arxiv_id":"2509.24467","paper_version":4,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"KREPES trains kernel models for self-supervised learning at scale with Nyström landmarks and reads out interpretability directly from the learned coefficients.","lead":"The paper introduces KREPES, a kernel-based framework for self-supervised learning that scales to large datasets and explains its own representations by pointing to the training examples that matter most. It is worth reading because it promises a rare combination in modern machine learning: representations that are competitive with deep networks and still auditable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CIFAR-10 comparison is confounded by ImageNet-pretrained eNTK features; the claimed label-free bias audit is also absent.","rationale":"The reader's weakest assumption correctly identifies the pretrained-eNTK confound in the CIFAR-10 comparison, and this is the most load-bearing issue because every headline empirical claim about scalability and parity with neural networks depends on Table 1. My concrete test directly isolates whether KREPES' reported 88–90% accuracy comes from its SSL training or simply from an ImageNet-pretrained feature representation. The additional missing Adult-1M bias audit and the label-dependent class-coverage metric reinforce the reader's REJECT verdict: the paper advertises label-free interpretability and a specific bias audit that the body does not deliver. These are not stylistic concerns; they are verifiability failures. I therefore agree with the reader's verdict, and my stress-test does not move it.","tokens_in":19316,"tokens_out":6019,"duration_ms":265404,"concrete_test":"Rerun the CIFAR-10 row of Table 1 under controlled conditions: (a) KREPES with eNTK from a randomly initialized ResNet34 using the same weight seed as the NN baseline, trained with the same SSL loss and hyperparameter budget; and (b) a linear probe directly on the frozen ImageNet-pretrained ResNet34 eNTK feature map (or pretrained ResNet34 features) using the same 10% labeled split, without any KREPES training. If random-eNTK KREPES accuracy falls to the NN baseline range, or if the frozen-feature linear probe alone already reaches ~90%, then Table 1's advantage is attributable to pretrained features rather than KREPES. Separately, check whether the Adult-1M bias audit exists in the supplementary materials; if not, the abstract claim should be removed or the experiment added.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim that KREPES performs comparably to neural networks (Table 1) is not supported as stated. Appendix B.2 says that for CIFAR-10 the kernel is the eNTK of an ImageNet-pretrained ResNet34, and the baseline is that same pretrained ResNet34; yet the NN column reports only 41–60% accuracy while KREPES reaches 89–90%. This gap is implausible if both methods actually start from the same pretrained initialization. Either the NN baseline is trained from random initialization or far fewer epochs, or the pretrained eNTK already carries ImageNet semantic structure that the linear probe exploits. In both cases, the comparison does not isolate what KREPES' SSL objective contributes. The same confound affects the scalability and interpretability claims that rest on these numbers. Additionally, the supplied abstract promises an Adult-1M demographic-proxy bias audit, but the full text contains no such experiment, and the 'label-free' interpretability claim is contradicted by Section 5's class-coverage metric κ, which uses ground-truth labels Y and y(x_l) to compute coverage. Together these missing and confounded pieces make the main contributions unverifiable as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KREPES, a kernel-based representation-learning framework in which an SSL loss (SimCLR, BYOL, VICReg, Barlow Twins, etc.) is minimized over functions of the form f(x)=A^T k_x + gamma, where k_x are kernel evaluations against a small set of Nyström landmarks. A principal-component initialization, two preconditioning schemes, and randomized leverage-score landmark selection are introduced to make optimization practical. The paper claims that KREPES matches deep-network SSL accuracy on datasets up to 1M samples while using far fewer trainable parameters, and that the resulting representations are interpretable through representer landmarks and influence scores. The interpretability section also promises concept-level explanation via concept activation vectors. However, the empirical comparison is confounded for CIFAR-10, several advertised deliverables are absent from the main text, and the interpretability metrics are largely definitional rather than validated.","tokens_in":19610,"tokens_out":7577,"duration_ms":59878,"significance":"If the claims were fully supported, the paper would make a useful contribution: scaling kernel-based SSL objectives via Nyström approximation and using the resulting representer structure for post-hoc explanations is a natural and potentially valuable idea. The algorithmic components (eNTK + Nyström integration, principal-component initialization, GGN preconditioning for SSL losses) are concrete and could benefit practitioners. However, the current validation does not establish the central claims: the headline CIFAR-10 comparison is unfair, the scalability and bias-audit promises in the abstract are not implemented, and the influence-score interpretability is a restatement of the model's definition rather than an independently validated explanation. The significance is therefore conditional on a major revision that supplies fair baselines, complete experiments, and a reframed interpretability evaluation.","major_comments":[{"comment":"The central empirical claim that KREPES performs comparably to deep SSL is not supported as stated. Appendix B.2 says that for CIFAR-10 the kernel is the eNTK of an ImageNet-pretrained ResNet34 and that the NN baseline is the same pretrained ResNet34. Table 1 then shows KREPES at 88.8–90.3% versus NN at 41.5–60.0%. If the baseline is actually trained from that initialization, the comparison conflates the pretrained feature geometry with the proposed SSL objective; if the baseline is trained from random initialization, KREPES receives an unacknowledged benefit of ImageNet features. In either case, the comparison does not isolate KREPES's contribution. The 'fewer parameters' statement is also misleading because the frozen eNTK ResNet34 backbone contributes roughly 22M parameters to the effective model. A fair comparison requires either a randomly initialized eNTK for CIFAR-10 or a frozen-f","section":"§4, Table 1; App. B.2"},{"comment":"The abstract promises deliverables that do not appear in the paper: (i) the Adult-1M demographic-proxy bias audit ('revealing an algorithmic bias ... where SSL uses demographic proxies for income'), (ii) scalability to ImageNet-1K, and (iii) the 'Feature Alignment Gap' metric. Section 5 and Appendix E contain only the CIFAR-100 sea example; no Adult-1M audit, no ImageNet-1K experiment, and no definition of 'Feature Alignment Gap' appear anywhere in the text. These are advertised contributions, not optional extras; they must either be implemented or removed from the claims.","section":"Abstract and §5"},{"comment":"The proposed influence score is definitional rather than validated. Since f(x)=A^T k_x + gamma, the quantity iota_{l→t} = k(x_test, x_l) ||A_l|| is exactly the norm of the l-th representer term that composes the model output. Showing that high-scoring landmarks are close to the test input and have large learned coefficients is a restatement of the kernel model, not evidence that the explanation is meaningful to a user. The paper should compare against alternative attribution methods (e.g., leave-one-out, SHAP) or provide a quantitative user study or ground-truth-based evaluation of explanation quality. Relatedly, the class-coverage metric kappa in §5 uses ground-truth labels Y and y(x_l), so the 'label-free' characterization of the interpretability analysis is overstated.","section":"§5.1, Eq. (3)"},{"comment":"The timing comparison is incomplete. Table 1 reports only the time per forward/backward pass multiplied by the number of epochs needed after the kernel is available. Appendix F states that computing the CIFAR-10 eNTK for ResNet34 takes less than 2 hours on four GPUs. KREPES is therefore not end-to-end 'seconds' training on CIFAR-10 when kernel construction is included. Because scalability is a central contribution, the total wall-clock including kernel construction should be reported, or the claims should be explicitly restricted to the optimization phase.","section":"§4, Table 1; App. F"},{"comment":"All experimental tables report point estimates without error bars or number of seeds. The differences in Table 1 are often fractions of a percent (e.g., Adult-1M: 83.78 vs 83.72), and Table 2 compares preconditioners on similar small margins. Given the many hyperparameters tuned per dataset and loss (lambda, m, h, tau, VICReg coefficients, early stopping), the stability of these comparisons is unknown. At least three independent runs with standard deviations should be reported for the main claims.","section":"§4, Tables 1 and 2"}],"minor_comments":[{"comment":"The manuscript's title in the provided text is 'Interpretable Kernel Representation Learning at Scale...', while the arXiv title is 'Interpretable Self-Supervised Learning via Representer Landmarks and Nyström Approximation'. The mismatch should be resolved in the camera-ready version.","section":"Title and abstract"},{"comment":"In the definition of Score(l→t, c), the term k(x_test, c_j) is likely a typo; it should presumably be k(x_test, x_l). The symbol c_j is not defined.","section":"§5.1, Eq. (3)"},{"comment":"The notation for global importance is inconsistent: earlier in Section 5 the landmark importance is defined as omega_l = ||a_l||_2 (row norm), but Section 5.1 refers to 'the landmark's corresponding column in the learned projection matrix ||A_l||'. Please clarify the exact indexing of A.","section":"§5.1"},{"comment":"The label 'Random Uniform' should be 'uniform random' or 'random sampling' to avoid implying a separate 'random' versus 'uniform' method.","section":"App. D, Table 5"},{"comment":"Figure 2 is referenced but not analyzed in the text beyond its caption. Please add a paragraph explaining what the figure demonstrates and how the influence scores should be read.","section":"§5, Figure 2"},{"comment":"The text says 'maximum of 50 iterations' and later 'maximum number of epochs is between 60 and 80'; the units (iterations vs epochs) need to be reconciled.","section":"App. B.3"}],"recommendation":"major_revision","confidential_remarks":"The CIFAR-10 confound is the load-bearing issue: the comparison as written cannot support the headline 'performs similar' claim. The missing Adult-1M bias audit and ImageNet-1K experiment are also serious because they are advertised in the abstract. I would ask the editor to require a fair baseline, removal or substantiation of the missing deliverables, and a reframing of the interpretability claims. If the authors cannot provide a fair CIFAR-10 comparison or choose to leave the abstract unmodified, rejection may be the appropriate outcome."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a plausible systems paper that fills a real gap, but the main empirical claim doesn't hold as stated. The CIFAR-10 comparison gives KREPES 89-90% accuracy using an eNTK built from an ImageNet-pretrained ResNet34, while the NN baseline is reported at 41-60%. Those numbers are not credible if the baseline really is the same pretrained network, and if it is trained from scratch the comparison doesn't isolate what KREPES contributes. Either way, Table 1 as presented is misleading. The same confound undercuts the scalability and interpretability claims that rest on those numbers.\n\nCredit where it's due: the Nyström + gradient-based optimization for kernel SSL is a sensible combination, and the representer-landmark interpretation is genuinely convenient -- you get influence scores for free from the learned coefficients. The GGN preconditioning detail is thoughtful, and the tabular results look sane. The paper is also honest about the hyperparameter-tuning-with-labels issue, which is a real constraint in this setting.\n\nSoft spots, in proportion: the Adult-1M bias audit promised in the abstract is absent from the body -- that is a load-bearing missing experiment. The 'label-free' interpretability claim is overstated because the class-coverage metric uses ground-truth labels. The influence score itself is close to a tautology: it decomposes the prediction by construction, so it is a descriptive decomposition, not an independently validated explanation. These are not cosmetic issues, but they are fixable.\n\nThis paper is for readers who care about scaling kernel methods to SSL or about representer-style interpretability. I would not cite the numbers yet, and I would not bring it to a reading group in its current form, but I would send it to peer review: the core idea deserves refereeing, and the flaws look addressable. A major revision with fair baselines, the promised bias experiment, and a toned-down interpretability claim could become a solid paper.","headline":"Reasonable kernel-SSL engineering, but the headline CIFAR-10 comparison is confounded by an ImageNet-pretrained eNTK and the promised bias audit is missing.","tokens_in":20173,"tokens_out":1840,"would_cite":false,"duration_ms":58114,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A kernel-based SSL framework scales to million-sample datasets and makes each embedding traceable to specific training examples.","keywords":["self-supervised learning","kernel methods","Nyström approximation","representer theorem","interpretability","neural tangent kernel","representation learning","influence scores"],"falsifier":"Train KREPES on CIFAR-10 with an empirical NTK computed from a randomly initialized ResNet34 (no ImageNet pretraining), keeping all other settings identical; if the 88–90% downstream accuracy collapses toward the 41–60% of the neural-network baseline, the paper's scaling claim is carried by pretrained features rather than by the kernel self-supervised machinery.","tokens_in":19156,"feed_emoji":"🔬","tokens_out":8460,"duration_ms":55844,"temperature":0.7,"pith_summary":"KREPES is a proposed framework for doing self-supervised representation learning with kernel machines instead of deep networks. The paper argues that by restricting the model to a Nyström subset of 'representer landmarks' and training the coefficient matrix with gradient descent, standard SSL objectives (SimCLR, BYOL, VICReg, Barlow Twins, and others) can be optimized at up to one-million-sample scale. Because the representer theorem makes the learned embedding a weighted sum of kernel similarities to the landmarks, interpretation is not an add-on: the trained weights tell you which unlabeled examples drive a given representation, and the paper introduces influence scores and concept-alignment profiles to read them out. Empirically the paper reports accuracy close to or better than the corresponding neural networks on CIFAR-10, Adult, Higgs, and CoverType, with an order of magnitude fewer trainable parameters. If correct, this would give large-scale SSL a route to transparency, including auditing for demographic bias in unlabeled data.","feed_headline":"Kernel self-supervised learning scales to 1M and explains its choices","feed_subtitle":"By anchoring embeddings to a few landmark examples, the method keeps representations auditable—and can expose demographic proxies.","key_machinery":"The representer theorem for kernels in the Nyström subspace: any optimal embedding has the form f(x)=Σ_{l=1}^{m} α_l k(c_l, x)+γ, where c_l are the selected landmarks (a subset of training points) and α_l are learned row vectors. This identity does double duty: it reduces an infinite-dimensional search to the m×h matrix Ã, making gradient-based SSL tractable, and it gives every coordinate of the embedding a direct attribution to specific training examples. The accompanying Nyström low-rank factorization K≈K_nm K_mm^{-1}K_mn and the principal-component initialization Ã0=U_hΛ_h^{-1/2} are what let the method reach 1M samples; the GGN preconditioners are what let it optimize arbitrary SSL losse","core_discovery":"The central discovery is that a kernel-based representation learner can be made practical and interpretable at the same time: for any of a wide class of SSL losses, optimizing over the finite-dimensional space spanned by m landmark features yields solutions f(x)=Σ_l α_l k(c_l,x)+γ, and the coefficient matrix à learned by gradient descent is itself the explanation. The Nyström approximation (m≈2000 landmarks), a principal-component initialization Ã0=U_hΛ_h^{-1/2}, and generalized Gauss–Newton preconditioners make training tractable; the representer theorem guarantees the optimal map lies in the landmark span, so influence scores (local kernel similarity times global column norm) and concept-c","pith_inferences":["A fair test of the 'kernel SSL scales' claim would run KREPES with a randomly initialized eNTK and a frozen-backbone baseline; the paper only compares against a fully-trained pretrained network, so it does not isolate whether the pretrained features or the kernel machinery produce the gains.","The abstract promises a 'Feature Alignment Gap' metric that the main text never defines; the transparency claims would be easier to check if that metric were specified, since the body only defines influence scores and concept-alignment profiles.","The conceptual influence profile requires a labeled set of concept-positive and concept-negative examples (e.g., 500 'Sea' samples), which reintroduces the labeling cost that SSL is meant to avoid; extending this to zero-shot concept definitions would be a natural next step.","The observation that SSL objectives self-organize landmark norms by semantic class (small κ for high accuracy) hints at a general principle—that contrastive and joint-embedding losses allocate representational weight across categories even without labels—which, if confirmed, would make κ a useful diagnostic for representation quality."],"forward_implications":["If KREPES's claims hold, self-supervised representation learning no longer requires a black-box deep network: a kernel model with a few thousand landmarks can be trained with the same losses at comparable accuracy.","Because the learned coefficients are the explanation, auditing a representation reduces to ranking landmarks by influence; the paper demonstrates this by exposing demographic proxies in Adult-1M without any supervision.","The framework is loss-agnostic, so new or custom SSL objectives can be plugged in and still inherit the interpretability and scaling benefits, as long as the loss is differentiable.","The reported parameter counts (0.5–0.9M versus 5–22M for the neural baselines) suggest kernel SSL could be a lighter-weight alternative for compute-limited settings, though the pretrained-kernel caveat matters.","The class-coverage metric κ, which tracks how few top landmarks span all classes, offers a training-free signal that correlates with downstream accuracy and could be used for model selection."],"fun_headline_variants":["Landmark kernels make SSL interpretable and scale to 1M","Audit SSL without labels via representer landmarks","SSL bias revealed by explainable landmark analysis","KREPES: auditable SSL at 1M scale, no labels required","Spot demographic bias in SSL with landmark kernels"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The accuracy numbers depend on the kernel being built from a network that is already pretrained on the same type of data (for images, a ResNet34 trained on ImageNet), and the paper never tests a randomly initialized kernel, so the parity with neural networks may be coming from the pretrained features rather than from the kernel self-supervised machinery.","fun_headline_variants_meta":{"raw":{"variants":["Landmark kernels make SSL interpretable and scale to 1M","Audit SSL without labels via representer landmarks","SSL bias revealed by explainable landmark analysis","KREPES: auditable SSL at 1M scale, no labels required","Spot demographic bias in SSL with landmark kernels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001992,"raw_usage":{"total_tokens":7608,"prompt_tokens":734,"completion_tokens":6874,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":6793}},"tokens_in":478,"tokens_out":6874,"duration_ms":38500,"temperature":1.0,"reasoning_tokens":6793,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T14:32:52.975006+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train KREPES on CIFAR-10 with an empirical NTK computed from a randomly initialized ResNet34 (no ImageNet pretraining), keeping all other settings identical; if the 88–90% downstream accuracy collapses toward the 41–60% of the neural-network baseline, the paper's scaling claim is carried by pretrained features rather than by the kernel self-supervised machinery.","supporting_citations":[],"review_version":1}