{"id":"d0b01c85-b9ad-41c9-ae9e-8425356c8b67","arxiv_id":"2501.18452","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ReSA derives a soft assignment target from the encoder's own clustered outputs and uses it to supervise the embedding, improving self-supervised representation quality on CIFAR, ImageNet-100, ImageNet, and transfer tasks.","lead":"This paper proposes ReSA, a self-supervised learning method that uses the clustering structure of the encoder's outputs to build training targets for the projection head. On standard benchmarks the authors report consistent gains over prior SSL methods, especially for k-nearest-neighbor and fine-grained classification.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ReSA's headline gains in Table 1 are confounded with weak augmentation and a different linear-evaluation protocol, so the self-clustering mechanism is not isolated.","rationale":"The paper is substantial: it includes multiple benchmarks, ablations, a long-tailed study, and a code repository, and several results (especially k-NN and ImageNet-1K linear) are less dependent on the linear-eval protocol. I do not think the central idea is false. However, the most load-bearing weakness is not the early-training behavior of the Sinkhorn assignment, but the confound between the proposed ReSA mechanism and two non-clustering changes: the weak augmentation design and the different linear-evaluation protocol. Figure 6 shows that the augmentation change alone has a large effect on the very benchmarks used in Table 1. Appendix B.3 concedes that the evaluation protocol affects baseline numbers. The reader's rationale already notes the protocol mismatch, so my verdict stays CONDITIONAL/UNCHANGED, but the condition should be sharpened: the authors need to demonstrate that ReSA beats matched baselines under identical augmentation and evaluation settings. The proposed test does exactly that. If the margin survives, the paper's claim is well supported; if it evaporates under matched conditions, the positive-feedback mechanism is not the source of the reported improvement.","tokens_in":28119,"tokens_out":7532,"duration_ms":83137,"concrete_test":"Re-run ReSA and every Table 1 baseline on CIFAR-10, CIFAR-100, and ImageNet-100 under the identical augmentation set (start with the standard solo-learn transform set) and the identical linear-evaluation protocol, with at least 3 seeds; then repeat the same comparison with weak augmentation applied to all methods. If ReSA's margin over the best baseline shrinks to below 1-2% or reverses when augmentation and evaluation protocol are matched, the claimed benefit of the self-clustering assignment in Eq. (2) is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that using the encoder's own clustering assignment as a training target (Eq. 2) improves SSL. But ReSA is not compared under matched conditions. Section 4.3 and Figure 6 show that moving ReSA from standard augmentation to the 'weak' augmentation (ResizedCrop + HorizontalFlip) improves ImageNet-100 linear accuracy by roughly 4 points and k-NN by roughly 3.5 points. Table 1 then compares ReSA-with-weak-augmentation against baselines trained with standard augmentation. In addition, Appendix B.3 states that ReSA is evaluated with the W-MSE/INTL linear protocol, and that the same protocol makes solo-learn baselines 'degrade noticeably,' so the paper reports the official solo-learn numbers instead. The k-NN results in Table 1 and the ImageNet-1K results in Table 2 are less affected by the linear-eval mismatch, so the method may well be strong; but the experimental comparison does not isolate whether the gain comes from the self-clustering assignment or from augmentation and evaluation choices. Without this isolation, the paper's claim that leveraging encoding clustering is what improves representations is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the clustering properties of self-supervised joint embedding architectures. It presents empirical evidence, using Silhouette Coefficient, Adjusted Rand Index, and linear/k-NN evaluation on CIFAR-10/100, that the encoder's output (the encoding) has stronger and more stable clustering properties than the embedding or the projector's hidden-layer outputs. It then proposes Representation Self-Assignment (ReSA), which replaces the identity target of InfoNCE with a doubly stochastic assignment matrix obtained by applying Sinkhorn-Knopp to the encodings' cosine-similarity matrix, and trains the model with a cross-entropy loss between embeddings and this self-assignment. Results are reported on CIFAR-10/100, ImageNet-100, ImageNet with ResNet-50 and ViT-S/16, COCO detection/segmentation transfer, fine-grained datasets, low-shot and long-tailed settings, along with ablations and a gradient analysis of the proposed loss.","tokens_in":28333,"tokens_out":6123,"duration_ms":58135,"significance":"The central idea is novel and plausible: if SSL encodings already carry useful clustering structure, that structure can be recycled as a training target. The paper's analysis of where clustering properties reside (encoding vs. embedding vs. projector hidden layers) is a useful empirical contribution, and ReSA is simple, prototype-free, and computationally cheaper than comparable clustering-based methods such as SwAV and DINO. The gradient derivation in Eq. (8) is clear, the ablations are informative, and the authors provide code. However, the reported performance gains are currently confounded by differences in augmentation and in the linear evaluation protocol between ReSA and the baselines, so the central claim that the self-assignment mechanism is what improves representations is not yet established.","major_comments":[{"comment":"The linear evaluation protocol is not matched between ReSA and the baselines. Appendix B.3 states that ReSA is evaluated on CIFAR-10/100 with the W-MSE/INTL Adam-based protocol and that applying this protocol to models from solo-learn makes their performance 'degrade noticeably,' so the paper reports the official solo-learn numbers instead. Consequently the linear columns of Table 1 compare ReSA under a favorable protocol against baselines under a different protocol, and the statement in §5.1 that the experiments 'strictly adhere to the same experimental settings as other methods' is not supported. The authors should either evaluate every method with the same protocol or report both protocols for all methods, quantifying the degradation explicitly.","section":"§5.1, Table 1; Appendix B.3"},{"comment":"ReSA is trained with weak augmentation (ResizedCrop + HorizontalFlip) for the clustering view, while the baselines are trained with the standard augmentation set. Figure 6 shows that moving ReSA from standard to weak augmentation improves ImageNet-100 linear accuracy by roughly 4 points and k-NN accuracy by roughly 3.5 points, so the gains in Table 1 cannot be attributed solely to the self-assignment mechanism. Because the weak augmentation is used in all reported ReSA pretraining runs, the comparisons in Tables 2, 3, 5, 6, and 10 are also affected. To isolate the contribution of the self-clustering loss, the authors should report ReSA with standard augmentation on both views and at least one baseline with weak augmentation, all under otherwise matched pretraining and evaluation settings.","section":"§4.3, Figure 6; Appendix B.2; Tables 1–6, 10"}],"minor_comments":[{"comment":"The text says 'can be writen as'; this should be 'written.'","section":"§4.2"},{"comment":"The legend label 'BarlowT wins' appears to be a typo for 'Barlow Twins'; please correct it for consistency with the main text.","section":"Figure 3 caption"},{"comment":"The word 'evalutaion' should be 'evaluation.'","section":"Appendix C.3"},{"comment":"The notation is ambiguous: Eq. (1) uses Fθ and F′θ′ for the two branches, but the subsequent definition Fθ(·) = Gθg(Eθe(·)) does not specify how θ′ relates to θ (e.g., a momentum or shared encoder). Please clarify the parameterization of both branches.","section":"§2.1"},{"comment":"The main text describes the assignment matrix as SH = H⊤H computed on a single encoding H, whereas the PyTorch-style algorithm computes Sinkhorn(cos_sim(h1, h1m)) between the online and momentum encodings of the weak view. Please align the description with the actual implementation.","section":"§4.1 vs. Appendix B.2"},{"comment":"The argument that diagonal dominance of AH prevents early clustering errors is heuristic; providing a quantitative measure, such as the mean diagonal-to-off-diagonal ratio of AH during the first epochs or the assignment accuracy at initialization, would make the claim more convincing.","section":"Appendix C.2"}],"recommendation":"major_revision","confidential_remarks":"The headline comparisons in Tables 1–6 and 10 need to be recomputed under matched conditions before the paper's central claim can be assessed. The authors have the resources to run these experiments, and the method itself is interesting. I would also ask the editor to consider whether the novelty relative to related relational and clustering-based SSL works (e.g., ReSSL and the re-weighting approach of Ma et al., 2023) is discussed sufficiently, since those works also use relational or cluster information in SSL objectives."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ReSA is a genuinely novel SSL objective, and the empirical story is more interesting than the headline numbers. The core idea is simple: compute a Sinkhorn assignment matrix from the encoder's self-similarity, then use it as a soft assignment target for the embedding cross-entropy instead of InfoNCE's identity target. The gradient analysis in Eq. (8) is correct and shows a clean softening of the contrastive target. The ablation that extracts assignments from encoding vs embedding (Figure 10) is exactly the right control and supports their \"where to read clusters\" claim. The clustering metrics study in Section 3 is also a useful empirical note for the field. Code is released.\n\nThe soft spot is real and the stress-test note has it right. Table 1 is not a matched comparison. ReSA is trained with weak augmentation for one view and evaluated with the W-MSE/INTL Adam linear protocol, while the baselines are inherited from solo-learn/official results. The authors admit the solo-learn baselines degrade noticeably under their protocol. Weak augmentation alone is worth about 4 points linear and 3.5 points k-NN on ImageNet-100 (Figure 6), so the Table 1 gains do not isolate the self-clustering mechanism. Table 12's weak-augmentation control only covers CIFAR-10/100 for SwAV, VICReg, MoCoV3 and does not rule out that the weak augmentation matters more on ImageNet-100 with this loss. Within-subfield, this is a load-bearing gap, not a fatal one, because the k-NN margins (e.g., 93.0 vs 90.0 on CIFAR-10) are large and the ImageNet-1K numbers in Table 2 are less affected by the linear-eval mismatch. The paper needs a matched run: all baselines under the same augmentation and evaluation protocol, multiple seeds, git commit. Without that, the claim that leveraging encoder clustering is what drives the gains is not established.\n\nI'd send this to referees but would not accept it as-is. The method is plausible, the analysis is mostly careful, and the confound is fixable with compute, not with a fundamentally new idea. A reader doing SSL will want to know about ReSA, and the diagnostic on encoding-vs-embedding clustering is worth citing. For my own work I'd cite it, but I would not trust the Table 1 deltas until the matched comparison appears.","headline":"ReSA is a genuinely new SSL objective with solid diagnostics, but the headline gains aren't matched and the self-clustering mechanism isn't isolated from augmentation and eval choices.","tokens_in":28906,"tokens_out":1968,"would_cite":true,"duration_ms":19702,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the encoder's output is the most clusterable component of a self-supervised model, and that feeding that clustering back as a soft assignment target in the loss — ReSA — improves SSL across classification, transfer…","keywords":["self-supervised learning","joint embedding architectures","representation self-assignment","Sinkhorn-Knopp","clustering properties","contrastive learning","positive-feedback learning","fine-grained and coarse-grained representations"],"falsifier":"Train ReSA on a standard dataset with the self-assignment matrix forced to the identity while keeping everything else identical; if this configuration still matches ReSA's reported accuracy, the clustering signal is not the driver. Conversely, compute the assignment matrix from a randomly initialized encoder, measure whether its off-diagonal entries agree with true class labels, and ablate those off-diagonal entries during training; if they carry no class information and their removal does not change performance, the paper's proposed mechanism is unsupported.","tokens_in":27901,"feed_emoji":"🧠","tokens_out":10075,"duration_ms":84180,"temperature":0.7,"pith_summary":"The paper argues that self-supervised learning models already contain a hidden asset: the encoder output, or encoding, has better and more stable clustering structure than the embedding the loss usually sees. It establishes this with clustering metrics across several standard SSL models, then turns the observation into a training method. ReSA computes a doubly stochastic assignment matrix from the encodings of one view and uses it as a soft target for aligning the embeddings of another view, instead of the hard identity target of InfoNCE. On CIFAR, ImageNet-100, ImageNet, COCO transfer, fine-grained, and low-shot benchmarks, the paper reports that ReSA outperforms prior SSL methods and produces representations that separate both fine-grained and coarse-grained clusters. The point is that self-supervision can bootstrap itself from its own clustering geometry rather than treating every image pair as unrelated.","feed_headline":"A model's own clusters become its self-supervised training signal","feed_subtitle":"ReSA replaces contrastive identity matching with encoder self-assignments, lifting ImageNet and transfer results.","key_machinery":"The load-bearing object is the self-assignment matrix $A_H = \\mathrm{Sinkhorn}(H^\\top H)$: the encoder's $L_2$-normalized batch outputs are simultaneously the points to be clustered and the anchors, and Sinkhorn-Knopp converts their cosine self-similarity into a doubly stochastic matrix. The matrix acts as a soft target in a cross-entropy loss between embeddings $Z$ and $Z'$, so the model is trained to make cross-view similarities follow its own current cluster geometry. Because $A_H = I$ recovers InfoNCE exactly, the design is a strict generalization of contrastive learning; because the diagonal of $A_H$ is sharply larger than off-diagonal entries, the optimization initially behaves like invariant alignment, which the paper argues prevents collapse and early clustering errors.","core_discovery":"ReSA replaces the identity matrix in the InfoNCE objective with the online self-assignment matrix $A_H = \\mathrm{Sinkhorn}(H^\\top H)$, computed without gradients from $L_2$-normalized encoder outputs $H$, and minimizes $\\ell_{\\mathrm{ReSA}} = -\\frac{1}{2m}\\sum_{i,j} A_H \\circ \\log D(Z^\\top Z')$ plus its symmetric term. The matrix is doubly stochastic and its diagonal dominates because each sample is most similar to itself, which keeps early training focused on pulling augmented views of the same image together while avoiding hard negative repulsion between same-class samples. The paper shows the gradient with respect to any similarity $s_{i,j}$ is $\\frac{1}{\\tau}(P_{i,j} - A_H(i,j))$, so ReSA smoothly up-weights or down-weights every pair instead of splitting similarities into positive and negative regimes. With this single change ReSA is reported to beat contrastive and clustering SSL methods on linear and k-NN classification, COCO detection and segmentation, fine-grained transfer, and low-shot evaluation, and the learned representations show cleaner fine- and coarse-grained cluster structure.","pith_inferences":["Editorial extension: the diagonal-dominance mechanism predicts ReSA's advantage should shrink with very small batch sizes, because the self-similarity matrix becomes so small that off-diagonal structure carries little semantic information; the paper does not directly test this regime.","Editorial extension: because only the cross-entropy form is tested, the same self-assignment target could plausibly be appended to other joint-embedding losses, such as covariance or redundancy-reduction objectives, but the paper leaves that open.","Editorial extension: a natural check is to run ReSA on modalities whose encodings do not obviously cluster, such as text or audio; the paper's own logic predicts that gains would scale with how clusterable the encoding is.","Editorial extension: the long-tailed dataset experiment suggests the method recovers from early noisy clusters, but does not isolate whether the benefit comes from softened negatives or from the diagonal prior; that separation is testable."],"forward_implications":["Pretraining can be improved without labels, new data, extra prototypes, or hand-designed negative pairs, by using the encoder's own cluster assignments as the training target.","Because same-class negatives are no longer pushed apart as hard, the embedding space preserves semantic clusters, explaining the reported gains on k-NN classification and fine-grained transfer.","The self-assignment is computed once per iteration regardless of the number of views, so multi-view training with this loss is cheaper than prototype-based clustering methods.","Weak augmentation on the clustering view is enough, so ReSA points toward simpler augmentation pipelines in large-scale pretraining.","The loss family parameterized by how far $A_H$ is from identity interpolates between InfoNCE and self-clustering, giving a controlled way to test how much cluster structure helps."],"supporting_citations":[{"why":"Establishes that SSL representations carry hierarchical, centroid-like semantic clusters, the property ReSA exploits.","marker":"(Ben-Shaul et al., 2023)"},{"why":"Defines InfoNCE, whose identity-matching target ReSA generalizes; the equivalence $A_H = I$ is the baseline relation.","marker":"(Oord et al., 2018)"},{"why":"Supplies the online Sinkhorn clustering precedent, the Sinkhorn hyperparameters, and the SwAV baseline ReSA improves on.","marker":"(Caron et al., 2020)"},{"why":"Provides the Sinkhorn-Knopp algorithm that computes the doubly stochastic self-assignment matrix.","marker":"(Cuturi, 2013)"},{"why":"Supplies the SimCLR baseline and the projector setup, whose information-buffer hypothesis motivates comparing encoding versus embedding.","marker":"(Chen et al., 2020a)"},{"why":"Supplies the SimSiam baseline and the momentum-network and predictor design that ReSA adopts.","marker":"(Chen & He, 2021)"},{"why":"Provides the InfoNCE gradient analysis whose positive/negative asymmetry ReSA replaces with the self-assignment gradient.","marker":"(Wang & Liu, 2021)"},{"why":"Shows weak augmentation helps relational SSL, the finding ReSA borrows for its clustering-view augmentation.","marker":"(Zheng et al., 2021)"},{"why":"Provides the INTL baseline and the exact optimizer settings ReSA uses for ConvNet pretraining.","marker":"(Weng et al., 2024)"}],"fun_headline_variants":["ReSA: self-assignment replaces identity in contrastive loss","Model's own clusters guide its self-supervised training","Self-supervised learning uses its clusters as a training signal","ReSA: let the model's own clusters train it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the encoder's output already carries a clustering structure stable and informative enough to serve as a training target, from random initialization onward; if early self-assignments are effectively identity matching, ReSA reduces to InfoNCE and the reported gains would disappear.","fun_headline_variants_meta":{"raw":{"variants":["ReSA: self-assignment replaces identity in contrastive loss","Model's own clusters guide its self-supervised training","Self-supervised learning uses its clusters as a training signal","ReSA: let the model's own clusters train it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000536,"raw_usage":{"total_tokens":2572,"prompt_tokens":941,"completion_tokens":1631,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":1564}},"tokens_in":557,"tokens_out":1631,"duration_ms":11648,"temperature":1.0,"reasoning_tokens":1564,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T23:24:54.000376+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ReSA on a standard dataset with the self-assignment matrix forced to the identity while keeping everything else identical; if this configuration still matches ReSA's reported accuracy, the clustering signal is not the driver. Conversely, compute the assignment matrix from a randomly initialized encoder, measure whether its off-diagonal entries agree with true class labels, and ablate those off-diagonal entries during training; if they carry no class information and their removal does not change performance, the paper's proposed mechanism is unsupported.","supporting_citations":[{"cited_title":"Reverse engineering self-supervised learning","cited_arxiv_id":null,"evidence_quote":"Establishes that SSL representations carry hierarchical, centroid-like semantic clusters, the property ReSA exploits."},{"cited_title":"Unsupervised learning of visual features by contrasting cluster assignments","cited_arxiv_id":null,"evidence_quote":"Supplies the online Sinkhorn clustering precedent, the Sinkhorn hyperparameters, and the SwAV baseline ReSA improves on."},{"cited_title":"Ressl: Relational self-supervised learning with weak augmentation","cited_arxiv_id":null,"evidence_quote":"Shows weak augmentation helps relational SSL, the finding ReSA borrows for its clustering-view augmentation."},{"cited_title":"M., Khan, S., Khan, F","cited_arxiv_id":null,"evidence_quote":"Provides the INTL baseline and the exact optimizer settings ReSA uses for ConvNet pretraining."}],"review_version":1}