{"id":"84bc83b1-153b-4562-971c-e3c9f93c95be","arxiv_id":"2412.02121","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Adding progressive pseudo-label classification to SSL frameworks, framed as instantiating the unique information component of PID, yields small accuracy improvements over four SSL baselines.","lead":"This paper reframes self-supervised learning (SSL) through partial information decomposition (PID) and adds a pseudo-label classification loss to existing SSL models, claiming it captures the unique information component. The authors report small accuracy gains (about 1-3%) on four SSL baselines and four image datasets, but the theoretical link between the loss and PID is asserted rather than derived.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 2's shared pseudo-label for both views is a redundant-information objective, not a unique-information one; since the PID claim rests on this mapping, the central theoretical contribution is unsupported.","rationale":"The reader's rejection is supported. The central claim is twofold: (1) PID resolves the mutual-information debate, and (2) adding L_PS extracts the unique PID component. The paper gives no formal specification of what unique information means in its SSL system, selects no redundancy measure, estimates no PID quantity, and provides no derivation from Eq. (2) to U(T; S1) + U(T; S2). The concern is stronger than 'unformalized': the specified loss assigns the same pseudo-label to both views, so any label-predictive signal learned by both views is, by the PID definitions, redundant information. Unique information would require an objective that rewards one view for carrying target-relevant information that is absent from the other view. The empirical gains are small, consistent, and exactly what cluster-based pseudo-labeling methods typically produce, so they do not independently validate the PID mechanism. The ablation shows sensitivity to choosing the true number of classes as the cluster count, which raises label-leakage concerns, and the absence of DeepCluster/SwAV-style baselines weakens the empirical comparison. Credit is due for testing four baselines across four datasets and for the progressive-weighting ablation, but those support an engineering contribution, not the claimed scientific resolution. Since the reader already recommended REJECT, the verdict remains unchanged.","tokens_in":14927,"tokens_out":4904,"duration_ms":55148,"concrete_test":"On CIFAR100 with B-Twins as the base model, train the 1000-epoch baseline and the upgraded pipeline; at checkpoints (e.g., epochs 200, 500, 1000), quantize z1, z2, and pseudo-labels Y into finite alphabets and compute a concrete PID measure (e.g., I_min or BROJA) for the three-variable system with S1 = z1, S2 = z2, T = Y. If the upgraded model does not increase U(T; S1) + U(T; S2) relative to baseline, or if R(T; S1, S2) increases instead, then Eq. (2) is not extracting unique information. Decision rule: an accuracy improvement without a corresponding increase in measured unique information refutes the paper's central theoretical claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4/Eq. (2) defines the proposed unique-information term as L_PS(z1, yhat) + L_PS(z2, yhat), where 'yhat is the pseudo-label for both views of a sample'. In Williams-Beer PID (Eq. 1), U(T; S_i) is information about target T available from source S_i but not from the other source. A loss that trains both z1 and z2 to predict the same cluster label encourages the two views to carry the same label-discriminative signal; under the PID definitions, such shared signal contributes to the redundant component R(T; S1, S2), not to U(T; S1) + U(T; S2). The fact that z1 and z2 are classified separately does not make the information unique, because uniqueness is defined relative to the other source, not by the number of forward passes. No derivation, no chosen PID measure (I_min, I_broja, etc.), and no estimator connects Eq. (2) to unique information, and no PID quantity is ever computed from the trained representations. If anything, the mechanism is closer to a standard cluster-assignment objective (DeepCluster/SwAV style), whose gains are explainable without PID. The paper's claim to resolve the mutual-information debate therefore rests on an asserted identification that is not only unproven but appears to instantiate the opposite component.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the debate about whether self-supervised learning (SSL) should increase or decrease mutual information between augmented views can be resolved by reformulating SSL within the framework of partial information decomposition (PID), using joint mutual information I(S1,S2;T) instead of pairwise mutual information. The authors propose a general pipeline that adds a progressively weighted pseudo-label classification loss L_PS to any existing SSL loss L_SSL, claiming that L_PS extracts the unique information component of PID while L_SSL covers redundant and synergistic components. They report improved linear and k-NN evaluation accuracy over four SSL baselines (SimCLR, BYOL, WMSE, Barlow Twins) on CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet, plus transfer-learning results.","tokens_in":15241,"tokens_out":3446,"duration_ms":34573,"significance":"If the core PID claim were correct, the paper would offer a principled resolution to a long-standing debate in SSL and a generic plug-in upgrade for existing methods. The experimental effort is substantial: four baselines, four datasets, linear and k-NN evaluation, transfer learning, and an ablation study. The paper also publicly builds on the Solo-Learn library, which aids reproducibility. However, the central theoretical mapping from the proposed loss to unique information is asserted rather than derived, and the specific loss appears to instantiate the opposite PID component (redundancy). The empirical gains are modest and are reported without error bars or multiple seeds. Because the theoretical contribution is load-bearing and unsupported, the paper as it stands does not establish its advertised claims.","major_comments":[{"comment":"The central claim that L_PS extracts the unique information component of PID is unsupported and appears inconsistent with the Williams-Beer PID definition in Eq. (1). In Eq. (2), the same pseudo-label ŷ is used for both views, so the loss trains both z1 and z2 to predict the same cluster. In PID, information about the target T that is available from both sources S1 and S2 is precisely the redundant component R(T;S1,S2), not the sum of unique components U(T;S1)+U(T;S2). The fact that z1 and z2 are classified separately does not make the information unique, because uniqueness is defined relative to the other source. No derivation, no specific PID measure (e.g., I_min or I_broja), and no estimator is provided to connect Eq. (2) to unique information, and no PID quantity is computed from the learned representations in the experiments. This is load-bearing: the paper's advertised contribution is the extraction of unique information, and the proposed mechanism appears to target redundant information instead.","section":"Section 3.4, Eq. (2)"},{"comment":"The intra-alignment of any valid SSL baseline with the information bottleneck principle is asserted without proof or a supporting citation: the text states 'since the first term of our loss function, L_SSL is already taken from a valid SSL framework that follows the IB principle (e.g., Barlow-Twins, SimCLR, ...)'. This premise is not demonstrated, and it is not obvious that arbitrary SSL losses (including contrastive, non-contrastive, and whitening losses) each correspond to the same IB objective with a fixed trade-off parameter β. Because the IB-alignment argument underlies the claimed theoretical resolution, this assertion needs to be made precise or replaced with a citation to a formal equivalence result.","section":"Section 3.5, Eq. (3)"},{"comment":"The specification of redundancy as feature correlation and synergy as 'pairwise complementariness' of learned features is ad hoc and is never connected to any formal PID measure. The paper does not estimate R, Sy, or U from the learned representations of any trained model, nor does it compare the proposed method against these estimates. Consequently, the central claim that existing SSL frameworks only use redundant and synergistic information while missing unique information is not empirically verified; it remains an unquantified metaphor rather than an information-theoretic analysis.","section":"Appendix B"},{"comment":"The experimental results are reported for single runs without error bars, multiple seeds, or significance tests. The reported improvements range from about 0.74% to 3.3% in top-1 accuracy, and run-to-run variance in SSL pretraining over 1000 epochs can easily be on this scale. The claim of generality across four baselines and four datasets is therefore not statistically established. In addition, key hyperparameters (the α schedule, the number of clusters K, and the re-clustering period) are chosen per-dataset, and the sensitivity analysis in Section 5 only varies K on CIFAR-100; the other choices are not ablated.","section":"Tables 1-3"},{"comment":"The proposed method is conceptually close to clustering-based SSL methods such as DeepCluster and SwAV, which also assign pseudo-labels from clustered features and train to predict those labels, yet the paper does not compare against any such method. The caption of Fig. 1 asserts that the framework 'differentiates itself from paradigms including clustering in SSL by joint invariance enforcement to augmented representations', but no experiment or analysis supports that distinction. Without a comparison to clustering-based baselines, the empirical contribution over the existing pseudo-labeling family is not established.","section":"Section 3 and Fig. 1 caption"}],"minor_comments":[{"comment":"Figures are referenced as 'Fig. ??' in several places, indicating missing figure references that should be fixed.","section":"Sections 3 and 4.1"},{"comment":"The sentence 'Following original implementation of SimCLR [5], we set τ = 0.1' is ambiguous: τ is used earlier for augmentation operations, and here it presumably refers to the SimCLR temperature parameter; please clarify the notation.","section":"Section 4.1"},{"comment":"There are typos such as 'prin ciple' and 'LP Senables' that should be corrected.","section":"Section 3.5"},{"comment":"The ablation text says improvements 'drop to 01.18%' for K=50; the leading zero is a formatting error.","section":"Section 5B"},{"comment":"The number of clusters is set to the number of ground-truth classes in the dataset, which uses label information and should be discussed as a limitation; the sensitivity analysis only examines K on CIFAR-100, not on the other datasets.","section":"Section 3.3 and 5B"}],"recommendation":"reject","confidential_remarks":"The paper's own Sec. 3.4 states that L_PS 'enables the model to learn representation via extracting the unique information of each view (unique component of PID)', but Eq. (2) uses the same pseudo-label for both views, which under standard PID is a redundant-information objective. This is a load-bearing inconsistency, not a mere presentation issue. The paper also cites the authors' prior work [23] on joint mutual information for SSL without clearly delineating the incremental contribution, which may be worth flagging to the editor. The experimental results are internally consistent but lack the statistical rigor needed to support the generality claim. I recommend rejection, as the central theoretical claim is not defensible within the manuscript's current scope; a rewrite that drops the PID-unique-information interpretation and reframes the method as a pseudo-labeling extension with a weaker information-theoretic heuristic could be resubmitted as a new paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper stacks a progressive pseudo-label classification loss onto SSL baselines, names the added term 'unique information' under PID, and reports small consistent gains. The gains look real. The theory does not.\n\nWhat's new: applying PID vocabulary to SSL, and reframing the mutual-information debate as a question about joint mutual information, is a fresh way to present the problem. The pipeline is clean and modular: train any SSL baseline for 100 epochs, k-means++ the projector features, then add a classification loss on each view with the same pseudo-label, ramping alpha over training. Results across SimCLR, BYOL, W-MSE, and B-Twins on four datasets show 1-3% top-1 gains, plus an honest ablation on cluster number and re-clustering frequency. That is a solid engineering contribution.\n\nThe soft spot is load-bearing. Section 3.4 claims L_PS captures the unique component because 'it learns from classification of views individually and independently.' But both views are trained against the same pseudo-label. In Williams-Beer PID, shared label information is redundant, not unique; uniqueness is defined by what one source provides that the other does not. The paper never chooses a specific PID measure (I_min, I_broja, etc.), never estimates any PID quantity, and never connects Eq. (2) to Eq. (1). The identification appears to instantiate the opposite component. That collapses the central claim that PID resolves the MI debate. What remains is a cluster-assignment objective close in spirit to DeepCluster or SwAV, which are cited but not run as baselines. The use of true class counts as K also leaks label information, though the K=50/150 ablation partially addresses it.\n\nMinor problems: no error bars or multiple seeds; IB alignment is assumed from the baseline rather than shown; missing figure references and typos throughout. All fixable.\n\nBottom line: as an empirical hack it is believable and mildly useful; as a theoretical contribution it is unsupported. I would still send it to peer review, because the topic is timely and a referee could either demand a real PID derivation/estimation or push the authors to reframe as a clustering method. I expect rejection as written, but the review would be informative. I would not cite it in my own work.","headline":"Pseudo-label supervision gives real but modest gains; the PID 'unique information' claim is asserted, not derived, and points at the wrong component.","tokens_in":15717,"tokens_out":3374,"would_cite":false,"duration_ms":34329,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","94A15","62H30"],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a progressive pseudo-label loss to any SSL baseline improves its representations by capturing the unique-information component of partial information decomposition.","keywords":["self-supervised learning","partial information decomposition","joint mutual information","unique information","pseudo-labeling","progressive supervision","representation learning","clustering"],"falsifier":"Estimate the three PID components directly from the learned representations (for example, using a deterministic redundancy measure such as the Williams-Beer I-min formulation) for a baseline and for the upgraded model. If the upgraded model's accuracy gain is not accompanied by a measurable increase in the unique information attributed to each view, the central claim is falsified. A simpler control: train the pipeline with randomly shuffled pseudo-labels; if the accuracy improvement persists, it cannot be attributed to unique-information extraction from real cluster structure.","tokens_in":14730,"feed_emoji":"🧠","tokens_out":4977,"duration_ms":44587,"temperature":0.7,"pith_summary":"This paper argues that the long-standing dispute over whether self-supervised learning should increase or decrease mutual information between augmented views is a false dichotomy. The authors recast SSL as a three-variable information system and apply partial information decomposition (PID), which splits joint mutual information into redundant, synergistic, and unique components. They claim that ordinary SSL objectives only exploit the redundant and synergistic parts, leaving the unique information of each individual view unused. To fix this, they add a progressively re-weighted pseudo-label classification loss, generated by k-means++ clustering, to any existing SSL baseline. Experimental results across four baselines and four datasets show consistent accuracy gains, which the paper takes as evidence that the unique-information component is being captured.","feed_headline":"Cluster pseudo-labels boost every SSL baseline they test","feed_subtitle":"A progressive pseudo-label loss aims to capture the unique-information component of PID, lifting accuracy on four datasets.","key_machinery":"The load-bearing identity is the Williams-Beer decomposition of joint mutual information, $I(S_1,S_2;T) = R + Sy + U(S_1) + U(S_2)$, which separates information about a target $T$ into redundant, synergistic, and unique parts. In this paper the two augmented views are the sources and the true data distribution is the target. The mechanism claimed to realize the unique components $U(S_1)$ and $U(S_2)$ is the pseudo-label cross-entropy loss $L_{PS}$ applied to each view's projector output, balanced against the original SSL loss by a progressively growing coefficient $\\alpha$. The progressive re-clustering with k-means++ is what turns the pseudo-labels from a static signal into a co-evolving supervisory signal.","core_discovery":"The paper's central claim is that the correct objective for SSL is not maximization or minimization of pairwise mutual information, but the joint utilization of all three PID components of the joint mutual information between two augmented views and the target representation. Existing SSL methods learn from two views jointly, thereby touching only synergy and redundancy; the unique information each view carries about the target is never supervised. The proposed pipeline adds a classification loss on each view's projection against pseudo-labels obtained by k-means++ on the current features, with a weight that grows during training and with pseudo-labels refreshed every 100 epochs. According to the paper, this loss lets each view be supervised individually and independently, which instantiates the missing unique component, and the progressive refinement turns sample-level supervision (local clustering) into cluster-level supervision (global clustering). The consistent improvements across SimCLR, BYOL, Whitening-MSE, and Barlow Twins on CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet are presented as empirical support for this interpretation.","pith_inferences":["An immediate testable extension is to compute the Williams-Beer PID quantities on the learned features of a baseline and of the upgraded model; the paper's interpretation predicts a measurable increase in unique information $U(S_1)+U(S_2)$ rather than only in redundancy or synergy.","If the PID mapping is the true source of the gains, then the same progressive pseudo-labeling recipe should transfer to non-visual SSL domains, such as speech or text, where views and clusters can be defined by augmentation and clustering respectively.","The paper's framing suggests a design principle for future SSL losses: any objective that supervises each view independently against a co-evolving target should inject unique information, which may explain why semi-supervised hybrids already showed such gains.","A null variant that shuffles pseudo-labels across samples would isolate whether the improvement comes from cluster structure or merely from an extra classification head; the paper does not report this control."],"forward_implications":["Applying the progressive pseudo-label pipeline to SimCLR, BYOL, Whitening-MSE, and Barlow Twins improves both linear and k-NN top-1 accuracy on CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet.","The gains are larger under k-NN evaluation than under linear evaluation, which the paper reads as evidence that the learned features are more task-relevant rather than merely better aligned with a linear head.","Progressive re-labeling matters: holding the pseudo-labels fixed for the entire second phase cuts the average improvement roughly in half, from 1.59% to 0.98% on CIFAR-100.","The number of clusters affects the gain: using the true class count works best, while overshooting the cluster count reduces the improvement more than undershooting it.","The total loss is argued to align with the information bottleneck principle at two levels: the SSL term balances augmentation removal against target information, and the pseudo-label term re-introduces global association."],"supporting_citations":[{"why":"Supplies the partial information decomposition with non-negative unique, redundant, and synergistic components that forms the theoretical lens of the paper.","marker":"[29]"},{"why":"Provides the SimCLR baseline and the standard augmentation protocol used to generate the two augmented views.","marker":"[5]"},{"why":"Provides the BYOL baseline, one of the four SSL frameworks the proposed pipeline upgrades.","marker":"[12]"},{"why":"Provides the Whitening-MSE baseline and the prior focus on redundancy reduction that the paper extends by adding the unique component.","marker":"[10]"},{"why":"Provides the Barlow Twins baseline, the redundancy-reduction perspective, and the projector architecture used in the experiments.","marker":"[30]"},{"why":"Provides the k-means++ algorithm used to generate pseudo-labels and to re-cluster during progressive supervision.","marker":"[1]"},{"why":"Represents the mutual-information-maximization side of the SSL debate that the paper claims to resolve with joint mutual information.","marker":"[25]"},{"why":"Represents the opposing InfoMin view that mutual information between views should be decreased, which the PID framework subsumes.","marker":"[26]"}],"fun_headline_variants":["Unique info from pseudo-labels lifts all SSL baselines","SSL rethought: PID says keep unique view info","Pseudo-labels unlock unique information for SSL","Progressive pseudo-labels improve four SSL methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes, without a derivation or direct estimation, that the per-view pseudo-label classification loss $L_{PS}$ equals the unique-information component $U$ of the Williams-Beer PID; if that mapping fails, the theoretical resolution of the mutual-information debate does not follow and the method remains an empirical pseudo-labeling extension.","fun_headline_variants_meta":{"raw":{"variants":["Unique info from pseudo-labels lifts all SSL baselines","SSL rethought: PID says keep unique view info","Pseudo-labels unlock unique information for SSL","Progressive pseudo-labels improve four SSL methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1326,"prompt_tokens":966,"completion_tokens":360,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":298}},"tokens_in":582,"tokens_out":360,"duration_ms":3944,"temperature":1.0,"reasoning_tokens":298,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:47:38.454107+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Estimate the three PID components directly from the learned representations (for example, using a deterministic redundancy measure such as the Williams-Beer I-min formulation) for a baseline and for the upgraded model. If the upgraded model's accuracy gain is not accompanied by a measurable increase in the unique information attributed to each view, the central claim is falsified. A simpler control: train the pipeline with randomly shuffled pseudo-labels; if the accuracy improvement persists, it cannot be attributed to unique-information extraction from real cluster structure.","supporting_citations":[{"cited_title":"A simple framework for contrastive learning of visual representations","cited_arxiv_id":null,"evidence_quote":"Provides the SimCLR baseline and the standard augmentation protocol used to generate the two augmented views."},{"cited_title":"Bootstrap your own latent-a new approach to self-supervised learning","cited_arxiv_id":null,"evidence_quote":"Provides the BYOL baseline, one of the four SSL frameworks the proposed pipeline upgrades."},{"cited_title":"Whitening for self-supervised representation learning","cited_arxiv_id":null,"evidence_quote":"Provides the Whitening-MSE baseline and the prior focus on redundancy reduction that the paper extends by adding the unique component."},{"cited_title":"Barlow twins: Self-supervised learning via redundancy reduction","cited_arxiv_id":null,"evidence_quote":"Provides the Barlow Twins baseline, the redundancy-reduction perspective, and the projector architecture used in the experiments."},{"cited_title":"K-means++ the ad- vantages of careful seeding","cited_arxiv_id":null,"evidence_quote":"Provides the k-means++ algorithm used to generate pseudo-labels and to re-cluster during progressive supervision."},{"cited_title":"Decomposed mutual information estimation for contrastive representation learning","cited_arxiv_id":null,"evidence_quote":"Represents the mutual-information-maximization side of the SSL debate that the paper claims to resolve with joint mutual information."},{"cited_title":"What makes for good 9 views for contrastive learning? Advances in Neural Infor- mation Processing Systems, 33:6827–6839, 2020","cited_arxiv_id":null,"evidence_quote":"Represents the opposing InfoMin view that mutual information between views should be decreased, which the PID framework subsumes."}],"review_version":1}