{"id":"67877582-a79b-497c-96ca-52bc825d8b4d","arxiv_id":"2412.01926","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SSLPM, a self-supervised method that reduces redundancy by making features hard to predict from one another, matches state-of-the-art performance, but higher-order redundancy reduction does not clearly improve downstream accuracy.","lead":"Self-supervised learning models are often trained to remove redundancy between their learned features, but usually only pairwise redundancy. This paper defines more general redundancy measures, proposes a method (SSLPM) that removes redundancy by pitting an encoder against a predictor, and finds that while SSLPM matches top methods, removing more complex redundancy does not improve downstream performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The implicit-redundancy conclusion depends on cross-method comparability of a fixed-capacity redundancy estimator; Figure 7's weak correlations make this assumption load-bearing.","rationale":"The reader's weakest assumption matches the most load-bearing point: the headline interpretation about implicit redundancy reduction is only as strong as the cross-method comparability of the redundancy estimates. The method claim itself—SSLPM-RR is competitive with SOTA—is supported by Table 1 and the ablation in Table 2, so I do not see grounds to reject the paper. However, the 'implicit redundancy reduction' sentence in the abstract goes beyond the within-method evidence and relies on a small between-method comparison with a fixed estimator. Since Fig. 7 shows weak and inconsistent correlations across methods, and since no error bars are given for redundancy values, this specific interpretive claim should be treated as conditional. A secondary theoretical issue is that the proof of Corollary 3.5 overstates its counterexample: from Z=1/2(X^2-Y^2), a predictor given X and Z cannot determine the sign of Y, so NLR equals 1/3 rather than 1; the conclusion LR=0 does not imply NLR=0 still holds, but the proof as written is not correct. This does not change the empirical verdict, but it supports keeping the reader's CONDITIONAL rather than moving to ACCEPT.","tokens_in":20792,"tokens_out":4706,"duration_ms":144359,"concrete_test":"Recompute the cross-method redundancy rankings using an estimator whose capacity and training budget are tuned per method to a common tolerance: for each method, select MLP width/depth and number of steps by early stopping on a held-out split, and compute LR on all 512 features instead of 20%. If the methods that beat SSLPM-RR no longer all lie below it in redundancy (or if the ImageNet-100 negative correlations disappear), the implicit-redundancy claim in the abstract should be withdrawn or weakened to a within-method observation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's second claim—that high-performing SSL methods implicitly reduce redundancy because 'all methods outperforming SSLPM-RR exhibit less redundancy'—depends on redundancy being measured fairly across methods. The protocol in Appendix A fixes estimator capacity: LR uses ridge regression with a cross-validated penalty, NLR uses one MLP with two hidden layers (128, 64) and a fixed number of training steps, and both are evaluated on a random 20% feature subsample. A fixed estimator can systematically underfit activation distributions that are harder to regress (e.g., due to scale, sparsity, or higher-order interactions), so the measured redundancy is really a joint property of the embedding and the estimator. This is not a technicality: cross-method correlations are weak (CIFAR-10 LR r=-0.12, p=0.78; NLR r=-0.11, p=0.80; Fig. 7), and the paper's own Fig. 4 shows the redundancy-performance link is strong mainly within SSLPM-RR and Barlow Twins sweeps. The conclusion that non-redundancy-reducing methods perform implicit redundancy reduction rests on a few between-method comparisons with no reported error bars on redundancy, so estimator bias aligned with method family could produce exactly the observed pattern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes embedding-space redundancy via three measures: average absolute covariance (AAC), linear redundancy (LR), and nonlinear redundancy (NLR). It proves basic inequalities among them and proposes SSLPM, a self-supervised method that reduces redundancy through a competitive game between an encoder and a predictor trained on randomly masked representation features. Two variants are presented: SSLPM-RR (ridge-regression predictor) and SSLPM-SGD (MLP predictor). Experiments on CIFAR-10, CIFAR-100, and ImageNet-100 with ResNet-18 show that SSLPM-RR is competitive with Barlow Twins and other SSL baselines, that within methods with explicit redundancy reduction lower LR is associated with higher downstream accuracy, and that across methods the correlation is weak. Nevertheless, the authors claim that the best-performing methods exhibit lower redundancy and therefore perform implicit redundancy reduction.","tokens_in":21054,"tokens_out":9295,"duration_ms":84126,"significance":"The paper addresses a timely question and provides a useful taxonomy of redundancy measures. The theoretical inequalities are simple but clearly presented, and the SSLPM method is a sensible modern instantiation of predictability minimization. The negative result that removing higher-order redundancies does not improve downstream performance is interesting and potentially important. However, the current version contains a flawed proof of a separating corollary and an over-reaching interpretive claim about implicit redundancy reduction, partly contradicted by the authors' own data and by the fact that two of the cited methods (Barlow Twins and VICReg) perform explicit redundancy reduction. With corrections and robustness checks, the paper could be a valuable contribution.","major_comments":[{"comment":"The counterexample claims that 'given any two variables, a nonlinear predictor is able to perfectly reconstruct the third variable.' This is false for Y from (X,Z): the equations determine only Y^2, not the sign of Y, so the minimum MSE predictor is E[Y|X,Z]=0 with MSE 1, giving predictability 0. The same holds for X from (Y,Z). Therefore NLR([X,Y,Z]) is 1/3, not 1, and the proof does not establish the corollary. The corollary may still be true (e.g., let Z = (X^2-1)/sqrt(2) with X ~ N(0,1); then AAC = LR = 0 but NLR = 1/2), but the proof must be corrected.","section":"Corollary 3.5 proof (page 21)"},{"comment":"The sentence 'Given that these methods do not perform active redundancy reduction' is inaccurate: Barlow Twins and VICReg both impose explicit covariance/redundancy penalties, as described in Sections 1 and 2. Including them in the 'implicit' argument conflates explicit and implicit mechanisms. The abstract's more cautious wording ('even methods without explicit redundancy reduction mechanisms') should be aligned with the body; as written, Section 4.5 overstates the evidence for implicit redundancy reduction.","section":"Section 4.5"},{"comment":"The cross-method redundancy ranking assumes that a single estimator (ridge with cross-validated penalty for LR; a fixed two-hidden-layer MLP with fixed training steps for NLR on a 20% feature subsample) is equally fair to all embedding distributions. Since LR and NLR are defined as 1 minus the achieved normalized MSE, a method whose features are harder to fit with the chosen estimator will appear less redundant without necessarily being so. The paper's own Figure 7 shows no significant across-method correlation on CIFAR-10 (LR r=-0.12, p=0.78; NLR r=-0.11, p=0.80), so the subset claim in Section 4.5 is not backed by a significance test and could be an artifact of estimator capacity. Please add robustness analyses (vary estimator capacity, report redundancy standard errors, or use a distribution-free measure) before drawing the implicit-redundancy conclusion.","section":"Appendix A / Section 4.5"}],"minor_comments":[{"comment":"\"SiamSiam\" is a typo; the method is SimSiam (Chen & He).","section":"Related Work"},{"comment":"The line \"although LR and NLR are zero\" should read \"although AAC and LR are zero.\"","section":"Corollary 3.5 proof"},{"comment":"The computation of AAC for the constructed example uses notation \"(n-1 2)\" and \"(n 2)\" that appears to be a typesetting error; please clarify the combinatorial expressions.","section":"Section 3.3"},{"comment":"The y-axis label \"Final Epochs Predictability Loss\" conflicts with the definition of predictability as 1 - MSE; the plotted quantity appears to be the prediction error, so rename or clarify.","section":"Figure 3"},{"comment":"The text mentions \"P values of 0.79 and 0.93\" but Figure 7 reports p=0.78 (LR) and p=0.80 (NLR) for CIFAR-10; the text and figures should be consistent.","section":"Section 4.5"},{"comment":"Most baselines are single-seed runs; adding at least three seeds for the main baselines would strengthen the competitive claim, though this may be infeasible.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The central method contribution (SSLPM-RR) appears sound and competitive, and the redundancy-measure taxonomy is a useful framework. The main obstacles are the incorrect proof of Corollary 3.5 and the overstated implicit-redundancy interpretation, both fixable with careful revision. The cross-method comparability of the redundancy estimator deserves explicit robustness checks; without them, the headline claim about implicit redundancy reduction is not fully supported. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know three things about arXiv:2412.01926. First, it introduces a competitive new SSL method, SSLPM-RR: a Siamese setup where a closed-form ridge-regression predictor tries to reconstruct randomly masked features while the encoder tries to make prediction hard. It lands in the Barlow Twins/BYOL band on CIFAR-10 (92.70), CIFAR-100 (70.46), and ImageNet-100 (79.17), with three-seed runs and solid ablations on lambda, masking, ridge penalty, and batch size. Second, the formal hierarchy—AAC, LR, NLR with AAC ≤ √LR ≤ √NLR—is useful and mostly correct, and the empirical measurement protocol (held-out evaluation, feature subsampling, cross-validated ridge penalty) is careful. Third, the paper's interpretive edges are weaker than the abstract, and one proof is wrong as written.\n\nCredit where it is due: the separation between pairwise correlation and linear predictability (the example where AAC is O(n^{-3/2}) while LR is 1) is genuinely clarifying, and the projector-depth result—deeper projectors reduce LR/NLR in the embeddings, with SSLPM-RR more robust to projector depth than Barlow Twins—is a real empirical finding. The null result for higher-order redundancy reduction is worth reporting even though the experiment is confounded.\n\nSoft spots, in proportion. Corollary 3.5's proof is incorrect. In the Gaussian counterexample, from X and Z you can recover Y^2 but not the sign of Y, so the optimal predictor of Y is 0 and NLR is 1/3, not 1. The corollary survives—LR=0 genuinely does not imply NLR=0, and a clean NLR=1 example exists with Rademacher variables X_3 = X_1 X_2—but the proof as printed needs fixing.\n\nThe 'implicit redundancy reduction' conclusion is thin and somewhat load-bearing. Cross-method correlations are non-significant on CIFAR-10 (LR p=0.78, NLR p=0.80) and CIFAR-100; redundancy is measured once per model with a fixed-capacity MLP, so the estimator is part of the measurement. The paper's Section 4.5 text hedges properly, but the abstract sells it as a finding. Also, the claim that all methods outperforming SSLPM-RR show less redundancy is sloppy as stated because Barlow Twins does not outperform SSLPM-RR on CIFAR-10.\n\nThe negative result for higher-order redundancy is confounded: SSLPM-SGD with 2–4 layer predictors removed less redundancy, not more, so the manipulation never achieved stronger redundancy reduction. The paper half-acknowledges this but then overclaims 'evidence to the contrary.' Measuring SSLPM-RR with LR, the same linear predictability it optimizes, gives a mild by-construction flavor; a sentence of caveat would do.\n\nWho this is for: SSL practitioners and anyone interested in the geometry of learned embeddings. My recommendation: send it to serious peer review. The method claims are credible, the hierarchy is worth having in the literature, and the errors are local and fixable. I would bring it to reading group.","headline":"A solid SSL method paper with a useful redundancy hierarchy and a fixable proof error, wrapped in an over-strong interpretive story.","tokens_in":21597,"tokens_out":8976,"would_cite":true,"duration_ms":79486,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that SSLPM, a self-supervised method that minimizes redundancy by making masked features hard to predict, is competitive with state-of-the-art SSL methods, and that the best-performing methods exhibit low embedding…","keywords":["self-supervised learning","redundancy reduction","predictability minimization","higher-order dependencies","linear redundancy","nonlinear redundancy","masked prediction","representation learning"],"falsifier":"Take the same set of trained SSL models and re-estimate LR and NLR using all 512 embedding features rather than a 20% subsample, with per-model capacity-tuned nonlinear estimators; if BYOL, VICReg, and NNCLR no longer have lower redundancy than SSLPM-RR, or if the redundancy-accuracy correlations reverse, the implicit-redundancy conclusion collapses.","tokens_in":20566,"feed_emoji":"🧠","tokens_out":7315,"duration_ms":59453,"temperature":0.7,"pith_summary":"The paper defines embedding-space redundancy beyond pairwise correlations: average absolute covariance (AAC), linear redundancy (LR), and nonlinear redundancy (NLR), and proves the hierarchy $AAC \\le \\sqrt{LR} \\le \\sqrt{NLR}$. On top of this, it introduces SSLPM, a self-supervised method in which a predictor tries to reconstruct randomly masked representation features and the encoder tries to make that prediction hard. SSLPM-RR, the variant with a closed-form ridge-regression predictor, reaches accuracy competitive with Barlow Twins and other leading SSL methods on CIFAR-10, CIFAR-100, and ImageNet-100. The paper's broader claim is that the best-performing SSL methods have low embedding redundancy even when they do not explicitly penalize redundancy, suggesting that high-quality representations reduce redundancy implicitly.","feed_headline":"SSL that strips feature redundancy matches top rivals","feed_subtitle":"A predictor-encoder game removes hidden dependencies, and the best SSL models appear to do it implicitly.","key_machinery":"The central objects are three redundancy measures computed on standardized embedding features: AAC is the mean absolute pairwise covariance; LR is $1$ minus the minimal squared error of a linear (ridge) regression predicting each feature from the others; NLR is $1$ minus the minimal squared error of an MLP predicting each feature from the others. The paper proves these form a hierarchy $AAC \\le \\sqrt{LR} \\le \\sqrt{NLR}$, with zero AAC equivalent to zero LR but not to zero NLR, illustrated by a Gaussian counterexample. SSLPM combines Barlow Twins' invariance loss with a masked-prediction loss: a Bernoulli mask hides roughly half the features of both augmented representations, a predictor reconstructs the hidden ones from the visible ones, and the encoder is updated to make reconstruction hard; SSLPM-RR solves the predictor in closed form with ridge regression.","core_discovery":"On its own terms, the paper establishes that redundancy reduction can be lifted from pairwise covariance to general predictability: a representation has high redundancy when one or more of its features can be reconstructed from the rest, and SSLPM removes that redundancy by forcing an encoder to defeat a predictor trained on randomly masked features. The theoretical contribution is the inequality chain $AAC \\le \\sqrt{LR} \\le \\sqrt{NLR}$ for standardized features, with proofs that linear and nonlinear predictability capture multi-feature and nonlinear interactions that pairwise covariance can miss, while corollaries show zero AAC is equivalent to zero LR but not to zero NLR. Empirically, SSLPM-RR attains 92.70 Top-1 on CIFAR-10, 70.46 on CIFAR-100, and 79.17 on ImageNet-100, on par with Barlow Twins and above several contrastive baselines; the authors find no evidence that removing higher-order redundancy helps downstream performance, and they find that methods outperforming SSLPM-RR show lower measured redundancy, which they interpret as implicit redundancy reduction.","pith_inferences":["Inference: If implicit redundancy reduction is what separates strong SSL embeddings from weak ones, then the redundancy term in SSLPM-RR may be replaceable by architectural choices such as a deeper projector; a direct test would be whether a lambda=0 SSLPM with a deep projector closes most of the gap to SSLPM-RR.","Inference: The closed-form ridge predictor makes the redundancy-reduction step cheap and stable, suggesting that other SSL objectives with quadratic prediction losses could use the same trick rather than training a learned predictor.","Inference: The paper's measures are computed on the final embedding layer of frozen encoders; measuring redundancy at intermediate layers or with different estimator capacities could change the method ranking, so the implicit-redundancy claim has a measurement-dependence worth probing.","Inference: The authors' observation that too much redundancy reduction collapses performance suggests some residual redundancy is useful; connecting this to information-bottleneck trade-offs is a natural next step."],"forward_implications":["SSLPM-RR is competitive with state-of-the-art SSL methods on CIFAR-10, CIFAR-100, and ImageNet-100, despite replacing pairwise decorrelation with a closed-form ridge-regression predictability loss.","Reducing nonlinear, higher-order redundancy via multilayer predictors does not improve downstream accuracy and can hurt it, so the practical value of redundancy reduction currently lies mainly in the linear regime.","The best-performing methods in the study, including BYOL, Barlow Twins, VICReg, and NNCLR, all have lower measured embedding redundancy than SSLPM-RR, which the paper reads as evidence for implicit redundancy reduction in high-quality SSL.","Deepening the projector reduces both linear and nonlinear embedding redundancy, making projector architecture a lever for implicit decorrelation.","Across SSL methods generally, redundancy and accuracy are only weakly correlated; the strong negative correlation appears specifically in methods with explicit redundancy reduction."],"supporting_citations":[{"why":"Supplies the predictability-minimization idea that SSLPM turns into a masked-feature prediction game.","marker":"Schmidhuber et al. (1996)"},{"why":"Defines Barlow Twins, the pairwise-redundancy baseline and the source of SSLPM's invariance loss.","marker":"Zbontar et al. (2021)"},{"why":"BYOL is a high-performing baseline that, without explicit redundancy reduction, shows lower redundancy and is used for the implicit-redundancy claim.","marker":"Grill et al. (2020)"},{"why":"SimCLR provides the contrastive SSL baseline that SSLPM-RR must match.","marker":"Chen et al. (2020)"},{"why":"NNCLR is a high-performing baseline used in the redundancy-versus-performance comparisons.","marker":"Dwibedi et al. (2021)"},{"why":"MoCoV3 is a contrastive baseline in the comparison table.","marker":"He et al. (2020); Chen et al. (2021)"},{"why":"VICReg is a variance-invariance-covariance baseline whose low redundancy supports the implicit-redundancy reading.","marker":"Bardes et al. (2021)"},{"why":"VIbCReg is an additional covariance-based baseline in the empirical comparison.","marker":"Lee & Aune (2021)"},{"why":"Provides the training and evaluation methodology used for all compared models.","marker":"Da Costa et al. (2022)"}],"fun_headline_variants":["New SSL game strips hidden redundancy, rivals top methods","Beyond pairwise: SSL tackles higher-order redundancy","Predictor-encoder duel reduces redundancy in SSL","Implicit redundancy reduction found in top SSL models","SSL method reduces higher-order redundancy, matches benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The implicit-redundancy finding assumes that the fixed linear and MLP estimators used on a 20% feature subsample measure redundancy fairly across models with different activation distributions; if that estimator favors some models, the redundancy ranking, and with it the central interpretation, could be an artifact.","fun_headline_variants_meta":{"raw":{"variants":["New SSL game strips hidden redundancy, rivals top methods","Beyond pairwise: SSL tackles higher-order redundancy","Predictor-encoder duel reduces redundancy in SSL","Implicit redundancy reduction found in top SSL models","SSL method reduces higher-order redundancy, matches benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000364,"raw_usage":{"total_tokens":1951,"prompt_tokens":923,"completion_tokens":1028,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":957}},"tokens_in":539,"tokens_out":1028,"duration_ms":9110,"temperature":1.0,"reasoning_tokens":957,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:02:17.631798+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same set of trained SSL models and re-estimate LR and NLR using all 512 embedding features rather than a 20% subsample, with per-model capacity-tuned nonlinear estimators; if BYOL, VICReg, and NNCLR no longer have lower redundancy than SSLPM-RR, or if the redundancy-accuracy correlations reverse, the implicit-redundancy conclusion collapses.","supporting_citations":[],"review_version":1}