Improving Pre-trained Self-Supervised Embeddings Through Effective Entropy Maximization

Deep Chakraborty; Erik Learned-Miller; Tim G. J. Rudner; Yann LeCun

REVIEW 4 major objections 6 minor 15 references

Improving Pre-trained Self-Supervised Embeddings Through Effective Entropy Maximization

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes E2MC: ten epochs of continued pre-training that maximize one-dimensional marginal entropies and decorrelate features improve already-trained self-supervised embeddings.

desk verdict A cheap 10-epoch add-on that can improve SSL embeddings in label-scarce regimes, but the headline gains are partly selected on the same 1%-label metric, so the robustness claim is weaker than the abstract suggests. read the letter →

arxiv 2411.15931 v2 pith:EATS47GN submitted 2024-11-24 cs.LG cs.CVcs.ITmath.ITstat.APstat.ML

Deep Chakraborty , Yann LeCun , Tim G. J. Rudner , Erik Learned-Miller This is my paper · ORCID

classification cs.LGcs.CVcs.ITmath.ITstat.APstat.ML

keywords self-supervisedlearningentropymaximizationcontinuedpre-trainingmarginalcovarianceregularizationm-spacingsestimatorembeddinguniformityImageNetlinearevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a small add-on to self-supervised learning (SSL) objectives, called the effective entropy maximization criterion (E2MC), and claims that ten epochs of continued pre-training with it consistently improves already-trained SSL embeddings. Rather than estimating entropy in the full embedding space, where sample-based estimates break down, the criterion enforces two cheap low-dimensional conditions: each one-dimensional marginal of the transformed embedding should be uniform, and every pair of coordinates should be decorrelated. Applied on top of VICReg, SwAV, or SimSiam, the update improves ImageNet linear-evaluation accuracy, with the largest gains in label-scarce 1% settings, and transfers to iNaturalist. The paper further shows that continuing with the base loss alone, or with alternative uniformity criteria, does not give comparable gains.

What carries the argument

The load-bearing object is the pair of low-dimensional statistics applied to compact-space-transformed embeddings. For unnormalized embeddings (VICReg) the compact transform is the sigmoid; for hypersphere-normalized embeddings (SwAV, SimSiam) it is the CDF of a standard Gaussian, chosen so that uniform marginals in the transformed space correspond to standard-normal marginals before normalization. Marginal entropy is estimated with the m-spacings estimator, a differentiable one-dimensional entropy estimate, and decorrelation is enforced by the squared Frobenius norm of the off-diagonal sample covariance matrix. The argument is that these are the only statistics one can reliably estimate from finite batches in high dimensions, while higher-order joint uniformity appears anyway in practice.

What would settle it

Train multiple E2MC continued-pretraining runs for VICReg, SwAV, and SimSiam with several random seeds while holding $\beta$ and $\gamma$ fixed, then evaluate on the 10% and 100% ImageNet splits and on transferring to iNat18; if the 1%-selected hyperparameters do not reproduce a consistent improvement outside the 1% split, or if seed-to-seed variance matches or exceeds the reported gaps, the central claim of consistent improvement would fail.

Watch

Extended reading notes

Core claim

The central claim is that maximizing a set of necessary-but-not-sufficient conditions for maximum entropy—uniform one-dimensional marginals plus pairwise decorrelation—is a practical way to increase the usefulness of an existing SSL embedding. The authors define $L = L_{\text{SSL}} - \beta L_{\text{entropy}} + \gamma L_{\text{covariance}}$, where $L_{\text{entropy}}$ is the average m-spacings entropy estimate of each marginal after transforming embeddings to a compact space, and $L_{\text{covariance}}$ penalizes squared off-diagonal sample covariances. They report that after ten epochs of continued pre-training, VICReg, SwAV, and SimSiam embeddings improve on ImageNet linear and semi-supervised evaluation and on transfer tasks, and they present ablations showing that neither entropy alone nor covariance alone reproduces the gains.

Load-bearing premise

The load-bearing premise is that the hyperparameters $\beta$ and $\gamma$, and the 10-epoch stopping point, which were selected using the 1%-label ImageNet linear-evaluation metric, are not overfit to that metric, and that the reported continued-pretraining runs are representative.

Editorial extensions

If this is right

Ten additional epochs of E2MC continued pre-training can be run on a single GPU in about one day, so the method offers a cheap way to squeeze further downstream accuracy from already-released SSL checkpoints.
The improved 400-epoch SwAV model approaches the linear-evaluation accuracy of the 800-epoch base model, suggesting E2MC may shorten the pre-training horizon needed for a target accuracy.
Gains concentrate in label-deficient regimes, such as the 1% ImageNet setting, which is where linear probes and fine-tuning have the least supervised signal.
Entropy and covariance components are both necessary: the paper's ablation shows that dropping either one removes most or all of the improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the criterion is expressed as generic constraints on marginal distributions, it could be bolted onto any joint-embedding objective, including transformer-based or multimodal SSL methods; the paper tests only ResNet-50 backbones with VICReg, SwAV, and SimSiam.
The 10-epoch stopping rule and $\beta, \gamma$ values were selected on the 1%-label ImageNet linear-evaluation accuracy; a stricter evaluation would reserve that metric for model selection and then report all other label fractions and transfer sets as held out.
The reported emergence of near-uniform two-dimensional marginals, though not explicitly enforced, suggests a possible line of work on why deep networks avoid high-order dependence structures like the 'X' distribution; testing this could clarify when the necessary conditions become sufficient in practice.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a useful, well-run empirical paper with a real caveat. The criterion itself is genuinely new: instead of estimating high-dimensional joint entropy, it maximizes 1D m-spacings entropy and minimizes pairwise covariance on compact-space-transformed embeddings, then uses that as a continued-pretraining add-on. The Gaussian-CDF transform for hypersphere-normalized models is a smart detail. The ablations against base-loss continuation, VCReg, AUH, and MMCR are the right comparisons, and they support the attribution of gains to E2MC. The statistical tests in Appendix D.1 are a plus, and code is released.

The soft spot is evaluation protocol, and it is the load-bearing one. Beta, gamma, and the 10-epoch stopping point were all chosen by searching on the 1%-label ImageNet linear-evaluation metric, which is exactly the setting where the headline gains appear (SwAV-800 goes from 53.70 to 55.27). The reported error bars come from retraining the linear classifier only, not from independent continued-pretraining runs; the significance test in D.1 picks a single post hoc model. At 10% and 100% labels and on transfer, gains mostly shrink to 0.1-0.25 points, and some are negative (VOC07 for SimSiam and SwAV-400). SimSiam is essentially flat. So the abstract's "consistent and significant improvement" is too strong. The evidence actually supports: E2MC can give a real gain at 1% labels when tuned, occasionally helps elsewhere, and does not seem to hurt much. That is still a useful result.

Two minor things. The "surprising" uniformity of 2D marginals is shown visually but not quantified, so it reads as anecdote. And the comparison methods also had their hyperparameters tuned on the same 1% metric, which is fair but narrow. None of this sinks the central mechanism claim; the argument that low-dimensional constraints are more sample-efficient than high-dimensional entropy estimation is clearly made and the ablation tables back it up.

Who this is for: practitioners with a trained SSL checkpoint and limited compute who want a fast label-scarce accuracy bump, and researchers working on entropy-based SSL objectives. It deserves a serious referee. If I were handling it, I would ask for multiple seeds of continued pretraining and a held-out hyperparameter selection—say, choosing beta and gamma on the 10% split or on a different dataset—before accepting the stronger generalization claims. As it stands, conditional accept.

Referee Report

4 major / 6 minor

Summary. The paper proposes an effective entropy maximization criterion (E2MC) as an add-on loss for continued pre-training of already-trained self-supervised learning (SSL) models. The criterion transforms embeddings into a compact space (sigmoid for VICReg, Gaussian CDF for normalized methods like SwAV and SimSiam), then maximizes the average one-dimensional m-spacings entropy of the marginals while minimizing pairwise covariance, with the base SSL loss retained. The authors evaluate 10 epochs of continued pre-training on ImageNet using VICReg, SwAV, and SimSiam checkpoints, reporting linear-evaluation, semi-supervised, and transfer-learning results, and they include ablations against continued training with the base loss only and against alternative criteria (VCReg, AUH, MMCR). The central claim is that E2MC yields consistent, sometimes significant, downstream improvements after only a handful of epochs.

Significance. If the claimed improvements are robust, the paper offers a computationally cheap and broadly applicable way to squeeze additional performance out of existing SSL checkpoints, which is practically valuable in low-label regimes. The information-theoretic motivation is clear, and the choice of low-dimensional statistics is well grounded given the known difficulty of high-dimensional entropy estimation. The paper includes several good empirical practices: continued-training baselines with the base loss, ablation against alternative information criteria, significance testing in Appendix D.1, and public code. The main significance is contingent on separating the reported gains from hyperparameter selection and seed variability, because the headline 1%-label improvements are obtained on the exact metric used for model selection.

major comments (4)

[Appendix B.1.2, Appendix D.2, Table 1, Figure 4]
[Table 1, Table 2, Appendix D.1]
[Abstract, Section 5.1, Table 1, Table 2]
[Section 3, Figure 2, Figure D.1]

minor comments (6)

[Abstract and Section 5.1]
[Table 1]
[Algorithm 1]
[Appendix B and Equation (4)]
[Section 3.2 and Appendix B.1.2]
[References]

Circularity Check

0 steps flagged · score 1.0 of 10

No definitional circularity: E2MC is a genuine add-on loss whose entropy and covariance terms are not defined in terms of downstream accuracy; the main caveat is that the headline 1%-label result is selected by tuning beta, gamma, and the 10-epoch stopping point on that same metric.

full rationale

The paper's derivation chain is not circular. Equation (6) defines the continued-pretraining loss as the base SSL loss plus a marginal-entropy term (Eq. 7, estimated by the standard m-spacings estimator of Vasicek, 1976) and minus a pairwise-covariance penalty (Eq. 8), and the compact-space transforms (sigmoid and Gaussian CDF) are standard mathematical tools (Muller, 1959; probability integral transform). None of these terms is defined in terms of the downstream accuracies reported in Tables 1-3, so the reported improvements are empirical outcomes rather than identities. The Gaussian-CDF argument is a valid derivation: applying the CDF of N(0,1) maps Gaussian marginals to U[0,1], so maximizing the transformed marginal entropy pushes pre-normalization marginals toward N(0,1); this is a mathematical implication, not an assumption of the target result. Self-citations (VICReg; Shwartz-Ziv et al., 2023) are used as baselines or related work, not as load-bearing justifications for E2MC. The one legitimate concern is evaluation selection rather than circularity: beta and gamma for SwAV were chosen by a grid search on the 1%-label ImageNet linear-evaluation metric (Appendix D.2), and the 10-epoch stopping point was chosen from the trajectory of the same metric (Figure 4). Thus the strongest 1%-label gain (53.70 to 55.27) is partly a selected maximum over configurations, and the significance test in Appendix D.1 picks a single model closest to the reported mean after the fact. These are statistical-selection caveats that lower confidence in the headline number, but they do not make the E2MC derivation equivalent to its inputs; the 10%-label, 100%-label, and transfer results were not used for hyperparameter selection and provide independent, if weaker, evidence.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The method relies on standard information-theoretic facts for its motivation, a standard entropy estimator, and a domain assumption about the reliability of linear evaluation. The only ad hoc premise is the claim that deep-network inductive bias prevents pathological distributions such as the 'X' distribution, which is asserted without direct evidence. Free parameters are the per-method loss coefficients beta and gamma and the training duration, all selected via validation performance.

free parameters (4)

beta (entropy weight) = VICReg: 1000, SwAV: 1, SimSiam: 0.001
Controls the strength of the marginal entropy term in Eq. 6. Chosen per method by hand/grid search on 1%-label ImageNet linear evaluation, not derived from theory.
gamma (covariance weight) = VICReg: 100, SwAV: 25, SimSiam: 0.01
Controls the strength of the pairwise covariance penalty in Eq. 6. Chosen per method alongside beta on the same validation metric.
Number of continued pre-training epochs = 10
Selected as a trade-off; Figure 4 shows gains saturate or reverse after 10 epochs for SwAV. This choice is made from the evaluation curve.
m (spacing order in m-spacings estimator) = round(sqrt(batch_size)) approximately 22 for batch size 512
Standard Vasicek estimator choice; affects the entropy estimate but is not tuned per method.

assumptions (6)

standard math A maximum-entropy distribution on the unit cube is uniform, and equivalently has uniform 1D marginals and mutually independent components.
Invoked in Section 3 to justify using marginal entropies and decorrelation as necessary conditions for max entropy.
standard math Pairwise decorrelation is a necessary condition for mutual independence.
Used to replace the high-dimensional independence requirement with the tractable pairwise covariance penalty.
standard math The m-spacings estimator is a consistent, differentiable estimator of 1D differential entropy.
Foundation of the L_Entropy term; cited to Vasicek (1976) and Learned-Miller and Fisher (2003).
standard math For normalized embeddings, applying the Gaussian CDF maps entropy maximization to marginal Gaussianity, and normalized i.i.d. Gaussians are uniform on the hypersphere (Muller 1959).
Used in Section 3.2 to extend the criterion to hypersphere-normalized SSL methods (SwAV, SimSiam).
ad hoc to paper Deep networks' inductive bias prevents non-smooth 'X'-shaped joint distributions from emerging under the 1D+2D criterion.
Asserted in Section 3 to explain why 2D marginals become near-uniform without explicit enforcement; no proof or measurement is provided.
domain assumption Linear evaluation accuracy on ImageNet validation with 1% labels is a valid proxy for embedding quality and a suitable metric for hyperparameter selection.
Used as the primary evaluation and selection criterion throughout; this is standard in SSL but an assumption about task transfer.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improving Pre-trained Self-Supervised Embeddings Through Effective Entropy Maximization." pith.science (2026). https://pith.science/paper/EATS47GN

@misc{pith2026241115931,
  author       = {Pith},
  title        = {Pith review of: Improving Pre-trained Self-Supervised Embeddings Through Effective Entropy Maximization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EATS47GN}},
  note         = {Machine review of arXiv:2411.15931}
}

read the original abstract

A number of different architectures and loss functions have been applied to the problem of self-supervised learning (SSL), with the goal of developing embeddings that provide the best possible pre-training for as-yet-unknown, lightly supervised downstream tasks. One of these SSL criteria is to maximize the entropy of a set of embeddings in some compact space. But the goal of maximizing the embedding entropy often depends -- whether explicitly or implicitly -- upon high dimensional entropy estimates, which typically perform poorly in more than a few dimensions. In this paper, we motivate an effective entropy maximization criterion (E2MC), defined in terms of easy-to-estimate, low-dimensional constraints. We demonstrate that using it to continue training an already-trained SSL model for only a handful of epochs leads to a consistent and, in some cases, significant improvement in downstream performance. We perform careful ablation studies to show that the improved performance is due to the proposed add-on criterion. We also show that continued pre-training with alternative criteria does not lead to notable improvements, and in some cases, even degrades performance.

Figures

Figures reproduced from arXiv: 2411.15931 by the authors.

**Figure 2.** (a). A 2-d uniform distribution. (b) An “X” distribution. Both (analytic) distributions have uniform (max-entropy) marginals and decorrelated components, and minimize our loss function. (c) Example 2-d marginal distribution over a random pair from VICReg (Bardes et al., 2021) (after transformation to compact space). (d) Our embeddings over the same pair of dimensions, where empirical results show, to our surprise, d… view at source ↗

**Figure 3.** Histograms of distances from a query point to its nearest neighbor (blue) and [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Top-1-Accuracy of a linear classifier trained [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 7 canonical work pages

[1]

VI- CReg: Variance-invariance-covariance regulariza- tion for self-supervised learning

Adrien Bardes, Jean Ponce, and Yann LeCun. VI- CReg: Variance-invariance-covariance regulariza- tion for self-supervised learning. arXiv preprint arXiv:2105.04906,

arXiv
[6]

and stride 1, followed by a batch normalization layer. A linear classifier is then trained on top of these representations using the SGD optimizer with batch size 256, weight decay10−4, momentum 0.9, and learning rate 0.01 reduced by a factor of 10 at epochs 24, 48, and 72, for a total of 84 epochs. These hyperparameters are used consistently across all m...

work page
[8]

The coefficient is determined by grid search overλ = [0.001, 0.005, 0.1, 0.5, 1, 2] verified using the same 1%-ImageNet split as before. B.1.3 SimSiam We start from the 100-epoch checkpoints released for SimSiam (Chen and He, 2021), using resnet50-backbone and 3-layer MLP (2048-2048-2048) as projector, and 2-layer MLP (2048-512) as predictor architecture....

work page 2021
[9]

Self-supervised learn- ing with rotation-invariant kernels.arXiv preprint arXiv:2208.00789,

Léon Zheng, Gilles Puy, Elisa Riccietti, Patrick Pérez, and Rémi Gribonval. Self-supervised learn- ing with rotation-invariant kernels.arXiv preprint arXiv:2208.00789,

arXiv
[13]

For SimSiam, the semi-supervised experiments were not conducted in the original paper, and therefore we skip this in our experiments

For SwAV, use the SGD optimizer with batch size 256, step decay of 0.2 at epochs 12 and 16 for a total of 20 epochs, and no weight decay using learning rate 0.02 for the backbone and 5 for the linear classifier in the 1% labels setting, and learning rate 0.01 for the encoder and 0.2 for the linear classifier in the 10% labels setting. For SimSiam, the sem...

work page 2020
[14]

and Pascal VOC 2007 (Everingham et al., 2010). For iNat18 (437,513 images and 8,142 classes), we useres5 features from the ResNet-50 backbone (before average pooling layer) subsampled to 8192-d using an average pooling layer of size (6,

work page 2007
[25]

See Appendix D.2 for an ablation study on the coefficients. We do continued pre-training for 10 epochs, with a learning rate of0.001 (i.e., 0.01× the base learning rate used to train SwAV), batch size of512, and all other hyperparameters left unchanged from the original method. The same settings are used for continued training ablation without our loss. W...

work page 2023
[1947]

Variance covariance regularization enforces pairwise independence in self-supervised representa- tions

Grégoire Mialon, Randall Balestriero, and Yann Le- Cun. Variance covariance regularization enforces pairwise independence in self-supervised representa- tions. arXiv preprint arXiv:2209.14905,

arXiv

Show all 15 references

[2015]

Spreading vectors for similarity search.arXiv preprint arXiv:1806.03198,

Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Spreading vectors for similarity search.arXiv preprint arXiv:1806.03198,

arXiv
[2016]

Learning deep repre- sentations by mutual information estimation and maximization

R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep repre- sentations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670,

arXiv
[2018]

Towards an improved understanding and utilization of maximum manifold capacity repre- sentations

Rylan Schaeffer, Victor Lecomte, Dhruv Bhandarkar Pai, Andres Carranza, Berivan Isik, Alyssa Unell, Mikail Khona, Thomas Yerxa, Yann LeCun, SueYeon Chung, et al. Towards an improved understanding and utilization of maximum manifold capacity repre- sentations. arXiv preprint ar...

arXiv
[2020]

For continued pretraining, our criterion is applied to the projector embeddingsZ before the cluster assignment layer and before normalization after mapping through the CDF function

with multicrop, using resnet50-backbone and 2-layer MLP (2048-128) as projector architecture. For continued pretraining, our criterion is applied to the projector embeddingsZ before the cluster assignment layer and before normalization after mapping through the CDF function. W...
[2022]

A new class of entropy estimators for multi-dimensional densities

Erik G Miller. A new class of entropy estimators for multi-dimensional densities. In 2003 IEEE Inter- Deep Chakraborty, Yann LeCun, Tim G. J. Rudner, Erik Learned-Miller national Conference on Acoustics, Speech, and Sig- nal Processing,

2003
[2023]

Emp-ssl: Towards self-supervised learning in one training epoch

Shengbang Tong, Yubei Chen, Yi Ma, and Yann Lecun. Emp-ssl: Towards self-supervised learning in one training epoch. arXiv preprint arXiv:2304.03977,

arXiv
[2024]

An information-theoretic perspective on variance- invariance-covariance regularization.arXiv preprint arXiv:2303.00633,

Ravid Shwartz-Ziv, Randall Balestriero, Kenji Kawaguchi, Tim GJ Rudner, and Yann LeCun. An information-theoretic perspective on variance- invariance-covariance regularization.arXiv preprint arXiv:2303.00633,

arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

VI- CReg: Variance-invariance-covariance regulariza- tion for self-supervised learning

Adrien Bardes, Jean Ponce, and Yann LeCun. VI- CReg: Variance-invariance-covariance regulariza- tion for self-supervised learning. arXiv preprint arXiv:2105.04906,

arXiv

[2] [6]

and stride 1, followed by a batch normalization layer. A linear classifier is then trained on top of these representations using the SGD optimizer with batch size 256, weight decay10−4, momentum 0.9, and learning rate 0.01 reduced by a factor of 10 at epochs 24, 48, and 72, for a total of 84 epochs. These hyperparameters are used consistently across all m...

work page

[3] [8]

The coefficient is determined by grid search overλ = [0.001, 0.005, 0.1, 0.5, 1, 2] verified using the same 1%-ImageNet split as before. B.1.3 SimSiam We start from the 100-epoch checkpoints released for SimSiam (Chen and He, 2021), using resnet50-backbone and 3-layer MLP (2048-2048-2048) as projector, and 2-layer MLP (2048-512) as predictor architecture....

work page 2021

[4] [9]

Self-supervised learn- ing with rotation-invariant kernels.arXiv preprint arXiv:2208.00789,

Léon Zheng, Gilles Puy, Elisa Riccietti, Patrick Pérez, and Rémi Gribonval. Self-supervised learn- ing with rotation-invariant kernels.arXiv preprint arXiv:2208.00789,

arXiv

[5] [13]

For SimSiam, the semi-supervised experiments were not conducted in the original paper, and therefore we skip this in our experiments

For SwAV, use the SGD optimizer with batch size 256, step decay of 0.2 at epochs 12 and 16 for a total of 20 epochs, and no weight decay using learning rate 0.02 for the backbone and 5 for the linear classifier in the 1% labels setting, and learning rate 0.01 for the encoder and 0.2 for the linear classifier in the 10% labels setting. For SimSiam, the sem...

work page 2020

[6] [14]

and Pascal VOC 2007 (Everingham et al., 2010). For iNat18 (437,513 images and 8,142 classes), we useres5 features from the ResNet-50 backbone (before average pooling layer) subsampled to 8192-d using an average pooling layer of size (6,

work page 2007

[7] [25]

See Appendix D.2 for an ablation study on the coefficients. We do continued pre-training for 10 epochs, with a learning rate of0.001 (i.e., 0.01× the base learning rate used to train SwAV), batch size of512, and all other hyperparameters left unchanged from the original method. The same settings are used for continued training ablation without our loss. W...

work page 2023

[8] [1947]

Variance covariance regularization enforces pairwise independence in self-supervised representa- tions

Grégoire Mialon, Randall Balestriero, and Yann Le- Cun. Variance covariance regularization enforces pairwise independence in self-supervised representa- tions. arXiv preprint arXiv:2209.14905,

arXiv