Pith. sign in

REVIEW 2 major objections 4 minor 78 references

A Theory of Contrastive Learning with Natural Images

T0 review · 2 major / 4 minor · reviewed 2026-07-09 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Contrastive learning on images reduces to partial whitening of Fourier power

desk verdict The Gaussianity bridge between GUPA and InfoNCE is the one real soft spot in an otherwise clean and genuinely novel theory paper. read the letter →

arxiv 2607.07470 v1 pith:2FIHLISM submitted 2026-07-08 cs.CV

Antonio Torralba , Yair Weiss This is my paper
classification cs.CV
keywords contrastivelearningpartialwhiteningwaterfillingalgorithmstationaryimagestatisticsDFTcoefficientssinusoidalfiltersInfoNCElossnaturalpowerspectrum
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks why contrastive learning produces useful image representations from simple augmentations and simple images. The authors analytically compute the globally optimal representation under a contrastive loss (called GUPA, which equals the standard InfoNCE loss when representations are Gaussian) for any image dataset with stationary statistics. They prove that for a range of augmentations — circular crop, linear jitter, ideal blur, and crop-plus-noise with arbitrary blur — the optimal representation is partial whitening of the discrete Fourier transform's power coefficients: select a subset of spatial frequencies, measure their power, and rescale each so that the representation has equal variance in all directions. The specific frequencies and their weights are determined by a waterfilling algorithm on the dataset's expected power spectrum. This optimal representation can be computed by a single-layer CNN whose first-layer filters are sinusoids, followed by a pointwise nonlinearity (squaring or ReLU), global average pooling, and a linear projection. The key mechanism is that stationarity of the image statistics forces DFT coefficients to be asymptotically Gaussian and pairwise independent, which diagonalizes both the covariance and augmentation-distortion matrices in Fourier space, making the optimization separable across frequencies. Experiments confirm that CNNs trained by SGD on real and synthetic datasets learn sinusoidal first-layer filters and perform partial whitening, matching the theoretical predictions.

What carries the argument

stationarity → Gaussian independent DFT coefficients → diagonal loss in Fourier space → waterfilling selects frequencies → partial whitening rescales them

What would settle it

Train a contrastive model on a dataset with strongly non-Gaussian DFT coefficients (violating stationarity) and check whether the learned filters deviate from sinusoids and the partial-whitening sensitivity pattern — if they do, the stationarity assumption is load-bearing rather than incidental.

Watch

Extended reading notes

Core claim

The central object is the partial whitening representation: for stationary image datasets, the contrastive loss is globally minimized by measuring power at a subset of spatial frequencies (selected by a waterfilling algorithm on the expected power spectrum) and rescaling each frequency's power inversely to its expected variance. The stationarity of natural images — meaning their statistics are translation-invariant — is the load-bearing property that makes this work, because it forces DFT coefficients to be Gaussian and pairwise independent, which in turn makes the contrastive loss separable across individual frequencies. A single-layer CNN with sinusoidal filters, a pointwise nonlinearity,,

Load-bearing premise

The theoretical results assume that the learned representation is Gaussian-distributed over the training set, which is what makes the analytically tractable GUPA loss equal to the standard InfoNCE contrastive loss. If the representation is not Gaussian, the optimality guarantees may not apply to standard contrastive learning as practiced.

Editorial extensions

If this is right

  • If the theory extends to deeper architectures, it would explain why early layers of contrastive learning models converge to Gabor-like or sinusoidal filters: they are computing the analytically optimal partial whitening representation dictated by the dataset's power spectrum.
  • The waterfilling algorithm provides a parameter-free prediction of which spatial frequencies a contrastive model should be sensitive to, given only the dataset's expected power spectrum — this is directly testable on any dataset and augmentation set.
  • The result that synthetic noise images transfer to real images only when their power spectra match explains the empirical finding that 1/f noise works better than white noise for pretraining: the waterfilling solution selects the same frequencies only when the spectra align.
  • The theory predicts that augmentations do not teach invariance per se but instead bias the waterfilling solution toward particular frequency bands, which means augmentation design can be reframed as frequency-band selection rather than invariance engineering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The gap between the simple-augmentation regime (where partial whitening is optimal) and SOTA augmentations (where localized filters emerge) suggests that standard augmentations like randomized crop size and nonlinear color jitter break the frequency-separability that makes the analysis tractable — extending the theory would require handling non-diagonal augmentation-distortion matrices in Fourier
  • If the Gaussianity assumption on representations fails for certain architectures or datasets, the GUPA loss diverges from InfoNCE and the optimality guarantees may not transfer; however, the empirical observation that learned representations are approximately Gaussian suggests this may be a mild restriction in practice.
  • The non-uniqueness of the optimal solution (any orthogonal rotation of the whitened representation is also optimal) means that gradient descent dynamics, not the loss landscape alone, determine which specific frequencies are learned — connecting implicit bias of optimization to the observed filter structure.
  • The finding that most recognition accuracy gains come from partial whitening rather than learned invariance suggests that simple spectral preprocessing might recover much of the benefit of contrastive pretraining for downstream tasks, at least for simple architectures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper analytically computes the optimal representation for contrastive learning (CL) under a proposed loss, LGUPA, for datasets with stationary statistics and a range of standard augmentations (circular crop, linear jitter, ideal blur, crop-plus-noise). The authors show that the globally optimal representation under LGUPA performs partial whitening of DFT power coefficients, computable by a single-layer CNN with sinusoidal filters, a pointwise nonlinearity, global average pooling, and a linear projection. The optimal frequencies and weights are determined by a waterfilling algorithm on the dataset's expected power spectrum. Experiments on synthetic noise, dead leaves, and real image datasets (CIFAR10/100, ImageNet) show that CNNs trained with SGD learn sinusoidal filters and perform partial whitening, consistent with the theoretical predictions.

Significance. The paper provides a valuable and analytically tractable theory connecting contrastive learning to classical natural image statistics. The derivation of the waterfilling optimality (Theorems 1.3, 3.1, 3.2) from first principles—stationarity, the CLT for DFT coefficients (Theorem 2.1), and the diagonal structure of the alignment and covariance matrices—is mathematically sound and yields falsifiable, parameter-light predictions (given the power spectrum). The empirical confirmation that SGD-trained CNNs converge to the predicted sinusoidal filters and partial whitening sensitivities across diverse datasets and augmentations is a notable strength. The bridge from LGUPA to InfoNCE via the Gaussianity assumption is the primary point of fragility, but the authors are transparent about this limitation.

major comments (2)
  1. Theorem 1.1 equates the InfoNCE loss to the LGUPA loss only when the representation y(x) is Gaussian over the training set. This is the linchpin connecting the theoretical results (which are about LGUPA) to standard contrastive learning practice (which uses InfoNCE). The paper supports this bridge with a citation to Betser et al. (2026), described as having 'reported' approximate Gaussianity (empirical, not a theorem), and with Figures 11-13 showing similar accuracy and filters under both losses. However, no theorem guarantees that InfoNCE training induces Gaussian representations, and the paper uses hedging language ('approximately Gaussian'). If y(x) is significantly non-Gaussian, the GUPA optimum (partial whitening of DFT power) need not be the InfoNCE optimum. The authors should strengthen this connection: for example, by providing a direct quantitative comparison of the two losses (
  2. Theorem 3.2's proof (Appendix F, Eqs. 36-37) models crop translations as cyclic translations plus noise, where the noise term η(t) = x(t−Δ)(W(t)−W(t−Δ)) is treated as independent of x. As the paper acknowledges, η(t) is explicitly signal-dependent. The diagonal structure of the B matrix (Appendix G) is derived under the assumption that η is independent white Gaussian noise. While Figure 14 provides empirical evidence that the two augmentation types behave similarly in the DFT domain, the signal-dependence of η means the B matrix for real crop translations may not be exactly diagonal, potentially affecting the optimality of sinusoidal filters. The authors should discuss the sensitivity of the theoretical predictions to this signal-dependence, or clarify the scope of Theorem 3.2 as applying strictly to the cyclic-crop-plus-independent-noise model.
minor comments (4)
  1. In Algorithm 1, the loop runs 'for i = 1 to 1/η'. It would be clearer to specify the stopping criterion (e.g., until total power budget is exhausted or convergence).
  2. Figure 3 references '32 randomly chosen squared DFT coefficients' but the covariance matrix shown appears to be larger. Clarify the dimensionality.
  3. The paper states (Section 4) that σ and temperature t were fixed (σ=0.0014, t=1) for all experiments. A brief discussion of how robust the predicted ring/diamond sensitivity patterns are to these hyperparameters would be helpful, since the waterfilling solution depends on them.
  4. The reference to 'Betser et al., 2026' appears to be a future-dated citation. If this is a forthcoming publication, please verify the citation details.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for a careful and constructive report. The two major comments both concern the scope of our theoretical guarantees relative to standard contrastive learning practice. We address each below and describe revisions we will make.

read point-by-point responses

  1. Referee: Theorem 1.1 equates InfoNCE to LGUPA only when y(x) is Gaussian. This is the linchpin connecting theory to practice. No theorem guarantees InfoNCE induces Gaussian representations. If y is significantly non-Gaussian, the GUPA optimum need not be the InfoNCE optimum. Strengthen this connection.

    Authors: We agree that the Gaussianity bridge is the primary point of fragility in our theory, and we appreciate the referee's precise articulation of the issue. We want to be clear about what we can and cannot claim. What we can claim: (1) Theorem 1.1 is a conditional result—if y is Gaussian, then InfoNCE and LGUPA are equivalent up to a constant. This is a theorem, not an approximation. (2) Empirically, representations trained with InfoNCE are approximately Gaussian (Betser et al., 2026), and training with LGUPA versus InfoNCE yields nearly identical learned filters (Figure 12) and nearly identical recognition accuracy across multiple augmentations and architectures (Figures 11, 13). (3) The same partial-whitening structure arises under other contrastive losses that do not require Gaussianity, such as VICReg (Bardes et al., 2022) and Barlow Twins (Zbontar et al., 2021), which are based on covariance rather than full distributional assumptions. This suggests the predictions are not an artifact of the Gaussian assumption specifically. What we cannot claim: we have no theorem guaranteeing that InfoNCE training induces Gaussian representations, nor can we rule out that for some datasets or architectures, significant non-Gaussianity could cause the LGUPA optimum to diverge from the InfoNCE optimum. We will revise the manuscript to make the scope of Theorem 1.1 more explicit and to add a quantitative comparison of the two losses. Specifically, we will add a table reporting the KL divergence (or other distributional distance) between the learned representation distribution and a fitted Gaussian for each dataset and augmentation, so readers can assess the degree of Gaussianity directly. We will also add a brief discussion of why the predictions may be robust to moderate non-Gaussianit revision: partial

  2. Referee: Theorem 3.2's proof models crop translations as cyclic translations plus noise, where η(t) = x(t−Δ)(W(t)−W(t−Δ)) is treated as independent of x. The noise is explicitly signal-dependent. The diagonal structure of B is derived under the assumption that η is independent white Gaussian noise. The B matrix for real crop translations may not be exactly diagonal. Discuss sensitivity or clarify scope.

    Authors: The referee is correct that η(t) is signal-dependent and that this means the B matrix for actual crop translations is not guaranteed to be exactly diagonal. Our current treatment in Appendix F acknowledges this signal-dependence but does not rigorously bound its effect on the diagonal structure of B. We will make two changes. First, we will clarify the scope of Theorem 3.2 (and its appendix version, Theorem G.1) as applying strictly to the cyclic-crop-plus-independent-Gaussian-noise model. This is already implicit in the proof but should be stated explicitly in the theorem statement. Second, we will add a discussion of the sensitivity to the signal-dependence. The key observation is that the diagonal structure of B requires not that η be independent of x, but rather that the cross-frequency terms E[δ[k]δ[l]] vanish for k≠l. For the signal-dependent noise model, these cross terms involve E[|xF[k]|² · (terms involving ηF)] which, under the Gaussianity of DFT coefficients (Theorem 2.1), factor into products of expectations. We will verify empirically that the off-diagonal elements of B for actual crop translations are small relative to the diagonal, complementing the visual evidence in Figure 14. We agree that a full theoretical treatment of the signal-dependent case is beyond the scope of this paper, and we will state this as an open problem. We believe the empirical evidence (Figure 14 showing similar DFT-domain behavior, and the experimental results in Figures 7-8 showing sinusoidal filters for crop translations across all datasets) supports the practical relevance of the theory, but we will be more precise about what is proven versus what is empirically supported. revision: partial

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity found; derivation chain is self-contained with external inputs and first-principles proofs.

full rationale

The paper's derivation chain proceeds through independently justified steps. Theorem 2.1 (DFT coefficients of stationary signals are Gaussian and pairwise independent) cites Peligrad & Wu (2010), an external probability result. Theorem 1.1 (InfoNCE = GUPA under Gaussianity) is derived from first principles in Appendix B via the cross-entropy formula for Gaussians; the Gaussianity assumption is supported by an external citation (Betser et al. 2026, no author overlap) and empirical comparisons (Figures 11–13), but is openly stated as an approximation, not hidden or forced. Theorems 3.1 and 3.2 construct representations that achieve zero alignment loss (by invariance to the specified augmentations) and white covariance (by inverse-variance scaling using the dataset power spectrum g(k), which is an external input computed from data, not a fitted parameter). The waterfilling algorithm (Algorithm 1) takes g(k), σ, and temperature t as inputs—all fixed or externally computed—and produces frequency selection and weights as outputs. The noise variance σ=0.0014 and temperature t=1 are held constant across all experiments rather than fit per dataset. The main theoretical results do not depend on self-citations: the one self-citation (Shifman & Weiss 2024) concerns invariant representations to crops and is not load-bearing for the central theorems. The predictions (sinusoidal filters, partial whitening, ring/diamond sensitivity patterns) are verified against independently trained CNNs, not against the same construction. The Gaussianity bridge between GUPA and InfoNCE is a correctness risk (an unproven distributional assumption), not a circularity: the paper does not define y(x) to be Gaussian and then claim it as a prediction. Overall, the derivation is self-contained against external benchmarks and the central claims have independent content.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities, particles, forces, or postulated objects. The GUPA loss is a mathematical reformulation of InfoNCE under a Gaussianity assumption, not a new entity. The waterfilling algorithm is a standard technique from communications theory. All axioms are either standard mathematical results or domain assumptions about image statistics that are empirically validated.

free parameters (3)
  • noise variance σ = 0.0014
    Used in the crop-plus-noise augmentation model. Fixed across all experiments rather than fit per dataset. The paper states 'Different values give qualitatively similar results but the distance of the rings from the origin changes.'
  • inverse temperature t = 1
    Appears in the GUPA loss as ϵ=1/(2t). Fixed at t=1 for all experiments.
  • partial whitening regularization λ = 0.1
    Used in the PCA-based partial whitening baseline (Section I, following Thiry 2021). Not a parameter of the theory itself but of the baseline comparison.
assumptions (4)
  • domain assumption The representation y(x) is Gaussian-distributed over the training set
    Invoked in Theorem 1.1 to equate InfoNCE with GUPA. The paper cites Betser et al. 2026 for empirical support and provides Figures 11-13 showing similar performance, but this is an approximation, not a proven identity.
  • domain assumption Image datasets have stationary (translation-invariant) statistics
    Invoked in Theorem 2.1 and all subsequent theorems. The paper notes this holds approximately for real datasets and discusses non-stationary datasets (CelebA) in Section K, finding that GAP still yields sinusoidal filters.
  • standard math The DFT coefficients of stationary signals satisfy the regularity conditions of Peligrad & Wu (2010)
    Theorem 2.1 cites this result for the asymptotic Gaussianity and pairwise independence of DFT coefficients. The regularity conditions are not verified explicitly for image datasets but are standard in time-series analysis.
  • ad hoc to paper Crop translations can be approximated by cyclic translations plus independent noise
    Used in Section F and the proof of Theorem 3.2 to make crop augmentation analytically tractable. Justified empirically by Figure 14 showing similar DFT behavior, but the noise η(t) is not actually independent of x(t) as noted in equation 37.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Theory of Contrastive Learning with Natural Images." pith.science (2026). https://pith.science/paper/2FIHLISM

@misc{pith2026260707470,
  author       = {Pith},
  title        = {Pith review of: A Theory of Contrastive Learning with Natural Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2FIHLISM}},
  note         = {Machine review of arXiv:2607.07470}
}
read the original abstract

Why does contrastive learning with simple images and augmentations yield useful representations for downstream tasks? We address this question by analytically computing the optimal representation in terms of a contrastive loss for a range of basic augmentations and any image dataset with stationary statistics. We show that for certain augmentations the optimum can be attained by a CNN whose first layer filters are sinusoids, followed by a pointwise nonlinearity, global average pooling, and a final linear layer that performs partial whitening. We also show that the optimal weights in such CNNs for more complicated augmentations are still sinusoids. The frequencies of the sinusoids and their weights can be computed using a simple waterfilling algorithm given the dataset's expected power spectrum. Experiments with different image datasets and augmentations show that such CNNs trained with SGD empirically learn sinusoids in their first layer and to perform partial whitening

Figures

Figures reproduced from arXiv: 2607.07470 by the authors.

Figure 1
Figure 1. Mysteries of contrastive learning. Columns show sim￾ple augmentations that are used in CL. A combination of these low-level augmentations yields state-of-the-art recognition perfor￾mance. Rows show different image datasets: real images (top), fractal noise (middle) and dead leaves (bottom). Even simple aug￾mentations with nonrealistic images yield useful features for real images (Baradad et al., 2021). works so well… view at source ↗
Figure 2
Figure 2. Filters that are learned in the first layer of a simple CNN trained with contrastive loss on different datasets (the highest variance filters are displayed). In this paper we prove that whenever the dataset has stationary statistics and for a wide range of augmentations, the optimal filters are sinusoids. an image to gray level and applying local contrast normal￾ization will make it invariant to color jitter and tec… view at source ↗
Figure 3
Figure 3. For any stationary signal, the DFT coefficients are Gaussian and pairwise independent as the size of the image goes to infinity and the variance of the power at a particular frequency can be computed from its expectation. Even though CIFAR10 images (top) are very different from fractal images (bottom), both datasets lead to approximately Gaussian DFT coefficients. Original Circular Crop Linear Jitter Ideal Blur [PI… view at source ↗
Figures from the paper (22 more)
Figure 4
Figure 4. Figure 4: A set of augmentations for which measuring squared DFT coefficients yields the global optimum of the LGUPA contrastive loss for any stationary signal. x input image Conv. filters hc ρ ReLU / square GAP per channel ϕ(x) Linear WT ϕ(x) y(x) Normalize to unit sphere y(x) …
Figure 6
Figure 6. Figure 6: top: Frequency sensitivity of the CNN with optimal weights (theorem 3.2) for different datasets. bottom: Frequency sensitivity multiplied by expected power g[k]. The CNNs learn to perform partial whitening. equation 43 in the appendix). We then apply theorem 1.3 and de…
Figure 7
Figure 7. Figure 7: Results with ReLU nonlinearity. The results are similar to those obtained with a quadratic nonlinearity. Row 1: learned kernels. Row 2: sensitivity. Row 3: average power spectrum of the training images. Row 4: product of the sensitivity and the square root of the avera…
Figure 8
Figure 8. Figure 8: Summary of our experimental results with different augmentations and quadratic and ReLU nonlinearities. Circular trans. Hor. flip Blur Contrast reversal No aug. Mixed SOTA ReLU Square Crop trans [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Kernels learned when training with color images on ImageNet. As predicted by the theory, for all the simple augmentations the kernels are sinusoids that are a function of only one of the uncorrelated color channels. For natural images, these uncorrelated channels are a…
Figure 10
Figure 10. Figure 10: Summary of recognition performance with ρ(z) = ReLU. Testing is always performed on CIFAR10. (left) training is performed on different datasets using crop translation as augmentation. (right) training is done on CIFAR10 using different augmentations. The yellow horizo…
Figure 11
Figure 11. Figure 11: In this paper we analyze the LGUPA contrastive loss which is equivalent to the InfoNCE loss when the distribution is Gaussian. The four subfigures show the recognition accuracy when training on color CIFAR10 images with the two different losses as a function of epoch …
Figure 12
Figure 12. Figure 12: Kernels learned when training with gray-scale images on CIFAR10 when minimizing the InfoNCE loss. The kernels are sinusoids. Compare the kernels with those of figure 8. Appendix A. Comparing the InfoNCE and LGUPA contrastive losses [PITH_FULL_IMAGE:figures/full_fig_p…
Figure 13
Figure 13. Figure 13: Recognition accuracy when training with gray-scale CIFAR10 with two different losses and different augmentations. The representation layer has one convolutional layer, a ReLU non-linearity and global average pooling. The projection layer is a fully-connected layer fol…
Figure 14
Figure 14. Figure 14: Typically crop translations are used in CL. In the proof we assume cyclic translations plus noise. This figure shows that they behave similarly in terms of their influence on the DFT. In both cases, the squared DFT features are similar and are more dissimilar for high…
Figure 15
Figure 15. Figure 15: The frequency sensitivity of linear partial whitening for different image datasets. Similar to the predicted sensitivities of the optimal CNN (figure 6) and the sensitivities of CNNs trained with SGD (figures 16,7), the representation is mostly sensitive to frequencie…
Figure 16
Figure 16. Figure 16: Summary of our experimental results with quadratic nonlinearity. For all the datasets, the filters in the first layer converge to sinusoids (figure 2). All the datasets learn to compare frequencies with similar expected power (top) and the sensitivity is qualitatively…
Figure 17
Figure 17. Figure 17: Summary of our experimental results with different augmentations and quadratic and ReLU nonlinearities. This figure extends the content of fig. 8 by adding the sensitivities in addition to the filters. Results are after 2000 training epochs. 22 [PITH_FULL_IMAGE:figur…
Figure 18
Figure 18. Figure 18: CelebA leads to approximately Gaussian DFT coefficients. Square Circular conv. Cyclic trans. ReLU Valid conv. Crop trans [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]
Figure 19
Figure 19. Figure 19: Results when training with CelebA scaled to 128×128. Left, results with square nonlinearity, when the augmentations are cyclic translations and the network uses circular convolutions. Right, results with ReLU, valid convolutions and augmentations are crop translations…
Figure 20
Figure 20. Figure 20: Kernels learned when training with ImageNet scaled to 128×128. Images are augmented with cyclic translations. The network uses squaring non-linearities, circular convolutions, 256 channels, and no bias terms. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_20.png]
Figure 21
Figure 21. Figure 21: Kernels learned when training with ImageNet scaled to 128×128. Images are augmented with crop translations. The network uses ReLU non-linearities, valid convolutions, 256 channels and bias terms in both layers. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_21.png]
Figure 22
Figure 22. Figure 22: Kernels learned when training with CIFAR10. Images are augmented with horizontal flip. The network uses squaring non-linearities, circular convolutions, 256 channels, and no bias terms. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_22.png]
Figure 23
Figure 23. Figure 23: Kernels learned when training with CIFAR10. Images are augmented with horizontal flip. The network uses ReLU non-linearities, valid convolutions, 256 channels and bias terms in both layers. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_23.png]
Figure 24
Figure 24. Figure 24: Evolution of the learned kernels over training epochs when training with CIFAR10. Images are augmented with crop translation. The network uses square non-linearities, valid convolutions, 256 channels and bias terms in both layers. The experimental settings are the sam…
Figure 25
Figure 25. Figure 25: Evolution of the GUPA loss and performance over training epochs when training with CIFAR10. Images are augmented with crop translation (same as fig. 24). The network uses square non-linearities, valid convolutions, 256 channels and bias terms in both layers. 29 [PITH…
Figure 26
Figure 26. Figure 26: Kernels learned when training with color images on CIFAR10. As predicted by the theory, for all the simple augmentations the kernels are sinusoids in the uncorrelated channels which are approximately: (gray, red-green, blue-yellow). N. Extensions to Color Images The t…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

78 extracted references · 78 canonical work pages

  1. [1]

    Langley , title =

    P. Langley , title =. Proceedings of the 17th International Conference on Machine Learning (ICML 2000) , address =. 2000 , pages =

    work page 2000

  2. [2]

    T. M. Mitchell. The Need for Biases in Learning Generalizations. 1980

    work page 1980

  3. [3]

    M. J. Kearns , title =

    work page

  4. [4]

    Machine Learning: An Artificial Intelligence Approach, Vol. I. 1983

    work page 1983

  5. [5]

    R. O. Duda and P. E. Hart and D. G. Stork. Pattern Classification. 2000

    work page 2000

  6. [6]

    Suppressed for Anonymity , author=

    work page

  7. [7]

    Newell and P

    A. Newell and P. S. Rosenbloom. Mechanisms of Skill Acquisition and the Law of Practice. Cognitive Skills and Their Acquisition. 1981

    work page 1981

  8. [8]

    A. L. Samuel. Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development. 1959

    work page 1959

Show all 78 references
  1. [9]

    and Redlich, A

    Atick, J. and Redlich, A. What does the retina know about natural scenes? Neural Computation, 4: 0 196--210, 03 1992. doi:10.1162/neco.1992.4.2.196

    1992 doi

  2. [10]

    and LeCun, Y

    Balestriero, R. and LeCun, Y. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran A...

    2022

  3. [11]

    and LeCun, Y

    Balestriero, R. and LeCun, Y. Lejepa: Provable and scalable self-supervised learning without the heuristics, 2025. URL https://arxiv.org/abs/2511.08544

    2025 arXiv

  4. [12]

    Understanding contrastive learning via gaussian mixture models

    Bansal, P., Kavis, A., and Sanghavi, S. Understanding contrastive learning via gaussian mixture models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=nCAdkkAeR9

    2025

  5. [13]

    M., Wulff, J., Wang, T., Isola, P., and Torralba, A

    Baradad, J. M., Wulff, J., Wang, T., Isola, P., and Torralba, A. Learning to see by looking at noise. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 2556--2569. Curran Assoc...

    2021

  6. [14]

    VICR eg: Variance-invariance-covariance regularization for self-supervised learning

    Bardes, A., Ponce, J., and LeCun, Y. VICR eg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=xm6YD62D1Ub

    2022

  7. [15]

    and Niyogi, P

    Belkin, M. and Niyogi, P. Laplacian eigenmaps and spectral techniques for embedding and clustering. In Proceedings of the 15th International Conference on Neural Information Processing Systems: Natural and Synthetic, NIPS'01, pp.\ 585–591, Cambridge, MA, USA, 2001. MIT Press

    2001

  8. [16]

    Y., and Gilboa, G

    Betser, R., Gofer, E., Levi, M. Y., and Gilboa, G. Info NCE induces gaussian distribution. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=BlSH7gNQSq

    2026

  9. [18]

    A simple framework for contrastive learning of visual representations

    Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020

    2020

  10. [19]

    An analysis of single-layer networks in unsupervised feature learning

    Coates, A., Ng, A., and Lee, H. An analysis of single-layer networks in unsupervised feature learning. In Gordon, G., Dunson, D., and Dudík, M. (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of...

    2011

  11. [20]

    Whitening for self-supervised representation learning

    Ermolov, A., Siarohin, A., Sangineto, E., and Sebe, N. Whitening for self-supervised representation learning. In International conference on machine learning, pp.\ 3015--3024. PMLR, 2021

    2021

  12. [22]

    HaoChen, J. Z. and Ma, T. A theoretical study of inductive biases in contrastive learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=AuEgNlEAmed

    2023

  13. [23]

    Z., Wei, C., Gaidon, A., and Ma, T

    HaoChen, J. Z., Wei, C., Gaidon, A., and Ma, T. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https://ope...

    2021

  14. [24]

    Preventing dimensional collapse in self-supervised learning via orthogonality regularization

    He, J., Du, J., and Ma, W. Preventing dimensional collapse in self-supervised learning via orthogonality regularization. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN...

    2024

  15. [25]

    Natural image statistics: a probabilistic approach to early computational vision

    Hyv \"a rinen, A., Hurri, J., and Hoyer, P. Natural image statistics: a probabilistic approach to early computational vision. Springer, United States, 2009. ISBN 978-1-84882-490-4

    2009

  16. [26]

    A style-based generator architecture for generative adversarial networks

    Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 4401--4410, 2019

    2019

  17. [28]

    Krogstad, H. E. On the covariance of the periodogram. Journal of Time Series Analysis, 3 0 (3): 0 195--207, 1982. URL https://EconPapers.repec.org/RePEc:bla:jtsera:v:3:y:1982:i:3:p:195-207

    1982

  18. [30]

    Y., Jordan, M

    Ng, A. Y., Jordan, M. I., and Weiss, Y. On spectral clustering: analysis and an algorithm. In Proceedings of the 15th International Conference on Neural Information Processing Systems: Natural and Synthetic, NIPS'01, pp.\ 849–856, Cambridge, MA, USA, 2001. MIT Press

    2001

  19. [31]

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Maira...

    2024

  20. [32]

    and Wu, W

    Peligrad, M. and Wu, W. B. Central limit theorem for fourier transforms of stationary processes. The Annals of Probability, 38 0 (5): 0 2009--2022, 2010. ISSN 00911798. URL http://www.jstor.org/stable/25734716

    2009

  21. [34]

    Understanding contrastive learning requires incorporating inductive biases

    Saunshi, N., Ash, J., Goel, S., Misra, D., Zhang, C., Arora, S., Kakade, S., and Krishnamurthy, A. Understanding contrastive learning requires incorporating inductive biases. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings o...

    2022

  22. [36]

    K Nearest Neighbors on CIFAR10

    Thiry, L. K Nearest Neighbors on CIFAR10 . https://gist.github.com/louity/c6b0c91810c9957f57c56c952323b29e, 2021. [Online; accessed 28-May-2026]

    2021

  23. [37]

    Representation learning with contrastive predictive coding, 2019

    van den Oord, A., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding, 2019. URL https://arxiv.org/abs/1807.03748

    2019 arXiv

  24. [38]

    and Isola, P

    Wang, T. and Isola, P. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning...

    2020

  25. [39]

    D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., and Srebro, N

    Woodworth, B., Gunasekar, S., Lee, J. D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., and Srebro, N. Kernel and rich regimes in overparametrized models. In Abernethy, J. and Agarwal, S. (eds.), Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Pro...

    2020

  26. [40]

    Xing, C., Jing, Y., Wang, S., Ma, S., and Poor, H. V. New viewpoint and algorithms for water-filling solutions in wireless communications. IEEE Transactions on Signal Processing, 68: 0 1618--1634, 2020. doi:10.1109/TSP.2020.2973488

    2020 doi

  27. [41]

    Barlow twins: Self-supervised learning via redundancy reduction

    Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volum...

    2021

  28. [42]

    arXiv preprint arXiv:2004.11362 , year=

    Supervised Contrastive Learning , author=. arXiv preprint arXiv:2004.11362 , year=

    2004 arXiv

  29. [43]

    Roy Betser and Eyal Gofer and Meir Yossef Levi and Guy Gilboa , booktitle=. Info. 2026 , url=

    2026

  30. [44]

    Field , journal =

    David J. Field , journal =. Relations between the statistics of natural images and the response properties of cortical cells , volume =. 1987 , url =. doi:10.1364/JOSAA.4.002379 , abstract =

    1987 doi

  31. [45]

    G. J. Burton and Ian R. Moorhead , journal =. Color and spatial structure in natural scenes , volume =. 1987 , url =. doi:10.1364/AO.26.000157 , abstract =

    1987 doi

  32. [46]

    Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages =

    An Analysis of Single-Layer Networks in Unsupervised Feature Learning , author =. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages =. 2011 , editor =

    2011

  33. [47]

    FirstName LastName , title =

  34. [48]

    FirstName Alpher , title =

  35. [49]

    Journal of Foo , volume = 13, number = 1, pages =

    FirstName Alpher and FirstName Fotheringham-Smythe , title =. Journal of Foo , volume = 13, number = 1, pages =

  36. [50]

    Journal of Foo , volume = 14, number = 1, pages =

    FirstName Alpher and FirstName Fotheringham-Smythe and FirstName Gamow , title =. Journal of Foo , volume = 14, number = 1, pages =

  37. [51]

    FirstName Alpher and FirstName Gamow , title =

  38. [52]

    Proceedings of the 37th International Conference on Machine Learning , articleno =

    Chen, Ting and Kornblith, Simon and Norouzi, Mohammad and Hinton, Geoffrey , title =. Proceedings of the 37th International Conference on Machine Learning , articleno =. 2020 , publisher =

    2020

  39. [53]

    2025 , eprint=

    LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics , author=. 2025 , eprint=

    2025

  40. [54]

    , title =

    Oliva, Aude and Torralba, Antonio B. , title =. Proceedings of the Second International Workshop on Biologically Motivated Computer Vision , pages =. 2002 , isbn =

    2002

  41. [55]

    CENTRAL LIMIT THEOREM FOR FOURIER TRANSFORMS OF STATIONARY PROCESSES , urldate =

    Magda Peligrad and Wei Biao Wu , journal =. CENTRAL LIMIT THEOREM FOR FOURIER TRANSFORMS OF STATIONARY PROCESSES , urldate =

  42. [56]

    Vincent , journal=

    Xing, Chengwen and Jing, Yindi and Wang, Shuai and Ma, Shaodan and Poor, H. Vincent , journal=. New Viewpoint and Algorithms for Water-Filling Solutions in Wireless Communications , year=

  43. [57]

    Learning to See by Looking at Noise , url =

    Baradad, Jurjo Manel and Wulff, Jonas and Wang, Tongzhou and Isola, Phillip and Torralba, Antonio , booktitle =. Learning to See by Looking at Noise , url =

  44. [58]

    Barlow Twins: Self-Supervised Learning via Redundancy Reduction , booktitle =

    Jure Zbontar and Li Jing and Ishan Misra and Yann LeCun and St. Barlow Twins: Self-Supervised Learning via Redundancy Reduction , booktitle =. 2021 , url =

    2021

  45. [59]

    2022 , url=

    Adrien Bardes and Jean Ponce and Yann LeCun , booktitle=. 2022 , url=

    2022

  46. [60]

    Ruderman and Thomas W

    Daniel L. Ruderman and Thomas W. Cronin and Chuan-Chin Chiao , journal =. Statistics of cone responses to natural images: implications for visual coding , volume =. 1998 , url =. doi:10.1364/JOSAA.15.002036 , abstract =

    1998 doi

  47. [61]

    Proceedings of Thirty Third Conference on Learning Theory , pages =

    Kernel and Rich Regimes in Overparametrized Models , author =. Proceedings of Thirty Third Conference on Learning Theory , pages =. 2020 , editor =

    2020

  48. [62]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    A style-based generator architecture for generative adversarial networks , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  49. [63]

    Proceedings of the 38th International Conference on Neural Information Processing Systems , articleno =

    He, Junlin and Du, Jinxiao and Ma, Wei , title =. Proceedings of the 38th International Conference on Neural Information Processing Systems , articleno =. 2024 , isbn =

    2024

  50. [64]

    Louis Thiry , title =

  51. [65]

    , title =

    Krogstad, Harald E. , title =. Journal of Time Series Analysis , volume =. doi:https://doi.org/10.1111/j.1467-9892.1982.tb00342.x , url =. https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-9892.1982.tb00342.x , abstract =

    1982 doi

  52. [66]

    , year =

    Atick, Joseph and Redlich, A. , year =. What Does the Retina Know about Natural Scenes? , volume =. Neural Computation , doi =

  53. [67]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere , author =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , editor =

    2020

  54. [68]

    1982 , journal =

    ON THE COVARIANCE OF THE PERIODOGRAM , author =. 1982 , journal =

    1982

  55. [69]

    , title =

    Hyvarinen, Aapo and Hurri, Jarmo and Hoyer, Patrick O. , title =. 2009 , isbn =

    2009

  56. [70]

    Alias-Free Convnets: Fractional Shift Invariance via Polynomial Activations

    Hagay Michaeli and Tomer Michaeli and Daniel Soudry. Alias-Free Convnets: Fractional Shift Invariance via Polynomial Activations. Proceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023. 2023. doi:10.1109/CVPR52729.2023.01567

    2023 doi

  57. [71]

    Lost in Translation: Modern Neural Networks Still Struggle with Small Realistic Image Transformations , booktitle =

    Ofir Shifman and Yair Weiss , editor =. Lost in Translation: Modern Neural Networks Still Struggle with Small Realistic Image Transformations , booktitle =. 2024 , url =. doi:10.1007/978-3-031-72890-7\_14 , timestamp =

    2024 doi

  58. [72]

    IEEE Transactions on audio and electroacoustics , keywords =

    Welch, Peter , biburl =. IEEE Transactions on audio and electroacoustics , keywords =. doi:10.1109/TAU.1967.1161901 , interhash =

    1967 doi

  59. [73]

    Biometrika , volume =

    Astfalck, Lachlan C and Sykulski, Adam M and Cripps, Edward J , title =. Biometrika , volume =. 2024 , month =. doi:10.1093/biomet/asae033 , url =

    2024 doi

  60. [74]

    Transactions on Machine Learning Research , issn=

    Maxime Oquab and Timoth. Transactions on Machine Learning Research , issn=. 2024 , url=

    2024

  61. [75]

    Proceedings of the 39th International Conference on Machine Learning , pages =

    Understanding Contrastive Learning Requires Incorporating Inductive Biases , author =. Proceedings of the 39th International Conference on Machine Learning , pages =. 2022 , editor =

    2022

  62. [76]

    Proceedings of the 15th International Conference on Neural Information Processing Systems: Natural and Synthetic , pages =

    Belkin, Mikhail and Niyogi, Partha , title =. Proceedings of the 15th International Conference on Neural Information Processing Systems: Natural and Synthetic , pages =. 2001 , publisher =

    2001

  63. [77]

    and Jordan, Michael I

    Ng, Andrew Y. and Jordan, Michael I. and Weiss, Yair , title =. Proceedings of the 15th International Conference on Neural Information Processing Systems: Natural and Synthetic , pages =. 2001 , publisher =

    2001

  64. [78]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Understanding Contrastive Learning via Gaussian Mixture Models , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  65. [79]

    The Eleventh International Conference on Learning Representations , year=

    A theoretical study of inductive biases in contrastive learning , author=. The Eleventh International Conference on Learning Representations , year=

  66. [80]

    Advances in Neural Information Processing Systems , editor=

    Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss , author=. Advances in Neural Information Processing Systems , editor=. 2021 , url=

    2021

  67. [81]

    Proceedings of the 36th International Conference on Neural Information Processing Systems , articleno =

    Balestriero, Randall and LeCun, Yann , title =. Proceedings of the 36th International Conference on Neural Information Processing Systems , articleno =. 2022 , isbn =

    2022

  68. [82]

    International conference on machine learning , pages=

    Whitening for self-supervised representation learning , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  69. [83]

    Natural image statistics: a probabilistic approach to early computational vision

    Aapo Hyv \"a rinen and Jarmo Hurri and Hoyer, \ Patrik O\. Natural image statistics: a probabilistic approach to early computational vision. 2009

    2009

  70. [84]

    2019 , eprint=

    Representation Learning with Contrastive Predictive Coding , author=. 2019 , eprint=

    2019

Pith tools

Reviewed July 9, 2026 · model on record in the stance chip above.