Pith. sign in

REVIEW 3 major objections 6 minor 65 references

Information-Maximized Soft Variable Discretization for Self-Supervised Image Representation Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A self-supervised learning objective that first turns each latent variable into a soft one-hot code can be optimized directly by information measures; the paper proves that its minimizer is a transform-invariant, non-collapsed…

desk verdict Genuinely new non-contrastive objective with solid experiments, but the main theorem is overclaimed for finite batches and the loss derivation has a circular step; method is salvageable and worth refereeing. read the letter →

arxiv 2501.03469 v1 pith:GPBSDKHF submitted 2025-01-07 cs.CV

Chuang Niu , Wenjun Xia , Hongming Shan , Ge Wang This is my paper · ORCID
classification cs.CV
keywords self-supervisedlearningimagerepresentationvariablediscretizationinformationtheorycross-jointentropyredundancyreductionnon-contrastivelinearclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that self-supervised representation learning can be driven directly by information measures rather than by similarity heuristics or linear decorrelation, if the latent space is first re-encoded as a set of discrete variables. It introduces IMSVD, which applies a softmax to each block of the feature vector so that each block behaves like a categorical variable. The proposed cross-joint entropy loss maximizes the information carried by each variable while forcing different views of the same image to share the same discrete code. The central result is Theorem 1: at the minimizer, every variable is one-hot, every variable is uniformly distributed, every pair of variables is jointly uniform and independent, and the mutual information between any two variables is zero. If true, this gives a non-contrastive SSL objective with proven invariance, collapse avoidance, and redundancy reduction, and it provides variable-level explainability.

What carries the argument

The central device is soft variable discretization: the projector output is split into $M$ contiguous blocks of size $D_M$, and each block is normalized by a softmax to form a categorical sub-vector $q_i(m,:)$. This turns a continuous latent vector into a collection of categorical variables whose marginal and joint distributions can be estimated from a batch, making entropy and mutual information computable. The training loss is the cross-joint entropy in Eq. (11): its first term is the average log inner product between the two views' discrete codes, and its second term is the negative entropy of the empirical cross-joint distribution over all variable pairs, masked to exclude diagonal elements inside each self-block. Theorem 1 shows that at the minimum of this loss the codes are exactly one-hot, marginals are uniform, all variable pairs are jointly uniform, and all pairwise mutual information vanishes.

What would settle it

Train an IMSVD model to convergence on ImageNet and compute the empirical marginal $p(m,d)$ and pairwise joint $P(m_1,m_2,d_1,d_2)$ over the training set. If $p(m,d)$ is not close to $1/D_M$ for every variable and $P(m_1,m_2,d_1,d_2)$ is not close to $1/D_M^2$ for every variable pair, or if the empirical mutual information between any two variables is not near zero, then the claimed optimum is not the one reached. A direct check on held-out augmentation pairs, whether $q'_i$ and $q''_i$ are both one-hot and equal, would also settle the mechanism.

Watch

Extended reading notes

Core claim

The paper's central claim is that softly discretizing each latent variable makes information measures directly computable from a training batch, so the SSL objective can be stated and optimized in information-theoretic terms. The author proves Theorem 1: minimizing the cross-joint entropy loss of Eq. (11) drives the two-view features $q'_i$ and $q''_i$ to identical one-hot vectors for every sample and variable, each variable to the uniform marginal $p(m,d)=1/D_M$, every pair of variables to the uniform joint distribution $1/D_M^2$, and the mutual information between any two variables to zero. This is a stronger redundancy guarantee than the pairwise linear decorrelation used by existing non-contrastive methods, because it removes arbitrary nonlinear dependence. The same optimal solution gives transform invariance without collapse and, because each variable pair can encode $(D_M)^2$ distinct samples, the method statistically performs instance discrimination even though it never uses negative pairs.

Load-bearing premise

The loss derivation assumes the two augmented views already give identical discrete codes, even though making them identical is precisely the goal of training; if the views differ during optimization, the loss is a surrogate rather than the exact information-theoretic objective.

Editorial extensions

If this is right

  • Minimizing the cross-joint entropy loss provably drives the representation to a configuration in which every variable is one-hot, every variable is uniformly distributed, and every variable pair is independent, so the learned code is simultaneously transform-invariant, non-collapsed, and redundancy-free.
  • Because redundancy is removed for arbitrary nonlinear dependence rather than only linear correlation, IMSVD can use a shorter embedding and a two-layer projector while matching or beating methods that require 8,192-dimensional features and three-layer projectors.
  • The optimal discrete code statistically performs instance discrimination: with $D_M=80$, each variable pair can encode 6,400 distinct samples, so different images receive distinct codes even though no negative pairs are used.
  • The discrete variables are interpretable: units of the first variable capture texture patterns and units of the second variable capture shape patterns, and these can be visualized and localized with Grad-CAM.
  • The method's performance degrades only mildly with smaller batch sizes and transfers to object detection, instance segmentation, and scene classification, suggesting it does not depend on a large bank of negative samples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The equivalence between the implementable cross-joint loss and the information-theoretic objective rests on assuming the two views already produce identical discrete codes; a variant that estimates the joint distribution symmetrically from both views, or adds an explicit invariance term before forming the cross-joint term, would place the derivation on firmer ground without changing the final theo
  • Theorem 1 describes the global minimizer over distributions, but achieving it requires the batch to be large enough to represent $1/D_M^2$ combinations for every variable pair; very small batches may not reach the stated optimum, and testing at batch sizes where $D_M^2$ exceeds the batch could reveal a different fixed point or a partial collapse.
  • Because the discretized variables are categorical and interpretable, the same loss could be adapted to hierarchical clustering, multi-modal alignment, or generative modeling by treating each variable as a discrete token; the paper notes this potential but does not demonstrate it.
  • The claim that a non-contrastive loss statistically performs contrastive learning suggests a broader principle: any method that maximizes the entropy of a factorization of the latent space may implicitly separate instances, which could guide the design of other non-contrastive objectives.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes IMSVD, a self-supervised representation-learning method in which the projector output is split into M segments, each softly discretized by a softmax over DM units. Marginal and joint distributions of these discrete variables are estimated over the batch, and the training objective (Eqs. (4) and (9)) maximizes cross-view agreement, per-variable entropy, and joint entropy, with r=2 in the implemented loss. The authors derive a cross-joint entropy loss (Eq. (11)) and state Theorem 1, which claims that minimizing this loss drives the two views to identical one-hot codes, uniform per-variable marginals, uniform pairwise joint distributions, and zero pairwise mutual information. The paper reports ImageNet linear and k-NN classification, transfer learning, efficiency, and ablations, plus visualizations of the learned discrete variables, and it makes code publicly available.

Significance. If the theoretical claim were established, IMSVD would be a notable contribution: a non-contrastive SSL objective directly based on information measures, with stronger redundancy reduction than linear decorrelation, and with interpretable, near-discrete embeddings obtained without a hard quantization step. The empirical results are broadly consistent with that narrative, and the paper contains useful strengths: reproducible public code, careful comparisons with Barlow Twins and VICReg, and ablations of batch size, projector depth, feature dimension, and the DM hyperparameter. However, the central theorem as stated is not defensible against the finite-batch loss that is actually optimized, and the derivation of the implemented loss from the information-theoretic objective contains a circular step. The contribution is therefore promising but requires a corrected theoretical statement and verifiable proofs before it can be accepted.

major comments (3)
  1. [Section III-D, Eqs. (10)-(11)] The derivation of the cross-joint entropy loss equates P^c(m1,m2;d1,d2) with the self-joint probability P(m1,m2;d1,d2) by asserting that the embedding features are optimized to be transform-invariant, i.e., q'_i = q''_i. That equality is the intended outcome of the optimization, not a premise available when the loss is derived. During training the two views can and do differ, so Eq. (11) is a surrogate for Eq. (9) rather than an equivalent loss. The information-theoretic interpretation of the actual minimized loss is therefore incomplete. The paper should state explicitly in what sense Eq. (11) approximates or bounds Eq. (9), or prove the desired minimizer properties directly for the cross-joint loss without assuming q'_i = q''_i.
  2. [Theorem 1, Eq. (11), and Section IV implementation settings] Theorem 1 is false as stated for the finite batches actually optimized. The loss in Eq. (11) uses empirical batch probabilities P^c = (1/N) sum_i q'_i(m1,d1) q''_i(m2,d2). With the paper's default settings N=2048 and DM=80, the conclusions p(m,d)=1/80 and P(m1,m2;d1,d2)=1/6400 cannot hold for any one-hot assignment, because they would require per-unit counts of 2048/80=25.6 and 2048/6400=0.32 respectively. Hence the global minimum of the empirical loss does not satisfy the equalities stated in Theorem 1. The theorem should be restated for the population loss, or the paper should provide finite-N discrepancy bounds. Figure 3(a) already shows only approximate uniformity, which is consistent with this limitation rather than with the exact statement of the theorem.
  3. [Appendices I, II, and III] The derivation of Eq. (11) from Eq. (9) and the proof of Theorem 1 are relegated to appendices that are not present in the reviewed manuscript. Because these arguments are load-bearing for the central claim, the review cannot verify them. The paper should include the full derivation and proof, or the supplementary material containing them should be part of the version under review.
minor comments (6)
  1. [Abstract and Introduction] The word "non-travail" in the abstract appears to be a typo for "non-collapsed"; please correct it.
  2. [Section III-B, Eq. (2)] The text says "where pi(m,d) denotes the probability..." but the quantity defined is p(m,d); the subscript i is not introduced and should be removed.
  3. [Section V-B, Table VIII] The column headers "DE+OE+TIC" and "DE+OE+TI" do not match the abbreviations DE, OE, TIC, and TI defined in the text; please make the table headers consistent with the defined terms.
  4. [References] Reference [10] is cited as "Unsupervised representation learning by predicting image rotations," but the method being compared is VICReg; the citation title and venue should be corrected.
  5. [Section III-E] The sentence "IMSVD statically assigns instances" should likely read "statistically" or "stochastically"; as written, "statically" is confusing.
  6. [Section V-C and cross-references] The cross-reference to "Sec. V-B3" does not match the numbered items in Section V-B, and the description of the 400x400 matrix in Fig. 3(a) should clarify which variables and dataset split are used.

Circularity Check

1 steps flagged · score 4.0 of 10

Moderate circularity: Eq. (11) is derived from Eq. (9) by assuming the transform-invariance and one-hotness that Theorem 1 is supposed to establish.

  1. self definitional [Section III-D, between Eq. (10) and Eq. (11)]
    "Since the embedding features are optimized to be transform-invariant; i.e., q′ i = q′′ i in the Eq. (9), the cross-joint probability will be equal to the self-joint entropy; i.e., P c(m1, m2; d1, d2) = P (m1, m2; d1, d2). Then, the cross-joint entropy loss function can be derived from Eq. (9) as follows..."

    Eq. (10) defines P^c using two different views q' and q''. The text then asserts q' = q'' to identify P^c with the self-joint P and to derive Eq. (11). But q' = q'' is the transform-invariance conclusion of Theorem 1, i.e., the very property the loss is supposed to produce, not an identity available during optimization. At non-optimal states the two views differ, so Eq. (11) is not an equivalent reformulation of Eq. (9); it is a surrogate whose derivation assumes the target optimum. Additionally, the indicator in Eq. (11) drops the diagonal-block off-diagonal terms, which is valid only if each q_i(m,:) is already a one-hot vector, another conclusion of Theorem 1. The 'equivalent' loss is therefore justified by substituting the theorem's own conclusions into the derivation.

full rationale

The only load-bearing circular step is in Section III-D: the cross-joint probability P^c in Eq. (10) is equated to the self-joint P by asserting q'_i = q''_i, and the indicator term further presumes one-hotness. Both are conclusions of Theorem 1, so the derivation of Eq. (11) from Eq. (9) is not independent of the result it is used to prove. The score is not higher because the theorem also contains substantive uniformity and independence claims that are not pre-assumed, the loss is implemented and tested against external benchmarks, and there is no load-bearing self-citation chain: the proof is deferred to Appendix-II rather than imported from the authors' prior work. Separately, there are correctness concerns not scored as circularity: Theorem 1 as stated cannot hold for the finite training batches used (N=2048, D_M=80, so 1/80 and 1/6400 are not integer multiples of 1/N), and the omitted Appendix-II proof prevents verification of the theorem from the submitted text.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method rests on the MultiView assumption, the categorical reparametrization of each latent segment, and the equality between cross-joint and self-joint probabilities asserted during the loss derivation. The main free parameters are the loss weights lambda/beta and the discretization dimension D_M, both set by hand or by small ablations. No new physical entities are introduced.

free parameters (5)
  • Lambda (λ) = 1
    Weight for the entropy terms in Eqs. (4) and (11); set to 1 by hand with the comment 'neither too small nor too large', no sensitivity experiment reported.
  • Beta (β) = 1
    Same weighting role as λ; combined with λ in the final loss, not tuned separately.
  • Sub-vector dimension D_M = 80
    Number of discrete units per variable; ablated in Table IX and chosen as best. The paper says it needs to be empirically selected per dataset.
  • Number of variables M = 102
    Chosen together with D_M so total embedding D=8160 matches VICReg and Barlow Twins dimensions.
  • Projector depth = 2
    Ablated in Table VI; unlike the baselines, a two-layer projector works best for IMSVD.
assumptions (5)
  • domain assumption MultiView assumption: shared information between different augmentations of the same image is sufficient for downstream tasks.
    Introduced in Sec. I and Sec. III-A as the premise for pulling views together. If false for a task, the invariance objective can discard needed information.
  • domain assumption Each latent variable is represented as a categorical variable with D_M exclusive states via softmax.
    Eq. (1) defines the discretization; the information measures are computed over these categorical variables rather than the original continuous features.
  • domain assumption Batch empirical frequencies approximate true latent distributions.
    Eqs. (2) and (3) estimate p and P from a batch of size N; the theorem assumes these estimates are accurate but no finite-batch correction is given.
  • ad hoc to paper Cross-joint probability may be replaced by self-joint probability because q'_i = q''_i.
    Used in Sec. III-D to justify Eq. (11). This assumes the invariance the loss is meant to produce and is the main circular step.
  • standard math Uniform categorical distributions maximize entropy over finite alphabets.
    Standard maximum-entropy result used in Theorem 1 to identify the form of the minimizer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Information-Maximized Soft Variable Discretization for Self-Supervised Image Representation Learning." pith.science (2026). https://pith.science/paper/GPBSDKHF

@misc{pith2026250103469,
  author       = {Pith},
  title        = {Pith review of: Information-Maximized Soft Variable Discretization for Self-Supervised Image Representation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GPBSDKHF}},
  note         = {Machine review of arXiv:2501.03469}
}
read the original abstract

Self-supervised learning (SSL) has emerged as a crucial technique in image processing, encoding, and understanding, especially for developing today's vision foundation models that utilize large-scale datasets without annotations to enhance various downstream tasks. This study introduces a novel SSL approach, Information-Maximized Soft Variable Discretization (IMSVD), for image representation learning. Specifically, IMSVD softly discretizes each variable in the latent space, enabling the estimation of their probability distributions over training batches and allowing the learning process to be directly guided by information measures. Motivated by the MultiView assumption, we propose an information-theoretic objective function to learn transform-invariant, non-travail, and redundancy-minimized representation features. We then derive a joint-cross entropy loss function for self-supervised image representation learning, which theoretically enjoys superiority over the existing methods in reducing feature redundancy. Notably, our non-contrastive IMSVD method statistically performs contrastive learning. Extensive experimental results demonstrate the effectiveness of IMSVD on various downstream tasks in terms of both accuracy and efficiency. Thanks to our variable discretization, the embedding features optimized by IMSVD offer unique explainability at the variable level. IMSVD has the potential to be adapted to other learning paradigms. Our code is publicly available at https://github.com/niuchuangnn/IMSVD.

Figures

Figures reproduced from arXiv: 2501.03469 by the authors.

Figure 1
Figure 1. Illustration of discrete variables for encoding images. (a) The feature vector is statistically optimized to be a set of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. SSL framework through IMSVD optimized with the joint entropy loss. For illustration purposes, the embedding feature [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of IMSVD. (a) Cross-joint probability [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visualization of learned IMSVD features on ImageNet validation set. The left side shows the samples assigned to the [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Local feature visualization using Grad-CAM. Here the Grad-CAM maps show the local features of the learned variables, [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 43 canonical work pages

  1. [1]

    On the opportunities and risks of foundation models,

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al. , “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  2. [2]

    Noisy-as-clean: Learning self-supervised denoising from corrupted image,

    J. Xu, Y . Huang, M.-M. Cheng, L. Liu, F. Zhu, Z. Xu, and L. Shao, “Noisy-as-clean: Learning self-supervised denoising from corrupted image,” IEEE Transactions on Image Processing , vol. 29, pp. 9316– 9329, 2020

    work page 2020

  3. [3]

    Noise suppression with similarity-based self-supervised deep learning,

    C. Niu, M. Li, F. Fan, W. Wu, X. Guo, Q. Lyu, and G. Wang, “Noise suppression with similarity-based self-supervised deep learning,” IEEE Transactions on Medical Imaging , vol. 42, no. 6, pp. 1590–1602, 2023

    work page 2023

  4. [4]

    Learning discrete representations from reference images for large scale factor image super-resolution,

    W. Sun and Z. Chen, “Learning discrete representations from reference images for large scale factor image super-resolution,” IEEE Transactions on Image Processing , vol. 31, pp. 1490–1503, 2022

    work page 2022

  5. [5]

    Gatcluster: Self-supervised gaussian-attention network for image clustering,

    C. Niu, J. Zhang, G. Wang, and J. Liang, “Gatcluster: Self-supervised gaussian-attention network for image clustering,” in Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16 . Springer, 2020, pp. 735–751

    work page 2020

  6. [6]

    Spice: Semantic pseudo-labeling for image clustering,

    C. Niu, H. Shan, and G. Wang, “Spice: Semantic pseudo-labeling for image clustering,” IEEE Transactions on Image Processing, vol. 31, pp. 7264–7278, 2022

    work page 2022

  7. [7]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in ICML, vol. 119, 2020, pp. 1597–1607

    work page 2020

  8. [8]

    Visualizing and under- standing contrastive learning,

    F. Sammani, B. Joukovsky, and N. Deligiannis, “Visualizing and under- standing contrastive learning,” IEEE Transactions on Image Processing, vol. 33, pp. 541–555, 2024

    work page 2024

Show all 65 references
  1. [9]

    Momentum contrast for unsupervised visual representation learning,

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in CVPR, June 2020

    2020

  2. [10]

    Unsupervised representation learning by predicting image rotations,

    A. Bardes, J. Ponce, and Y . LeCun, “Unsupervised representation learning by predicting image rotations,” in ICLR, 2022

    2022

  3. [11]

    Self-supervised feature augmentation for large image object detection,

    X. Pan, F. Tang, W. Dong, Y . Gu, Z. Song, Y . Meng, P. Xu, O. Deussen, and C. Xu, “Self-supervised feature augmentation for large image object detection,” IEEE Transactions on Image Processing , vol. 29, pp. 6745– 6758, 2020

    2020

  4. [12]

    Unsupervised learning of image segmentation based on differentiable feature clustering,

    W. Kim, A. Kanezaki, and M. Tanaka, “Unsupervised learning of image segmentation based on differentiable feature clustering,” IEEE Transactions on Image Processing , vol. 29, pp. 8055–8068, 2020. 11

    2020

  5. [13]

    Dis- criminative unsupervised feature learning with convolutional neural networks,

    A. Dosovitskiy, J. T. Springenberg, M. Riedmiller, and T. Brox, “Dis- criminative unsupervised feature learning with convolutional neural networks,” NeurIPS, vol. 27, 2014

    2014

  6. [14]

    To compress or not to compress–self- supervised learning and information theory: A review,

    R. Shwartz-Ziv and Y . LeCun, “To compress or not to compress–self- supervised learning and information theory: A review,” arXiv preprint arXiv:2304.09355, 2023

    2023 arXiv

  7. [15]

    What makes for good views for contrastive learning?

    Y . Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, “What makes for good views for contrastive learning?” in NeurIPS, vol. 33, 2020, pp. 6827–6839

    2020

  8. [16]

    Self- supervised learning with an information maximization criterion,

    S. Ozsoy, S. Hamdan, S. Arik, D. Yuret, and A. Erdogan, “Self- supervised learning with an information maximization criterion,” Ad- vances in Neural Information Processing Systems , vol. 35, pp. 35 240– 35 253, 2022

    2022

  9. [17]

    On the information bottleneck theory of deep learning,

    A. M. Saxe, Y . Bansal, J. Dapello, M. Advani, A. Kolchinsky, B. D. Tracey, and D. D. Cox, “On the information bottleneck theory of deep learning,” in International Conference on Learning Representations , 2018

    2018

  10. [18]

    Data-efficient image recognition with contrastive predictive coding,

    O. Henaff, “Data-efficient image recognition with contrastive predictive coding,” in ICML, 2020, pp. 4182–4192

    2020

  11. [19]

    Barlow twins: Self-supervised learning via redundancy reduction,

    J. Zbontar, L. Jing, I. Misra, Y . LeCun, and S. Deny, “Barlow twins: Self-supervised learning via redundancy reduction,” in ICML, 2021, pp. 12 310–12 320

    2021

  12. [20]

    Opening the black box of deep neural networks via information,

    R. Shwartz-Ziv and N. Tishby, “Opening the black box of deep neural networks via information,” arXiv preprint arXiv:1703.00810 , 2017

    2017 arXiv

  13. [21]

    Attribute learning in large-scale datasets,

    O. Russakovsky and L. Fei-Fei, “Attribute learning in large-scale datasets,” in ECCV. Springer, 2010, pp. 1–14

    2010

  14. [22]

    Categorical reparameterization with gumbel-softmax,

    E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” arXiv preprint arXiv:1611.01144 , 2016

    2016 arXiv

  15. [23]

    The concrete distribution: A continuous relaxation of discrete random variables,

    C. J. Maddison, A. Mnih, and Y . W. Teh, “The concrete distribution: A continuous relaxation of discrete random variables,” in International Conference on Learning Representations , 2017

    2017

  16. [24]

    Discretization techniques: A recent survey,

    S. Kotsiantis and D. Kanellopoulos, “Discretization techniques: A recent survey,” GESTS International Transactions on Computer Science and Engineering, vol. 32, no. 1, pp. 47–58, 2006

    2006

  17. [25]

    A survey of discretization techniques: Taxonomy and empirical analysis in supervised learning,

    S. Garc ´ıa, J. Luengo, J. A. S ´aez, V . L ´opez, and F. Herrera, “A survey of discretization techniques: Taxonomy and empirical analysis in supervised learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 25, pp. 734–750, 2013

    2013

  18. [26]

    Estimating information flow in deep neural networks,

    Z. Goldfeld, E. Van Den Berg, K. Greenewald, I. Melnyk, N. Nguyen, B. Kingsbury, and Y . Polyanskiy, “Estimating information flow in deep neural networks,” in 36th International Conference on Machine Learning, ICML 2019. International Machine Learning Society (IMLS), 2019, pp....

    2019

  19. [27]

    Zero-shot text-to-image generation,

    A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. V oss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in International Conference on Machine Learning . PMLR, 2021, pp. 8821–8831

    2021

  20. [28]

    Beit: Bert pre-training of image transformers,

    H. Bao, L. Dong, S. Piao, and F. Wei, “Beit: Bert pre-training of image transformers,” arXiv preprint arXiv:2106.08254 , 2021

    2021 arXiv

  21. [29]

    Neural discrete representation learning,

    A. Van Den Oord, O. Vinyals et al. , “Neural discrete representation learning,” Advances in neural information processing systems , vol. 30, 2017

    2017

  22. [30]

    Taming transformers for high- resolution image synthesis,

    P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high- resolution image synthesis,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 12 873–12 883

    2021

  23. [31]

    Unsupervised feature learning via non-parametric instance discrimination,

    Z. Wu, Y . Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in CVPR, 2018, pp. 3733– 3742

    2018

  24. [32]

    Unsupervised embed- ding learning via invariant and spreading instance feature,

    M. Ye, X. Zhang, P. C. Yuen, and S.-F. Chang, “Unsupervised embed- ding learning via invariant and spreading instance feature,” in CVPR, 2019, pp. 6210–6219

    2019

  25. [33]

    With a little help from my friends: Nearest-neighbor contrastive learning of visual representations,

    D. Dwibedi, Y . Aytar, J. Tompson, P. Sermanet, and A. Zisserman, “With a little help from my friends: Nearest-neighbor contrastive learning of visual representations,” in ICCV, 2021, pp. 9588–9597

    2021

  26. [34]

    Representation learning with contrastive predictive coding,

    A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv:1807.03748, 2018

    2018 arXiv

  27. [35]

    Deep clustering for unsupervised learning of visual features,

    M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in ECCV, 2018, pp. 132–149

    2018

  28. [36]

    Unsupervised pre- training of image features on non-curated data,

    M. Caron, P. Bojanowski, J. Mairal, and A. Joulin, “Unsupervised pre- training of image features on non-curated data,” in ICCV, 2019, pp. 2959–2968

    2019

  29. [37]

    Self-labelling via simulta- neous clustering and representation learning,

    Y . M. Asano, C. Rupprecht, and A. Vedaldi, “Self-labelling via simulta- neous clustering and representation learning,” arXiv:1911.05371, 2019

    1911 arXiv

  30. [38]

    Clusterfit: Improving generalization of visual representations,

    X. Yan, I. Misra, A. Gupta, D. Ghadiyaram, and D. Mahajan, “Clusterfit: Improving generalization of visual representations,” in CVPR, 2020, pp. 6509–6518

    2020

  31. [39]

    Unsupervised deep learning by neighbourhood discovery,

    J. Huang, Q. Dong, S. Gong, and X. Zhu, “Unsupervised deep learning by neighbourhood discovery,” in ICML, 2019, pp. 2849–2858

    2019

  32. [40]

    Local aggregation for unsuper- vised learning of visual embeddings,

    C. Zhuang, A. L. Zhai, and D. Yamins, “Local aggregation for unsuper- vised learning of visual embeddings,” in ICCV, 2019, pp. 6002–6012

    2019

  33. [41]

    Obow: Online bag-of-visual-words generation for self-supervised learn- ing,

    S. Gidaris, A. Bursuc, G. Puy, N. Komodakis, M. Cord, and P. Perez, “Obow: Online bag-of-visual-words generation for self-supervised learn- ing,” in CVPR, 2021, pp. 6830–6840

    2021

  34. [42]

    Unsupervised learning of visual features by contrasting cluster assign- ments,

    M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assign- ments,” NeurIPS, vol. 33, pp. 9912–9924, 2020

    2020

  35. [43]

    Byol works even without batch statistics,

    P. H. Richemond, J.-B. Grill, F. Altch ´e, C. Tallec, F. Strub, A. Brock, S. Smith, S. De, R. Pascanu, B. Piot et al., “Byol works even without batch statistics,” arXiv:2010.10241, 2020

    2010 arXiv

  36. [44]

    Exploring simple siamese representation learning,

    X. Chen and K. He, “Exploring simple siamese representation learning,” in CVPR, June 2021, pp. 15 750–15 758

    2021

  37. [45]

    Whitening for self-supervised representation learning,

    A. Ermolov, A. Siarohin, E. Sangineto, and N. Sebe, “Whitening for self-supervised representation learning,” in ICML, 2021, pp. 3015–3024

    2021

  38. [46]

    Understanding contrastive representation learning through alignment and uniformity on the hypersphere,

    T. Wang and P. Isola, “Understanding contrastive representation learning through alignment and uniformity on the hypersphere,” in ICML, vol. 119, 2020, pp. 9929–9939

    2020

  39. [47]

    How does simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning,

    C. Zhang, K. Zhang, C. Zhang, T. X. Pham, C. D. Yoo, and I. S. Kweon, “How does simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning,” in ICLR, 2021

    2021

  40. [48]

    Understanding self-supervised learn- ing dynamics without contrastive pairs,

    Y . Tian, X. Chen, and S. Ganguli, “Understanding self-supervised learn- ing dynamics without contrastive pairs,” in ICML, 2021, pp. 10 268– 10 278

    2021

  41. [49]

    Contrastive and non-contrastive self- supervised learning recover global and local spectral embedding meth- ods,

    R. Balestriero and Y . LeCun, “Contrastive and non-contrastive self- supervised learning recover global and local spectral embedding meth- ods,” arXiv preprint arXiv:2205.11508 , 2022

    2022 arXiv

  42. [50]

    On the du- ality between contrastive and non-contrastive self-supervised learning,

    Q. Garrido, Y . Chen, A. Bardes, L. Najman, and Y . LeCun, “On the du- ality between contrastive and non-contrastive self-supervised learning,” in The Eleventh International Conference on Learning Representations , 2023

    2023

  43. [51]

    Information theoretical analysis of multivariate correla- tion,

    S. Watanabe, “Information theoretical analysis of multivariate correla- tion,” IBM Journal of research and development, vol. 4, no. 1, pp. 66–82, 1960

    1960

  44. [52]

    On feature decorrelation in self-supervised learning,

    T. Hua, W. Wang, Z. Xue, S. Ren, Y . Wang, and H. Zhao, “On feature decorrelation in self-supervised learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 9598–9608

    2021

  45. [53]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, June 2016

    2016

  46. [54]

    Large batch training of convolu- tional networks,

    Y . You, I. Gitman, and B. Ginsburg, “Large batch training of convolu- tional networks,” arXiv:1708.03888, 2017

    2017 arXiv

  47. [55]

    Accurate, large minibatch sgd: Training imagenet in 1 hour,

    P. Goyal, P. Doll ´ar, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y . Jia, and K. He, “Accurate, large minibatch sgd: Training imagenet in 1 hour,” arXiv:1706.02677, 2017

    2017 arXiv

  48. [56]

    Sgdr: Stochastic gradient descent with warm restarts,

    I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” arXiv:1608.03983, 2016

    2016 arXiv

  49. [57]

    Mask r-cnn,

    K. He, G. Gkioxari, P. Doll ´ar, and R. Girshick, “Mask r-cnn,” in ICCV, 2017, pp. 2961–2969

    2017

  50. [58]

    Detectron2,

    Y . Wu, A. Kirillov, F. Massa, W.-Y . Lo, and R. Girshick, “Detectron2,” https://github.com/facebookresearch/detectron2, 2019

    2019

  51. [59]

    Self-supervised learning of pretext- invariant representations,

    I. Misra and L. v. d. Maaten, “Self-supervised learning of pretext- invariant representations,” in CVPR, 2020, pp. 6707–6717

    2020

  52. [60]

    Improving self- supervised learning by characterizing idealized representations,

    Y . Dubois, S. Ermon, T. Hashimoto, and P. Liang, “Improving self- supervised learning by characterizing idealized representations,” in Ad- vances in Neural Information Processing Systems , A. H. Oh, A. Agar- wal, D. Belgrave, and K. Cho, Eds., 2022

    2022

  53. [61]

    Prototypical contrastive learning of unsupervised representations,

    J. Li, P. Zhou, C. Xiong, and S. Hoi, “Prototypical contrastive learning of unsupervised representations,” in ICLR, 2021

    2021

  54. [62]

    Bootstrap your own latent-a new approach to self-supervised learning,

    J.-B. Grill, F. Strub, F. Altch ´e, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar et al. , “Bootstrap your own latent-a new approach to self-supervised learning,” NeurIPS, vol. 33, pp. 21 271–21 284, 2020

    2020

  55. [63]

    Learning deep features for scene recognition using places database,

    B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva, “Learning deep features for scene recognition using places database,” NeurIPS, vol. 27, 2014

    2014

  56. [64]

    The pascal visual object classes (voc) challenge,

    M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisser- man, “The pascal visual object classes (voc) challenge,” International journal of computer vision , vol. 88, no. 2, pp. 303–338, 2010

    2010

  57. [65]

    Grad-cam: Visual explanations from deep networks via gradient-based localization,

    R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 618–626

    2017

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.